[HZ-341] Fix Solr cron full-indexing functionality

[jsperhac]

Summary

The core plugin plgCronSolr is meant to provide full and queue-based Solr reindexing of Hub components via Hubzero cron. The plugin's runFullIndex() function for full reindexing of all indexed Solr content was buggy and did not reindex Hub content as expected. This PR repairs that functionality.

This work is described in Jira Card #HZ-341.

Fix details

The following issues are addressed:

• plugin's runFullIndex() function should call indexSearchResults() for each indexed component, and should return a boolean when all indexing is complete
• plugin should only reindex those components that were previously indexed
• language files and xml manifest were cleaned up and corrected; references to other plugins and unused options were removed
• an isset() check was done for $recordsIndexed['offset'] in components/com_search/admin/controllers/searchable.php prior to accessing the variable.

Note

Previous PRs have corrected 500 errors thrown by this plugin (See #1586). The present PR builds on this earlier work by fixing the reindexing functionality. An additional function is roughed out in this plugin to enable queue-based indexing of components. This will be implemented in future work, as it is also non-functional at present (see [HZ-342]).

Motivation

The plgCronSolr plugin's runFullIndex() function could not be run reliably via Hubzero cron as designed. It only intermittently indexed components; it attempted to index components not configured for indexing; and it did not return a boolean, as required by Hubzero cron.

From the Google Doc called “Search improvements for hubzero platform”:

Running the full index task originally picked the non-indexed component with the lowest unique id from jos_solr_search_searchcomponents, enabled it regardless of its state, and (re)indexed it.

Testing

This work was tested by scheduling the Solr Full Index option via Hubzero cron for various configurations of enabled and disabled components. During testing, monitoring various logs and database tables to ensure that the reindexing occurred as specified.

Testing is performed using the admin back-end to set up Solr indexing and Hubzero cron full-indexing; using terminals to monitor logs; and querying database tables.

For Solr administration, access the admin interface, Components:Search, then Searchable Components. Indexing is enabled on a given component (e.g. Groups or Resources) by clicking its "Active" icon. The Records column for a component indicates the number of records currently indexed in Solr.

Logs:

• Solr log: /var/log/hubzero-solr/solr.log (may be found in /srv/hubzero-solr/logs/, in which case symlink needed)
• Hubzero cron log: /var/log/hubzero-cms/hub-cmscron.log

Database tables:

• jos_cron_jobs contains cron jobs and their status and schedules
• jos_solr_search_searchcomponents should show indexed components, index item counts

Other

No hotfix is needed. Review will be assigned to Nick and Dave.

Out of scope

Other findings were noted on undertaking this work. These should be scoped separately:

• searchable components that are not presently indexed sometimes show a nonzero record count in the admin Solr searchable components interface. This should be fixed.
• note that regardless, these items are not indexed, but the total record count should be 0. Likely this is in: components/com_search/admin/controllers/searchable.php
• On indexing the Blog component, then clicking on the record count: admin interface displays 500 error: "undefined index: access_level". This is likely to be a model class problem.
• Similarly, in Citations, no records display on indexing. Then, on clicking through on the record count, admin interface throws "Undefined index: owner_type". This is likely to be a model class problem.
• It would be helpful to display indexing timestamp (indexed) and actual record count (indexed_records) in the jos_solr_search_searchcomponents table
• Logging and error handling is wanted. Presently, one accomplishes this by scrutinizing the solr.log and/or the hub-cmscron.log while indexing.