[5.0] Smart Search: Index custom fields

[Hackwar]

Pull Request for Issue #12829. Re-do of #36747.

Summary of Changes

This PR implements indexing of the custom fields for a component. This is controlled on a per field basis, where you can select if the content should be just added to the search index or handled as a taxonomy or both or actually none of the above. You can then implement this feature for your respective finder plugin with a single line of code.
Each custom field has a new option in its edit screen where you can select how the indexer should treat this field. The finder plugin then has to call Helper::addCustomFields($item, 'context'); and the rest is just magic. ;)
Right now this is implemented for com_content and com_contacts.

This is a re-do of #36747, which had to be reverted due to issues when indexing via CLI. This issue should be solved when #38585 and #39550 are merged.

Testing Instructions

0. Make sure that Fix FileLayout to work in non Web context, like Cli, Api #38585 and Fix PluginHelper::getLayoutPath to work in non Web context, like Cli, Api #39550 are merged.
1. Your installation should have some custom fields for com_content or com_contact in it.
2. Edit a custom field and set the "Smart Search" option to "Make searchable and add as taxonomy".
3. Run the indexing process either by running the mass indexer in the smart search component or by editing an item and saving it again.
4. See that the custom field shows up as a taxonomy and can be find by searching for its value.

Documentation Changes Required

The documentation needs to be amended to contain how to implement this feature for third party finder plugins.

[bembelimen]

Thanks @Hackwar for re-doing this, it's a nice and important function missing in the indexer!

[coolcat-creations]

Thank you @Hackwar is this already ready to test?

[crystalenka]

Happy to test this also when it's ready!

[Hackwar]

I would love to claim that this is ready to test, but it turns out I overlooked something and I don't know how to solve it right now. Yes, this code properly indexes the custom fields and this is all working fine, however the content is not updated or modified when the definition of the custom field is changed. So when you unpublish, delete or change the access to a custom field after indexing, that is not updated in the index. There also is no easy solution to make this automatic. My current solution would be to add warning messages if the access, publish or smart search setting for a custom field is changed and then to display those after saving.

[Hackwar]

So after some considerations there is no feasible solution to update the index automatically when a field is unpublished or the access is changed. The next best thing would be to display a message after saving that the smart search index might need updating, which I've added for the cases when a field has its access or publish state changed or when it has been added or removed as to-be-indexed field. Thus: We are ready to test (and merge) this feature.

[coolcat-creations]

Can index be triggered automatically after doing so? How did the search component handle this?

[alikon]

well you can use console command php cli/joomla.php finder:indexer maybe wtih cron
or we should create a scheduled task plugin for this

[crystalenka]

Doing it automatically would be preferred for sure.

If not automatically, then the notice message after update with a link or button to trigger reindexing would be second best. Update a field, see the notice, click the button to reindex OR ignore because you're still making changes, and then click the button to reindex when you're done

[Hackwar]

It is not possible to automatically reindex all of this, because we are changing a field and this field might be referenced in thousands of articles (or other content items). Depending on the server and everything, it might take days to re-index all of that and that is simply not something that can be done when clicking on save in the field edit form. We also don't have a link (at the moment) to redirect to the indexing process, because that process normally opens a modal and then fires of a few ajax requests. It might be worth to revisit that in the future and to redesign that process to go to a specific page instead of inside a modal.

The search component always searched directly on the live data, which is why you couldn't get sorting by relevance, filters and depending on your setup could run into performance issues.

[coolcat-creations]

Can smart Search get a switch between "live" and "index" ?
On bigger sites it seems to make sense - I see on a website with 90.000 articles the indexing never gets finished and ends up in a server error. (It's a Joomla 3.10.11 site tough)

[Hackwar]

No, you can't switch between "live" and "index".

[bembelimen]

Just an idea: write a scheduler task plugin for reindexing and mention it in the blue description box of smart search or enable it by default (including a check if smart search is active)

[alexandreelise]

Thanks for this feature. Joomla is cooler and cooler versions after versions bringing useful and powerful features for developers and content oriented websites thanks for all your hard work Super Joomlers!

[carlitorweb]

@Hackwar after apply the patch, only worked after I edit an article. But did not worked when I first click the mass indexer in the smart search component.

Also, I got this. I think this is expected right? And the user need add a override to this string in the administrator?

[laoneo]

Thank you for your contribution, unfortunately the 4.4-dev branch doesn't get new features. When #40181 is merged this pr should be rebased to 5.0-dev. Thanks for understanding.

[richard67]

Why closed? Should we reopen the original issue?

[HLeithner]

thanks, we need documentation for this.

[richard67]

I get the following PHP warnings now on a clean 5.0-dev branch when installing blog sample data:

PHP Warning:  Undefined array key "searchindex" in /home/richard/lamp/public_html/joomla-cms-5.0-dev/administrator/components/com_fields/src/Model/FieldModel.php on line 119
PHP Warning:  Attempt to read property "params" on null in /home/richard/lamp/public_html/joomla-cms-5.0-dev/administrator/components/com_fields/src/Model/FieldModel.php on line 120
PHP Warning:  Trying to access array offset on value of type null in /home/richard/lamp/public_html/joomla-cms-5.0-dev/administrator/components/com_fields/src/Model/FieldModel.php on line 120
PHP Warning:  Undefined array key "searchindex" in /home/richard/lamp/public_html/joomla-cms-5.0-dev/administrator/components/com_fields/src/Model/FieldModel.php on line 120
PHP Warning:  Undefined array key "searchindex" in /home/richard/lamp/public_html/joomla-cms-5.0-dev/administrator/components/com_fields/src/Model/FieldModel.php on line 121