[ESQL] New and improved autosuggestion

[dej611]

Summary

With this PR the autocomplete feature for ESQL is set at the same/better level than the previous one in main.
In particular:

• New "shared" grammar should be supported for autosuggest
• the suggestion logic is now based on the AST data structure and the logic is mostly generic*
• the suggestion logic is now aware of types when proposing fields, functions and variables
• the suggestion logic has automatic re-suggest on specific types (i.e. + or other builtin functions)
• previous tests have been mainly retained to verify feature-parity and in most part extended**
• some suggestions are more location-aware than others, and can automatically inject , together with the selected argument
• new variables are automatically injected with = where it make sense

File hierarchy has changed as well in this PR to accomodate better modularization for different services: ast, validation, autocomplete, hover and signature.

Also various bug fixes have been provided for existing features/services:

• rename AST walker had some bugs for partial/incomplete commands
• rename validation had some bugs due to the bug above
• ON option for ENRICH was set as mandatory, now it's set to optional
• ENRICH options had several bugs on location extents computation. This was due to the specific grammar definition that didn't make it automatic as other commands to define correct location. It should be fixed now.
• some AST functions/options contained empty values which were confusing for services built upon it, and all occurency of this type of bug has been fixed
• fixed various function/command definitions

* there are still some edge cases that were required to be handled specifically. But most of the times the logic goes thru the definition data structure.
** some tests - i.e. suggest open brackets - have been removed as now functions are proposed entirely. For the "closing brackets" suggestion the editor is now configured to automatically open/close them.

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[stratoula]

@dej611 this looks fantastic. I found some cases where autocomplete can be improved:

1. On stats after selecting comma, comma is again suggested instead of a new variable

2. The fourth option here is wrong. (We have a bug in Discover table and list and for some reason we display the column with underscores while it is not correct but I will create an issue). Can we not suggest it? Also the third is correct but unfortunately should be like avg(bytes) to work properly. Can we add the `` when the user clicks it?

3. Not autocomplete related but this is red while it should not. This is a valid syntax

4. Eval function arguments suggestions are wrong

5. Nothing is suggested here :(

[stratoula]

@dej611 we have this problem also in main but it is more prominent here. It seems that we error out on the capitalized version of the BY operator.

Also not related to autocomplete. We can fix on a separate PR.

[ninoslavmiskovic]

@dej611 this is great work ! Huge improvement! So nice to e.g get suggestions on index nr 2,3,4 and the meta field .

[dej611]

@dej611 we have this problem also in main but it is more prominent here. It seems that we error out on the capitalized version of the BY operator.

Fixed by b07f157 and 819e40c (added unit test as well)

On stats after selecting comma, comma is again suggested instead of a new variable

Fixed by 985486f (added unit test as well)

Eval function arguments suggestions are wrong

Fixed by e9a514d (added unit test that checks no consecutive fn suggested)

Solving this issue avoiding duplicates has a side effect the fix also for the tooltip scroll and now the scroll is moved on top to fields again.

Nothing is suggested here :(

Fixed by 2816c95 (added unit tests as well)

Not autocomplete related but this is red while it should not. This is a valid syntax

Fixed by 304621d (added unit tests as well)

The fourth option here is wrong. (We have a bug in Discover table and list and for some reason we display the column with underscores while it is not correct but I will create an issue). Can we not suggest it?

Fixed by d448041 (added unit tests as well)

Also the third is correct but unfortunately should be like avg(bytes) to work properly. Can we add the `` when the user clicks it?

Fixed by 43d8d84 (added unit tests as well)

[stratoula]

Supercalifragilisticexpialidocious! Bugs fixed, changes LGTM!

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: cf8eac0

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
textBasedLanguages	150.9KB	151.0KB	+74.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
kbnUiSharedDeps-srcJs	2.3MB	2.7MB	⚠️ +379.0KB

History

• 💔 Build #178740 failed d448041
• 💛 Build #178420 was flaky 7f70029
• 💛 Build #178364 was flaky 4304c3f
• 💛 Build #177843 was flaky f601627
• 💔 Build #177818 failed 8873cc5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream