[ES|QL] Remove command option definitions

[drewdaemon]

Summary

This PR removes the declarative objects that were meant to describe the behavior of "options" (see details section below if you don't know what I'm talking about). It does not remove "options" as a concept from our AST. "Option" is probably the wrong name for the subcommands in the AST but, at the moment, it is working fine how it is.

Here is a list of what these definitions were being used for and where I ended up.

Use	How it worked	What I did
To generate command declarations for display in suggestions menu	It had some complex logic to try to construct a declaration string from the information in the signature property	I replaced this with statically declared declaration strings on the command definitions. I took most of them directly from our docs. They are a better result than the autogenerated stuff
To build the METADATA suggestion	the definition was passed into buildOptionDefinition	I declared the METADATA suggestion statically in the FROM autocomplete code.
To check for field correctness in METADATA	This logic lived in the option definition's validate method	I moved it to the FROM command's validate method
To validate the type of the value assigned to APPEND_SEPARATOR in DISSECT	This logic lived in the option definition's validate method	I moved it to the DISSECT command's validate method
To check if the left side of the equals sign in DISSECT is "APPEND_SEPARATOR	In most cases, the parser catches stuff like this, but for some reason DISSECT's grammar is very loose so we have been stepping in with our own validation (maybe we should suggest changing this). This was the only case that was triggering the "Unknown option" message.	I moved it to the DISSECT command's validate method
To prevent default column validation for METADATA	This was the only true use of the skipCommonValidation property which would prevent the validator trying to find metadata fields in the standard field list	I inserted an option name check directly into the validation code. It's not a good long-term solution, but it is actually an improvement since the former code pretended to be general but was actually just for METADATA. At least now it is clear what the exception is for.
To filter functions and operators that are available after BY	Function definitions sometimes declare that they are supported in a by statement. The validator checks if the function does.	This didn't change. The option nodes in the AST are still there and we are still relying on the supportedCommands and supportedOptions properties in the function definitions.

Pictures

New, statically-defined declarations

In cases besides APPEND_SEPARATOR, incorrect keywords produce syntax errors.

Didn't break the APPEND_SEPARATOR datatype validation

Didn't break METADATA fields validation

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

Background/details

Till now, "options" have been a concept in our code. Their definition isn't clear, but it essentially comes down to any capitalized keyword after the command name. For example STATS... >BY<, DISSECT... >APPEND_SEPARATOR<. You could think of them as roughly subcommands or substatements.

There was a hope that commands would be uniform enough that these "options" would deserve to be their own special first-class citizen. But they break conceptually...

For example APPEND_SEPARATOR is not a keyword with an expression after it... it is a variable APPEND_SEPARATOR=":"... or filtering in stats.... STATS AVG(bytes) >WHERE< .... so is WHERE an option now?

FORK will break this even further.

So, we are moving the architecture to allow for complexity and variance among the commands. Command-specific logic will have the final say in how autocomplete and validation work for anything with that command.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: ef1adac

Failed CI Steps

• FTR Configs #85

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
esql	189	188	-1
lists	421	420	-1
securitySolution	7087	7086	-1
unifiedSearch	349	348	-1
total			-4

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/esql-validation-autocomplete	174	166	-8

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
kbnUiSharedDeps-srcJs	3.6MB	3.6MB	-1.3KB

Unknown metric groups

API count

id	before	after	diff
@kbn/esql-validation-autocomplete	187	184	-3

Unreferenced deprecated APIs

id	before	after	diff
@kbn/esql-validation-autocomplete	2	1	-1

History

• 💔 Build #285940 failed feecfd9

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)

[stratoula]

This is a very nice cleanup 👏

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13990332729

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [ES|QL] Restructure validation code, remove command settings (#215056)

Manual backport

To create the backport manually run:

node scripts/backport --pr 215425

Questions ?

Please refer to the Backport tool documentation

[drewdaemon]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation