[ES|QL] improve available column lists#233221
Conversation
stratoula
left a comment
There was a problem hiding this comment.
Inlinestats bug got fixed. Added 2 more comments but I am approving as ithere is no need to review again.
Wonderful cleanup 👏
| } | ||
|
|
||
| context?.fields.set('_fork', { | ||
| context?.columns.set('_fork', { |
| type: 'keyword', | ||
| }, | ||
| ]); | ||
| context?.columns.set(targetName, { |
There was a problem hiding this comment.
We no longer want to use custom column names like RERANK col0 = ? if we remove that probably we receive a validation error. I also need this input to remove the col0 suggestion from autocomplete
There was a problem hiding this comment.
I dont think we will with Drew's changes in this PR but maybe I am wrong. Drew keep me honest here
There was a problem hiding this comment.
Stratoula is right. As of this change we no longer check the left-side of an assignment against the column list. We never should have done it... it was a fluke that those columns were being included in the list even before they were even created in the query.
| { name: '@timestamp', type: 'date', userDefined: false }, | ||
| ]; | ||
|
|
||
| const queryString = `FROM a | STATS AVG(field1) BY buckets=BUCKET(@timestamp,50,?_tstart,?_tend)`; |
There was a problem hiding this comment.
we are testing stats instead of inlinestats. Why are all the previous tests gone?
There was a problem hiding this comment.
I guess inlinestats is just my bogeyman
There was a problem hiding this comment.
updated in 74432cb
Why are all the previous tests gone?
I thought this was adequate. Inlinestats now uses stats columns after under the hood which is tested extensively in stats. Not sure though. I could see an argument the other way, too if we were doing black-box testing.
💔 Build Failed
Failed CI StepsTest Failures
Metrics [docs]Module Count
Public APIs missing comments
Async chunks
Unknown metric groupsAPI count
References to deprecated APIs
History
|
## Summary Fixes the column name detection (bug introduced in #233221) **Before** <img width="711" height="203" alt="Screenshot 2025-09-11 at 9 00 34 AM" src="https://github.com/user-attachments/assets/285a97a5-4341-4f25-a969-46cd00cc669b" /> **After** <img width="703" height="176" alt="Screenshot 2025-09-11 at 9 42 13 AM" src="https://github.com/user-attachments/assets/69f923f6-7d15-4cf9-aeaa-027544240c2e" />
## Summary We want _one_ system that efficiently computes a list of available columns at any given command position in the query. This is crucial for both validation (column existence checks) and autocomplete (suggested columns). The chosen system is [`getColumnsByTypeHelper`](https://github.com/elastic/kibana/pull/233221/files#diff-1aaabc255965859d8acff49327980a6d72ce47858f252dab6257bf74719b971bR88) which relies on each command's `columnsAfter` method to report any changes to the column list made by that command. **Key changes** - `collectUserDefinedColumns` has been subsumed by command-specific `columnsAfter` methods - a unique column list is computed for each command during validation ([ref](https://github.com/elastic/kibana/pull/233221/files#diff-ff2bbabf0d966619b3e9d5449b4ae7ad70095f7f2ad0029c7e376d8460a25f08R142)). Prior to this change, we were collecting all columns available _anywhere_ in the query and using that list. - additional fields (such as from a lookup index or an enrichment policy) are added to the column list within the `columnsAfter` methods of the respective commands - user-defined columns and fields from ES are now merged into a single list on the context object called `columns`. This list contains objects defined by the `ESQLColumnData` interface. The two types of column are still differentiated per-object (specifically by the presence of `userDefined: true/false`) - the `getQueryForFields` function is now internal to `getColumnsByTypeHelper` - each `JOIN` and `ENRICH` command in a query generates its own network call. These calls happen serially. (This is potentially bit slower than what was happening before. There are some optimizations we could do, but we can monitor for feedback before investing.) ### User-facing improvements #### lookup index fields suggested <img width="849" height="272" alt="Screenshot 2025-09-05 at 11 13 26 AM" src="https://github.com/user-attachments/assets/ed0972ed-86d1-4fd8-a904-eade0c672509" /> #### enrichment policy fields suggested <img width="816" height="105" alt="Screenshot 2025-09-05 at 11 18 29 AM" src="https://github.com/user-attachments/assets/6fb9c4c7-c12d-4129-96c1-a1f5aca4b171" /> #### enrichment fields correctly validated <img width="516" height="81" alt="Screenshot 2025-09-05 at 11 21 14 AM" src="https://github.com/user-attachments/assets/41e22b90-1d57-4b0d-a2ad-0bac493ccfb7" /> _here `agent.keyword` is a field available in the source index, but it is not an enrichment field so it is not available in `WITH`_ #### catching columns used before they are defined <img width="270" height="88" alt="Screenshot 2025-09-05 at 11 12 14 AM" src="https://github.com/user-attachments/assets/4d8ecc02-5415-4f18-ab7f-154141e3630a" /> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Release note The ES|QL autocomplete feature has been extended so that fields from the lookup index are suggested after a complete `LOOKUP JOIN` command. The same enhancement has been applied to columns from an enrichment policy after a complete `ENRICH` command. The client-side validator's column detection has also been refined. It now reports use-before-defined errors and reliably knows the type of each column.
## Summary Fixes the column name detection (bug introduced in elastic#233221) **Before** <img width="711" height="203" alt="Screenshot 2025-09-11 at 9 00 34 AM" src="https://github.com/user-attachments/assets/285a97a5-4341-4f25-a969-46cd00cc669b" /> **After** <img width="703" height="176" alt="Screenshot 2025-09-11 at 9 42 13 AM" src="https://github.com/user-attachments/assets/69f923f6-7d15-4cf9-aeaa-027544240c2e" />
## Summary #233221 made the column suggestion incrementer look _behind_ the current command for user-defined columns. This created a regression when multiple columns are defined within a single command (minus `EVAL` which is handled specially). In the end, we decided that it isn't important to account for dropped columns when calculating the new column suggestion, so we just search the whole query for column names as we did before #233221. Without fix <img width="864" height="171" alt="Screenshot 2025-09-12 at 3 42 35 PM" src="https://github.com/user-attachments/assets/967187c7-0a8a-4680-aee3-3799d4af126e" /> With fix <img width="862" height="162" alt="Screenshot 2025-09-12 at 3 43 25 PM" src="https://github.com/user-attachments/assets/64ef7247-d91e-454e-8d55-b137cf97e610" /> Co-authored-by: Stratou <efstratia.kalafateli@elastic.co>
## Summary We want _one_ system that efficiently computes a list of available columns at any given command position in the query. This is crucial for both validation (column existence checks) and autocomplete (suggested columns). The chosen system is [`getColumnsByTypeHelper`](https://github.com/elastic/kibana/pull/233221/files#diff-1aaabc255965859d8acff49327980a6d72ce47858f252dab6257bf74719b971bR88) which relies on each command's `columnsAfter` method to report any changes to the column list made by that command. **Key changes** - `collectUserDefinedColumns` has been subsumed by command-specific `columnsAfter` methods - a unique column list is computed for each command during validation ([ref](https://github.com/elastic/kibana/pull/233221/files#diff-ff2bbabf0d966619b3e9d5449b4ae7ad70095f7f2ad0029c7e376d8460a25f08R142)). Prior to this change, we were collecting all columns available _anywhere_ in the query and using that list. - additional fields (such as from a lookup index or an enrichment policy) are added to the column list within the `columnsAfter` methods of the respective commands - user-defined columns and fields from ES are now merged into a single list on the context object called `columns`. This list contains objects defined by the `ESQLColumnData` interface. The two types of column are still differentiated per-object (specifically by the presence of `userDefined: true/false`) - the `getQueryForFields` function is now internal to `getColumnsByTypeHelper` - each `JOIN` and `ENRICH` command in a query generates its own network call. These calls happen serially. (This is potentially bit slower than what was happening before. There are some optimizations we could do, but we can monitor for feedback before investing.) ### User-facing improvements #### lookup index fields suggested <img width="849" height="272" alt="Screenshot 2025-09-05 at 11 13 26 AM" src="https://github.com/user-attachments/assets/ed0972ed-86d1-4fd8-a904-eade0c672509" /> #### enrichment policy fields suggested <img width="816" height="105" alt="Screenshot 2025-09-05 at 11 18 29 AM" src="https://github.com/user-attachments/assets/6fb9c4c7-c12d-4129-96c1-a1f5aca4b171" /> #### enrichment fields correctly validated <img width="516" height="81" alt="Screenshot 2025-09-05 at 11 21 14 AM" src="https://github.com/user-attachments/assets/41e22b90-1d57-4b0d-a2ad-0bac493ccfb7" /> _here `agent.keyword` is a field available in the source index, but it is not an enrichment field so it is not available in `WITH`_ #### catching columns used before they are defined <img width="270" height="88" alt="Screenshot 2025-09-05 at 11 12 14 AM" src="https://github.com/user-attachments/assets/4d8ecc02-5415-4f18-ab7f-154141e3630a" /> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Release note The ES|QL autocomplete feature has been extended so that fields from the lookup index are suggested after a complete `LOOKUP JOIN` command. The same enhancement has been applied to columns from an enrichment policy after a complete `ENRICH` command. The client-side validator's column detection has also been refined. It now reports use-before-defined errors and reliably knows the type of each column.
## Summary Fixes the column name detection (bug introduced in elastic#233221) **Before** <img width="711" height="203" alt="Screenshot 2025-09-11 at 9 00 34 AM" src="https://github.com/user-attachments/assets/285a97a5-4341-4f25-a969-46cd00cc669b" /> **After** <img width="703" height="176" alt="Screenshot 2025-09-11 at 9 42 13 AM" src="https://github.com/user-attachments/assets/69f923f6-7d15-4cf9-aeaa-027544240c2e" />
## Summary elastic#233221 made the column suggestion incrementer look _behind_ the current command for user-defined columns. This created a regression when multiple columns are defined within a single command (minus `EVAL` which is handled specially). In the end, we decided that it isn't important to account for dropped columns when calculating the new column suggestion, so we just search the whole query for column names as we did before elastic#233221. Without fix <img width="864" height="171" alt="Screenshot 2025-09-12 at 3 42 35 PM" src="https://github.com/user-attachments/assets/967187c7-0a8a-4680-aee3-3799d4af126e" /> With fix <img width="862" height="162" alt="Screenshot 2025-09-12 at 3 43 25 PM" src="https://github.com/user-attachments/assets/64ef7247-d91e-454e-8d55-b137cf97e610" /> Co-authored-by: Stratou <efstratia.kalafateli@elastic.co>
## Summary We want _one_ system that efficiently computes a list of available columns at any given command position in the query. This is crucial for both validation (column existence checks) and autocomplete (suggested columns). The chosen system is [`getColumnsByTypeHelper`](https://github.com/elastic/kibana/pull/233221/files#diff-1aaabc255965859d8acff49327980a6d72ce47858f252dab6257bf74719b971bR88) which relies on each command's `columnsAfter` method to report any changes to the column list made by that command. **Key changes** - `collectUserDefinedColumns` has been subsumed by command-specific `columnsAfter` methods - a unique column list is computed for each command during validation ([ref](https://github.com/elastic/kibana/pull/233221/files#diff-ff2bbabf0d966619b3e9d5449b4ae7ad70095f7f2ad0029c7e376d8460a25f08R142)). Prior to this change, we were collecting all columns available _anywhere_ in the query and using that list. - additional fields (such as from a lookup index or an enrichment policy) are added to the column list within the `columnsAfter` methods of the respective commands - user-defined columns and fields from ES are now merged into a single list on the context object called `columns`. This list contains objects defined by the `ESQLColumnData` interface. The two types of column are still differentiated per-object (specifically by the presence of `userDefined: true/false`) - the `getQueryForFields` function is now internal to `getColumnsByTypeHelper` - each `JOIN` and `ENRICH` command in a query generates its own network call. These calls happen serially. (This is potentially bit slower than what was happening before. There are some optimizations we could do, but we can monitor for feedback before investing.) ### User-facing improvements #### lookup index fields suggested <img width="849" height="272" alt="Screenshot 2025-09-05 at 11 13 26 AM" src="https://github.com/user-attachments/assets/ed0972ed-86d1-4fd8-a904-eade0c672509" /> #### enrichment policy fields suggested <img width="816" height="105" alt="Screenshot 2025-09-05 at 11 18 29 AM" src="https://github.com/user-attachments/assets/6fb9c4c7-c12d-4129-96c1-a1f5aca4b171" /> #### enrichment fields correctly validated <img width="516" height="81" alt="Screenshot 2025-09-05 at 11 21 14 AM" src="https://github.com/user-attachments/assets/41e22b90-1d57-4b0d-a2ad-0bac493ccfb7" /> _here `agent.keyword` is a field available in the source index, but it is not an enrichment field so it is not available in `WITH`_ #### catching columns used before they are defined <img width="270" height="88" alt="Screenshot 2025-09-05 at 11 12 14 AM" src="https://github.com/user-attachments/assets/4d8ecc02-5415-4f18-ab7f-154141e3630a" /> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Release note The ES|QL autocomplete feature has been extended so that fields from the lookup index are suggested after a complete `LOOKUP JOIN` command. The same enhancement has been applied to columns from an enrichment policy after a complete `ENRICH` command. The client-side validator's column detection has also been refined. It now reports use-before-defined errors and reliably knows the type of each column.
## Summary Fixes the column name detection (bug introduced in #233221) **Before** <img width="711" height="203" alt="Screenshot 2025-09-11 at 9 00 34 AM" src="https://github.com/user-attachments/assets/285a97a5-4341-4f25-a969-46cd00cc669b" /> **After** <img width="703" height="176" alt="Screenshot 2025-09-11 at 9 42 13 AM" src="https://github.com/user-attachments/assets/69f923f6-7d15-4cf9-aeaa-027544240c2e" />
## Summary #233221 made the column suggestion incrementer look _behind_ the current command for user-defined columns. This created a regression when multiple columns are defined within a single command (minus `EVAL` which is handled specially). In the end, we decided that it isn't important to account for dropped columns when calculating the new column suggestion, so we just search the whole query for column names as we did before #233221. Without fix <img width="864" height="171" alt="Screenshot 2025-09-12 at 3 42 35 PM" src="https://github.com/user-attachments/assets/967187c7-0a8a-4680-aee3-3799d4af126e" /> With fix <img width="862" height="162" alt="Screenshot 2025-09-12 at 3 43 25 PM" src="https://github.com/user-attachments/assets/64ef7247-d91e-454e-8d55-b137cf97e610" /> Co-authored-by: Stratou <efstratia.kalafateli@elastic.co>
## Summary elastic#233221 made the column suggestion incrementer look _behind_ the current command for user-defined columns. This created a regression when multiple columns are defined within a single command (minus `EVAL` which is handled specially). In the end, we decided that it isn't important to account for dropped columns when calculating the new column suggestion, so we just search the whole query for column names as we did before elastic#233221. Without fix <img width="864" height="171" alt="Screenshot 2025-09-12 at 3 42 35 PM" src="https://github.com/user-attachments/assets/967187c7-0a8a-4680-aee3-3799d4af126e" /> With fix <img width="862" height="162" alt="Screenshot 2025-09-12 at 3 43 25 PM" src="https://github.com/user-attachments/assets/64ef7247-d91e-454e-8d55-b137cf97e610" /> Co-authored-by: Stratou <efstratia.kalafateli@elastic.co>
Summary
We want one system that efficiently computes a list of available columns at any given command position in the query. This is crucial for both validation (column existence checks) and autocomplete (suggested columns).
The chosen system is
getColumnsByTypeHelperwhich relies on each command'scolumnsAftermethod to report any changes to the column list made by that command.Key changes
collectUserDefinedColumnshas been subsumed by command-specificcolumnsAftermethodscolumnsAftermethods of the respective commandscolumns. This list contains objects defined by theESQLColumnDatainterface. The two types of column are still differentiated per-object (specifically by the presence ofuserDefined: true/false)getQueryForFieldsfunction is now internal togetColumnsByTypeHelperJOINandENRICHcommand in a query generates its own network call. These calls happen serially. (This is potentially bit slower than what was happening before. There are some optimizations we could do, but we can monitor for feedback before investing.)User-facing improvements
lookup index fields suggested
enrichment policy fields suggested
enrichment fields correctly validated
here
agent.keywordis a field available in the source index, but it is not an enrichment field so it is not available inWITHcatching columns used before they are defined
Checklist
Release note
The ES|QL autocomplete feature has been extended so that fields from the lookup index are suggested after a complete
LOOKUP JOINcommand. The same enhancement has been applied to columns from an enrichment policy after a completeENRICHcommand.The client-side validator's column detection has also been refined. It now reports use-before-defined errors and reliably knows the type of each column.