ESQL: Generative testing with full text functions#142961

Merged

astefan merged 18 commits intoelastic:mainfrom

astefan:generative_full-text-functions

Mar 2, 2026

Contributor

astefan commented Feb 24, 2026

This adds support for generative testing for full-text search functions. They are tricky because they have many restrictions. Many of them are already caught and handled in this PR's code.
I've also refactored the functions generation code since the FunctionGenerator became harder to grasp.
Co-authored with AI-tools help.

astefan added 5 commits

February 24, 2026 09:35


          Full-text functions support

2f34abb


          Merge branch 'main' of https://github.com/elastic/elasticsearch into …

7c4516d

…generative_full-text-functions


          more updates

4eaa690


          Refactor and cleanup

e6d91d0


          Refactor the functions generator

a5a8d70

astefan requested a review from luigidellaquila

February 24, 2026 15:20

astefan added >test :Analytics/ES|QL v9.4.0 labels

elasticsearchmachine added the Team:Analytics label

Collaborator

elasticsearchmachine commented Feb 24, 2026

Pinging @elastic/es-analytical-engine (Team:Analytics)

astefan commented

View reviewed changes

...src/main/java/org/elasticsearch/xpack/esql/generator/function/FullTextFunctionGenerator.java Show resolved Hide resolved

astefan commented

View reviewed changes

...src/main/java/org/elasticsearch/xpack/esql/generator/function/FullTextFunctionGenerator.java Show resolved Hide resolved


          [CI] Auto commit changes from spotless

5a47151

luigidellaquila reviewed

View reviewed changes

Contributor

luigidellaquila left a comment

Thanks @astefan, overall it looks pretty good.
I left a first round of comments.

...server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java Outdated

+                      String errorMessage,
+                      String query,
+                      List<CommandGenerator.CommandDescription> commandsForFieldOriginTracing,
+                      List<CommandGenerator.CommandDescription> commandsForPlacementChecks

Contributor

luigidellaquila Feb 24, 2026

commandsForPlacementChecks is not used, is it?

...server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java Outdated

+                  private record FailureContext(
+                      String errorMessage,
+                      String query,
+                      List<CommandGenerator.CommandDescription> commandsForFieldOriginTracing,

Contributor

luigidellaquila Feb 24, 2026

I wonder if it makes sense to just call it previousCommands. Maybe they will be useful also for other purposes, apart from origin tracing

...server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java Outdated

                   ) {
+                      List<CommandGenerator.CommandDescription> commands = new ArrayList<>(previousCommands.size() + 1);
+                      commands.addAll(previousCommands);
+                      commands.add(commandDescription);

Contributor

luigidellaquila Feb 24, 2026

Maybe I'm overlooking something, but I think we don't need this.

...server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java Show resolved Hide resolved

...src/main/java/org/elasticsearch/xpack/esql/generator/function/FullTextFunctionGenerator.java Show resolved Hide resolved

.../qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/generator/FunctionGenerator.java Show resolved Hide resolved

...server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java Show resolved Hide resolved

astefan added 2 commits

February 25, 2026 14:54


          Address reviews

179d445


          Merge branch 'main' of https://github.com/elastic/elasticsearch into …

75335df

…generative_full-text-functions

astefan requested a review from luigidellaquila

February 25, 2026 12:55

luigidellaquila reviewed

View reviewed changes

...src/main/java/org/elasticsearch/xpack/esql/generator/function/FullTextFunctionGenerator.java Show resolved Hide resolved

astefan and others added 5 commits

February 25, 2026 17:47


          More

df90236


          Merge branch 'main' of https://github.com/elastic/elasticsearch into …

d1ad626

…generative_full-text-functions


          Unmute recently muted test and adjust the error message handling

bc2041c

for unmapped_fields


          Merge branch 'main' of https://github.com/elastic/elasticsearch into …

0e828aa

…generative_full-text-functions


          [CI] Auto commit changes from spotless

eace550

astefan requested a review from luigidellaquila

February 25, 2026 16:57

astefan and others added 2 commits

February 25, 2026 21:51


          Add one more allowed error related to full text search functions

d8bcdef


          Merge branch 'main' into generative_full-text-functions

5fe6622

luigidellaquila approved these changes

View reviewed changes

Contributor

luigidellaquila left a comment

Thanks @astefan, the changes look correct and super useful IMHO, so LGTM.

The CI is still red, but I'm not sure the failure is related to these changes.
The failing query has a multi_match query in a FORK branch, maybe this is related. Or maybe FORK is just too expensive by itself for the small CI instance.

Anyway, the query looks correct, if there is a bug it's likely not in this PR.

Contributor Author

astefan commented Feb 26, 2026

@luigidellaquila unfortunately one of the failures is a true AssertionError that our own code (not the test infra in generative testing) throws. See #143121

Contributor Author

astefan commented Mar 2, 2026

I have opened #143249 to address the AssertionError encountered earlier.

astefan and others added 3 commits

March 2, 2026 15:50


          Merge branch 'main' of https://github.com/elastic/elasticsearch into …

86b1151

…generative_full-text-functions


          update

f98ecb8


          [CI] Auto commit changes from spotless

d6128ac

astefan merged commit d998c33 into elastic:main

35 checks passed

astefan deleted the generative_full-text-functions branch

March 2, 2026 16:09

astefan mentioned this pull request

[CI] GenerativeIT test failing #143023

Closed

szybia added a commit to szybia/elasticsearch that referenced this pull request


          Merge remote-tracking branch 'upstream/main' into get-reindex-with-re…

90ebd93

…locations

* upstream/main: (94 commits)
  Mute org.elasticsearch.xpack.esql.qa.mixed.EsqlClientYamlIT test {p0=esql/40_tsdb/TS Command grouping on text field} elastic#142544
  Mute org.elasticsearch.index.store.StoreDirectoryMetricsIT testDirectoryMetrics elastic#143419
  Mute org.elasticsearch.xpack.esql.qa.multi_node.GenerativeIT test elastic#143023
  TS_INFO information retrieval command (elastic#142721)
  ESQL: External source parallel execution and distribution (elastic#143349)
  Mute org.elasticsearch.index.mapper.blockloader.FlattenedFieldRootBlockLoaderTests testBlockLoaderForFieldInObject {preference=Params[syntheticSource=false, preference=DOC_VALUES]} elastic#143414
  Mute org.elasticsearch.index.mapper.blockloader.FlattenedFieldRootBlockLoaderTests testBlockLoaderForFieldInObject {preference=Params[syntheticSource=false, preference=NONE]} elastic#143413
  Mute org.elasticsearch.index.mapper.blockloader.FlattenedFieldRootBlockLoaderTests testBlockLoaderForFieldInObject {preference=Params[syntheticSource=false, preference=STORED]} elastic#143412
  Removing ingest random sampling (elastic#143289)
  Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeIT test elastic#143023
  [Transform] Clean up internal tests (elastic#143246)
  Skip time series field type merge for non-TS agg queries (elastic#143262)
  Enable zero-copy SIMD vector scoring on searchable snapshots (frozen tier) (elastic#141718)
  Mute org.elasticsearch.xpack.search.CrossClusterAsyncSearchIT testCancelViaExpirationOnRemoteResultsWithMinimizeRoundtrips elastic#143407
  Fix MemorySegmentUtilsTests (elastic#143391)
  Unmute testWorkflowsRestrictionAllowsAccess (elastic#143308)
  Cancel async query on expiry (elastic#143016)
  ESQL: Finish migrating error testing (elastic#143322)
  Reduce LuceneOperator.Status memory consumption with large QueryDSL queries (elastic#143175)
  ESQL: Generative testing with full text functions (elastic#142961)
  ...

tballison pushed a commit to tballison/elasticsearch that referenced this pull request


          ESQL: Generative testing with full text functions (elastic#142961)

c792df8

This was referenced Mar 6, 2026

[ML] Wait for cluster state in test #143767

Merged

[Transform] Disable PIT for CPS #143876

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL Team:Analytics >test v9.4.0