[NL2ESQL] Fix missing LOOKUP JOIN ES|QL command in docs by qn895 · Pull Request #225382 · elastic/kibana

qn895 · 2025-06-25T23:24:33Z

Summary

Follow up of #224868. This PR fixes missing LOOKUP JOIN ES|QL command in docs due to the built-docs not having ES|QL in the file name.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

elasticmachine · 2025-06-25T23:26:12Z

Pinging @elastic/appex-ai-infra (Team:AI Infra)

elasticmachine · 2025-06-25T23:26:12Z

Pinging @elastic/ml-ui (:ml)

darnautov · 2025-06-26T10:11:46Z

x-pack/platform/plugins/shared/inference/scripts/load_esql_docs/extract_doc_entries.ts

      sourceFile: basename,
-      name: basename === 'esql.html' ? 'overview' : basename.substring(5, basename.length - 5),
-      content: getSimpleText($element),
+      name: basename === 'esql.html' ? 'overview' : basename.replace('esql-', ''),


Is this change intentional? We used to strip first 5 characters (esql-) and the last 5 characters (.html), so esql-my-page.html becomes my-page. With you change .html is preserved. Shall we use basename.replace(/^esql-/, '').replace(/\.html$/, '') instead?

Updated here fa9a5b2 (#225382)

darnautov · 2025-06-26T10:15:28Z

x-pack/platform/plugins/shared/inference/scripts/load_esql_docs/generate_doc.ts

  await Promise.all(
    pages.map(async (page) => {
      return limiter(async () => {
+        if (!page.sourceFile) return;


why do we need this check? looking at the pages, sourceFile is always defined.

pgayvallet

see comment.

pgayvallet · 2025-06-26T13:13:00Z

x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/esql_docs/esql-lookup-join.txt

+# LOOKUP JOIN
+
+The `LOOKUP JOIN` command combines data from a query results table with matching records from a specified lookup index. It adds fields from the lookup index as new columns to the results table based on matching values in the join field. This is particularly useful for enriching or correlating data across multiple indices, such as logs, IPs, user IDs, or hosts.
+
+## Syntax
+
+`LOOKUP JOIN <lookup_index> ON <field_name>`
+
+### Parameters
+
+#### lookup_index


We need to add an entry in https://github.com/elastic/kibana/blob/5b9691278165417d4cc12853f58de728aaeff011/x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/system_message.txt (otherwise the LLM won't even know the function exists), and we probably also want to add a couple examples too (in the system message I mean, bottom of the file are a bunch of examples), as LOOKUP JOIN is a very specific method. Otherwise I doubt the LLM will be able to think about using when it should.

Probably need to update the aliases file too (

kibana/x-pack/platform/plugins/shared/inference/server/tasks/nl_to_esql/doc_base/aliases.ts

Lines 12 to 27 in a0f5a7f

const aliases: Record<string, string[]> = {

STATS: ['STATS_BY', 'BY', 'STATS...BY', 'STATS ... BY'],

OPERATORS: ['LIKE', 'RLIKE', 'IN'],

};

const getAliasMap = () => {

return Object.entries(aliases).reduce<Record<string, string>>(

(aliasMap, [command, commandAliases]) => {

commandAliases.forEach((alias) => {

aliasMap[alias] = command;

});

return aliasMap;

},

{}

);

};

), (for "LOOKUP", "JOIN" as least)

Adding a couple tests to the evaluation suite would be amazing too.

Updated here fa9a5b2 (#225382)

…upjoin

elasticmachine · 2025-07-09T16:12:11Z

💚 Build Succeeded

Buildkite Build
Commit: fa9a5b2

Metrics [docs]

✅ unchanged

History

💚 Build #313021 succeeded 4933738
💔 Build #312874 failed 640a611

cc @qn895

alvarezmelissa87

Aside from dima's small comment LGTM ⚡

kibanamachine · 2025-07-10T08:47:52Z

Starting backport for target branches: 9.0, 9.1

https://github.com/elastic/kibana/actions/runs/16190487601

) ## Summary Follow up of elastic#224868. This PR fixes missing LOOKUP JOIN ES|QL command in docs due to the built-docs not having ES|QL in the file name. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... (cherry picked from commit f7c3fe0)

qn895 · 2025-07-10T08:50:08Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

) ## Summary Follow up of elastic#224868. This PR fixes missing LOOKUP JOIN ES|QL command in docs due to the built-docs not having ES|QL in the file name. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... (cherry picked from commit f7c3fe0)

kibanamachine · 2025-07-10T08:58:09Z

💔 Some backports could not be created

Status	Branch	Result
❌	9.0	Backport failed because of merge conflicts
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 225382

Questions ?

Please refer to the Backport tool documentation

…) (#227382) # Backport This will backport the following commits from `main` to `9.1`: - [[AI Infra] Fix missing LOOKUP JOIN ES|QL command in docs (#225382)](#225382)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Quynh Nguyen (Quinn) <43350163+qn895@users.noreply.github.com>

) (#227379) # Backport This will backport the following commits from `main` to `8.19`: - [[AI Infra] Fix missing LOOKUP JOIN ES|QL command in docs (#225382)](#225382)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

) ## Summary Follow up of elastic#224868. This PR fixes missing LOOKUP JOIN ES|QL command in docs due to the built-docs not having ES|QL in the file name. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...

Fix missing lookup join command

640a611

qn895 requested a review from a team as a code owner June 25, 2025 23:24

qn895 self-assigned this Jun 25, 2025

qn895 added :ml Team:AI Infra Platform AppEx AI Infrastructure Team t// v9.1.0 v9.0.4 labels Jun 25, 2025

qn895 added backport:version Backport to applied version labels release_note:skip Skip the PR/issue when compiling release notes labels Jun 25, 2025

peteharverson requested a review from darnautov June 26, 2025 08:30

Fix types

4933738

darnautov reviewed Jun 26, 2025

View reviewed changes

pgayvallet reviewed Jun 26, 2025

View reviewed changes

qn895 added 3 commits July 9, 2025 05:15

Update html regex, aliases, system message

00b4b50

Merge remote-tracking branch 'upstream/main' into ai-fix-missing-look…

d37c9f0

…upjoin

Update logic for look up join

fa9a5b2

qn895 requested review from alvarezmelissa87 and pgayvallet July 9, 2025 14:41

alvarezmelissa87 approved these changes Jul 9, 2025

View reviewed changes

qn895 merged commit f7c3fe0 into elastic:main Jul 10, 2025
13 checks passed

qn895 added the v8.19.0 label Jul 10, 2025

kibanamachine added the v9.2.0 label Jul 10, 2025

qn895 mentioned this pull request Jul 10, 2025

[8.19] [AI Infra] Fix missing LOOKUP JOIN ES|QL command in docs (#225382) #227379

Merged

kibanamachine mentioned this pull request Jul 10, 2025

[9.1] [AI Infra] Fix missing LOOKUP JOIN ES|QL command in docs (#225382) #227382

Merged

kibanamachine mentioned this pull request Jul 10, 2025

[Security Solution][Sourcerer] Display explore data view as managed #227230

Merged

kibanamachine mentioned this pull request Aug 12, 2025

[NL2ESQL] Fix tool going into endless loop for ES|QL LOOKUP JOIN #231217

Merged

10 tasks

qn895 changed the title ~~[AI Infra] Fix missing LOOKUP JOIN ES|QL command in docs~~ [NL2ESQL] Fix missing LOOKUP JOIN ES|QL command in docs Aug 14, 2025

	const aliases: Record<string, string[]> = {
	STATS: ['STATS_BY', 'BY', 'STATS...BY', 'STATS ... BY'],
	OPERATORS: ['LIKE', 'RLIKE', 'IN'],
	};

	const getAliasMap = () => {
	return Object.entries(aliases).reduce<Record<string, string>>(
	(aliasMap, [command, commandAliases]) => {
	commandAliases.forEach((alias) => {
	aliasMap[alias] = command;
	});
	return aliasMap;
	},
	{}
	);
	};

Comments

Conversation

qn895 commented Jun 25, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Identify risks

Uh oh!

elasticmachine commented Jun 25, 2025

Uh oh!

elasticmachine commented Jun 25, 2025

Uh oh!

darnautov Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

qn895 Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

darnautov Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

pgayvallet left a comment

Choose a reason for hiding this comment

Uh oh!

pgayvallet Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qn895 Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 9, 2025

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kibanamachine commented Jul 10, 2025

Uh oh!

qn895 commented Jul 10, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Jul 10, 2025

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

qn895 commented Jun 25, 2025 •

edited by kibanamachine

Loading

pgayvallet Jun 26, 2025 •

edited

Loading