Skip to content

[DOCS][ESQL] Document _source metadata field #105237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leemthompo merged 4 commits into from
Feb 7, 2024
Merged

[DOCS][ESQL] Document _source metadata field #105237

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e894834
[DOCS][ESQL] Document _source metadata field
leemthompo Feb 7, 2024
d51c2e9
Minor copyedit to entire page
leemthompo Feb 7, 2024
f53d4bf
Reword per review
leemthompo Feb 7, 2024
07167e0
Fix Freudian typo
leemthompo Feb 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions docs/reference/esql/metadata-fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
<titleabbrev>Metadata fields</titleabbrev>
++++

{esql} can access <<mapping-fields, metadata fields>>. The currently
supported ones are:
{esql} can access <<mapping-fields, metadata fields>>. The following are currently
supported:

* <<mapping-index-field,`_index`>>: the index to which the document belongs.
* <<mapping-index-field,`_index`>>: the index the document belongs to.
The field is of the type <<keyword, keyword>>.

* <<mapping-id-field,`_id`>>: the source document's ID. The field is of the
Expand All @@ -17,20 +17,26 @@ supported ones are:
* `_version`: the source document's version. The field is of the type
<<number,long>>.

To enable the access to these fields, the <<esql-from,`FROM`>> source command needs
to be provided with a dedicated directive:
* <<mapping-source-field,`_source`>>: the original JSON document body
that was passed at index time (or a reconstructed version if
<<synthetic-source, synthetic `_source_`>> is enabled). The field is
loaded as a special `_source` type. This field is not supported by
functions.

To enable access to these fields, the <<esql-from,`FROM`>> source command
requires a dedicated directive:

[source,esql]
----
FROM index [METADATA _index, _id]
FROM index [METADATA _index, _id, _source]
Copy link
Copy Markdown
Contributor

@alex-spies alex-spies Feb 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that a syntax change is coming in #105221, which may require a follow-up PR.

----

Metadata fields are only available if the source of the data is an index.
Consequently, `FROM` is the only source commands that supports the `METADATA`
directive.

Once enabled, the fields are then available to subsequent processing commands, just
like the other index fields:
Once enabled, these fields will be available to subsequent processing commands, just
like other index fields:

[source.merge.styled,esql]
----
Expand All @@ -41,9 +47,9 @@ include::{esql-specs}/metadata-IT_tests_only.csv-spec[tag=multipleIndices]
include::{esql-specs}/metadata-IT_tests_only.csv-spec[tag=multipleIndices-result]
|===

Also, similar to the index fields, once an aggregation is performed, a
Similar to index fields, once an aggregation is performed, a
metadata field will no longer be accessible to subsequent commands, unless
used as grouping field:
used as a grouping field:

[source.merge.styled,esql]
----
Expand Down