diff --git a/docs/reference/mapping.asciidoc b/docs/reference/mapping.asciidoc index ebea396cd279e..af79ceea80c74 100644 --- a/docs/reference/mapping.asciidoc +++ b/docs/reference/mapping.asciidoc @@ -5,30 +5,61 @@ -- Mapping is the process of defining how a document, and the fields it contains, -are stored and indexed. For instance, use mappings to define: +are stored and indexed. + +Each document is a collection of fields, which each have their own +<>. When mapping your data, you create a mapping +definition, which contains a list of fields that are pertinent to the document. +A mapping definition also includes <>, like the +`_source` field, which customize how a document's associated metadata is +handled. + +Use _dynamic mapping_ and _explicit mapping_ to define your data. Each method +provides different benefits based on where you are in your data journey. For +example, explicitly map fields where you don't want to use the defaults, or to +gain greater control over which fields are created. You can then allow {es} to +add other fields dynamically. + +NOTE: Before 7.0.0, the mapping definition included a type name. +{es} 7.0.0 and later no longer accept a _default_ mapping. See <>. + +.Experiment with mapping options +**** +<> to +experiment with different mapping options, and also fix mistakes in your index +mapping values by overriding values in the mapping during the search request. +**** -* which string fields should be treated as full text fields. -* which fields contain numbers, dates, or geolocations. -* the <> of date values. -* custom rules to control the mapping for - <>. - -A mapping definition includes metadata fields and fields: - -<>:: - -Metadata fields are used to customize how a document's associated metadata is -treated. Examples of metadata fields include the document's -<>, <>, and -<> fields. +[discrete] +[[mapping-dynamic]] +== Dynamic mapping +<> allows you to experiment with +and explore data when you’re just getting started. {es} adds new fields +automatically, just by indexing a document. You can add fields to the top-level +mapping, and to inner <> and <> fields. -<>:: +Use <> to define custom mappings that are +applied to dynamically added fields based on the matching condition. You can +<> +so that new fields are automatically added to the index mapping as runtime +fields. -A mapping contains a list of fields or `properties` pertinent to the -document. Each field has its own <>. +[discrete] +[[mapping-explicit]] +== Explicit mapping +<> allows you to precisely choose how to +define the mapping definition, such as: + +* Which string fields should be treated as full text fields. +* Which fields contain numbers, dates, or geolocations. +* The <> of date values. +* Custom rules to control the mapping for + <>. -NOTE: Before 7.0.0, the 'mappings' definition used to include a type name. -For more details, please see <>. +<> allows you to make +schema changes without reindexing. You can use runtime fields in conjunction +with indexed fields to balance resource usage and performance. Your index will +be smaller, but with slower search performance. [discrete] [[mapping-limit-settings]] @@ -45,190 +76,20 @@ Use the <> to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. -[discrete] -[[runtime-fields]] -== Runtime fields -Typically, you index data into {es} to promote faster search. However, indexing -can be slow and requires more disk space, and you have to reindex your data to -add fields to existing documents. - -<> are not indexed, which saves disk space and makes -data ingest faster. You can add runtime fields to existing documents without -reindexing your data and calculate field values dynamically at search time. - -[discrete] -[[dynamic-mapping-intro]] -== Dynamic mapping - -Fields and mapping types do not need to be defined before being used. Thanks -to _dynamic mapping_, new field names will be added automatically, just by -indexing a document. New fields can be added both to the top-level mapping -type, and to inner <> and <> fields. - -The <> rules can be configured to customise -the mapping that is used for new fields. - -[discrete] -== Explicit mappings - -You know more about your data than Elasticsearch can guess, so while dynamic -mapping can be useful to get started, at some point you will want to specify -your own explicit mappings. - -You can create field mappings when you <> and -<>. - -[discrete] -[[create-mapping]] -=== Create an index with an explicit mapping - -You can use the <> API to create a new index -with an explicit mapping. - -[source,console] ----- -PUT /my-index-000001 -{ - "mappings": { - "properties": { - "age": { "type": "integer" }, <1> - "email": { "type": "keyword" }, <2> - "name": { "type": "text" } <3> - } - } -} ----- - -<1> Creates `age`, an <> field -<2> Creates `email`, a <> field -<3> Creates `name`, a <> field - -[discrete] -[[add-field-mapping]] -== Add a field to an existing mapping - -You can use the <> API to add one or more new -fields to an existing index. - -The following example adds `employee-id`, a `keyword` field with an -<> mapping parameter value of `false`. This means values -for the `employee-id` field are stored but not indexed or available for search. - -[source,console] ----- -PUT /my-index-000001/_mapping -{ - "properties": { - "employee-id": { - "type": "keyword", - "index": false - } - } -} ----- -// TEST[continued] - -[discrete] -[[update-mapping]] -=== Update the mapping of a field - -include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=change-field-mapping] - -include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=rename-field] - -[discrete] -[[view-mapping]] -== View the mapping of an index - -You can use the <> API to view the mapping of -an existing index. - -[source,console] ----- -GET /my-index-000001/_mapping ----- -// TEST[continued] - -The API returns the following response: - -[source,console-result] ----- -{ - "my-index-000001" : { - "mappings" : { - "properties" : { - "age" : { - "type" : "integer" - }, - "email" : { - "type" : "keyword" - }, - "employee-id" : { - "type" : "keyword", - "index" : false - }, - "name" : { - "type" : "text" - } - } - } - } -} ----- - - -[discrete] -[[view-field-mapping]] -== View the mapping of specific fields - -If you only want to view the mapping of one or more specific fields, you can use -the <> API. - -This is useful if you don't need the complete mapping of an index or your index -contains a large number of fields. - -The following request retrieves the mapping for the `employee-id` field. - -[source,console] ----- -GET /my-index-000001/_mapping/field/employee-id ----- -// TEST[continued] - -The API returns the following response: - -[source,console-result] ----- -{ - "my-index-000001" : { - "mappings" : { - "employee-id" : { - "full_name" : "employee-id", - "mapping" : { - "employee-id" : { - "type" : "keyword", - "index" : false - } - } - } - } - } -} - ----- - -- -include::mapping/removal_of_types.asciidoc[] - -include::mapping/mapping-settings-limit.asciidoc[] +include::mapping/dynamic-mapping.asciidoc[] -include::mapping/types.asciidoc[] +include::mapping/explicit-mapping.asciidoc[] include::mapping/runtime.asciidoc[] +include::mapping/types.asciidoc[] + include::mapping/fields.asciidoc[] include::mapping/params.asciidoc[] -include::mapping/dynamic-mapping.asciidoc[] +include::mapping/mapping-settings-limit.asciidoc[] + +include::mapping/removal_of_types.asciidoc[] diff --git a/docs/reference/mapping/dynamic-mapping.asciidoc b/docs/reference/mapping/dynamic-mapping.asciidoc index e62f243b669eb..875e879495864 100644 --- a/docs/reference/mapping/dynamic-mapping.asciidoc +++ b/docs/reference/mapping/dynamic-mapping.asciidoc @@ -1,11 +1,11 @@ [[dynamic-mapping]] -== Dynamic Mapping +== Dynamic mapping -One of the most important features of Elasticsearch is that it tries to get +One of the most important features of {es} is that it tries to get out of your way and let you start exploring your data as quickly as possible. To index a document, you don't have to first create an index, define a mapping type, and define your fields -- you can just index a document and the index, -type, and fields will spring to life automatically: +type, and fields will display automatically: [source,console] -------------------------------------------------- @@ -17,7 +17,7 @@ PUT data/_doc/1 <1> called `count` with data type `long`. The automatic detection and addition of new fields is called -_dynamic mapping_. The dynamic mapping rules can be customised to suit your +_dynamic mapping_. The dynamic mapping rules can be customized to suit your purposes with: <>:: @@ -35,4 +35,3 @@ automatically or explicitly. include::dynamic/field-mapping.asciidoc[] include::dynamic/templates.asciidoc[] - diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index bf5a9165d6f28..ae7e0e0b7b0cc 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -2,36 +2,45 @@ === Dynamic field mapping When {es} detects a new field in a document, it _dynamically_ adds the field to -the type mapping. The <> parameter controls this behavior. +the type mapping by default. The <> parameter controls this behavior. -You can disable dynamic mapping, both at the document and at the -<> level. Setting the `dynamic` parameter to -`false` ignores new fields, and `strict` rejects the document if {es} -encounters an unknown field. - -When dynamic field mapping is enabled, {es} uses the following rules to +You can explicitly instruct {es} to dynamically create fields based on incoming +documents by setting the `dynamic` parameter to `true` or `runtime`. When +dynamic field mapping is enabled, {es} uses the rules in the following table to determine how to map data types for each field. -NOTE: These are the only <> that {es} detects -dynamically. All other data types must be mapped explicitly. +NOTE: The field data types in the following table are the only +<> that {es} detects dynamically. You must +explicitly map all other data types. +[[dynamic-field-mapping-types]] [cols="3"] |=== h| JSON data type h| `dynamic:true` h| `dynamic:runtime` |`null` 2*| No field added |`true` or `false` 2*| `boolean` - |floating point number | `float` | `double` + |`double` | `float` | `double` |`integer` 2*| `long` |`object`<> 2*| `object` |`array` 2*| Depends on the first non-`null` value in the array |`string` that passes <> 2*| `date` - |`string` that passes <> | `double` or `long` | `double` + |`string` that passes <> | `float` or `long` | `double` or `long` |`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword` 3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | | |=== -Besides the following options, you can customize dynamic field mapping rules -with <>. +You can disable dynamic mapping, both at the document and at the +<> level. Setting the `dynamic` parameter to +`false` ignores new fields, and `strict` rejects the document if {es} +encounters an unknown field. + +TIP: Use the <> to update the `dynamic` +setting on existing fields. + +You can customize dynamic field mapping rules for +<> and <>. +To define custom mappings rules that you can apply to additional dynamic +fields, use <>. [[date-detection]] ==== Date detection @@ -83,9 +92,9 @@ PUT my-index-000001/_doc/1 <1> <1> The `create_date` field has been added as a <> field. -===== Customising detected date formats +===== Customizing detected date formats -Alternatively, the `dynamic_date_formats` can be customised to support your +Alternatively, the `dynamic_date_formats` can be customized to support your own <>: [source,console] diff --git a/docs/reference/mapping/dynamic/templates.asciidoc b/docs/reference/mapping/dynamic/templates.asciidoc index 7f13007bc61f1..ce13b2a02bd83 100644 --- a/docs/reference/mapping/dynamic/templates.asciidoc +++ b/docs/reference/mapping/dynamic/templates.asciidoc @@ -2,18 +2,20 @@ === Dynamic templates Dynamic templates allow you to define custom mappings that can be applied to -dynamically added fields based on: +dynamically added fields based on the matching condition: -* the <> detected by Elasticsearch, with <>. -* the name of the field, with <> or <>. -* the full dotted path to the field, with <>. +* <> operates on the +<> that {es} detects +* <> use a pattern to match on the field +name +* <> operate on the full +dotted path to the field -The original field name `{name}` and the detected data type -`{dynamic_type}` <> can be used in -the mapping specification as placeholders. +Use the `{name}` and `{dynamic_type}` <> +in the mapping specification as placeholders. IMPORTANT: Dynamic field mappings are only added when a field contains a -concrete value -- not `null` or an empty array. This means that if the +concrete value -- not `null` or an empty array. If the `null_value` option is used in a `dynamic_template`, it will only be applied after the first document with a concrete value for the field has been indexed. @@ -37,6 +39,8 @@ Dynamic templates are specified as an array of named objects: <2> The match conditions can include any of : `match_mapping_type`, `match`, `match_pattern`, `unmatch`, `path_match`, `path_unmatch`. <3> The mapping that the matched field should use. +[[dynamic-templates-validation]] +==== Validating dynamic templates If a provided mapping contains an invalid mapping snippet, a validation error is returned. Validation occurs when applying the dynamic template at index time, and, in most cases, when the dynamic template is updated. Providing an invalid mapping @@ -61,12 +65,12 @@ reordered or deleted after they were initially added. [[match-mapping-type]] ==== `match_mapping_type` -The `match_mapping_type` is the data type detected by the JSON parser. Since +The `match_mapping_type` is the data type detected by the JSON parser. Because JSON doesn't distinguish a `long` from an `integer` or a `double` from -a `float`, it will always choose the wider data type, i.e. `long` for integers +a `float`, it always chooses the wider data type such as `long` for integers and `double` for floating-point numbers. -The following data types may be automatically detected: +{es} automatically detects the following data types: - `boolean` when `true` or `false` are encountered. - `date` when <> is enabled and a string matching @@ -76,7 +80,7 @@ The following data types may be automatically detected: - `object` for objects, also called hashes. - `string` for character strings. -`*` may also be used in order to match all data types. +Use a wildcard (`*`) to match all data types. For example, if we wanted to map all integer fields as `integer` instead of `long`, and all `string` fields as both `text` and `keyword`, we @@ -124,6 +128,32 @@ PUT my-index-000001/_doc/1 <1> The `my_integer` field is mapped as an `integer`. <2> The `my_string` field is mapped as a `text`, with a `keyword` <>. +[[match-mapping-runtime-fields]] +===== Mapping runtime fields in a dynamic template +You can also map runtime fields in a dynamic template. For example, the +following request adds a dynamic template named `dayOfWeek` that maps all +incoming `string` fields as `keyword` fields in the `runtime` section of the +mapping. Although the `runtime` definition is blank, new `string` fields will +be mapped as `keyword` runtime fields based on the <> that {es} uses for adding field types to the +mapping. Any `string` that doesn't pass date detection or numeric detection is +automatically mapped as a `keyword` when `dynamic` is set to `runtime`. + +[source,console] +---- +PUT my-index-000001/ +{ + "mappings": { + "dynamic_templates": [ + { + "dayOfWeek": { + "match_mapping_type": "string", + "runtime": {} + } + } + ] + } +} +---- [[match-unmatch]] ==== `match` and `unmatch` @@ -131,11 +161,21 @@ PUT my-index-000001/_doc/1 The `match` parameter uses a pattern to match on the field name, while `unmatch` uses a pattern to exclude fields matched by `match`. +The `match_pattern` parameter adjusts the behavior of the `match` parameter +to support full Java regular expressions matching on the field name +instead of simple wildcards. For example: + +[source,js] +-------------------------------------------------- + "match_pattern": "regex", + "match": "^profit_\d+$" +-------------------------------------------------- +// NOTCONSOLE + The following example matches all `string` fields whose name starts with `long_` (except for those which end with `_text`) and maps them as `long` fields: - [source,console] -------------------------------------------------- PUT my-index-000001 @@ -166,20 +206,6 @@ PUT my-index-000001/_doc/1 <1> The `long_num` field is mapped as a `long`. <2> The `long_text` field uses the default `string` mapping. -[[match-pattern]] -==== `match_pattern` - -The `match_pattern` parameter adjusts the behavior of the `match` parameter -such that it supports full Java regular expression matching on the field name -instead of simple wildcards, for instance: - -[source,js] --------------------------------------------------- - "match_pattern": "regex", - "match": "^profit_\d+$" --------------------------------------------------- -// NOTCONSOLE - [[path-match-unmatch]] ==== `path_match` and `path_unmatch` @@ -243,7 +269,7 @@ PUT my-index-000001/_doc/2 // TEST[catch:bad_request] [[template-variables]] -==== `{name}` and `{dynamic_type}` +==== Template variables The `{name}` and `{dynamic_type}` placeholders are replaced in the `mapping` with the field name and detected dynamic type. The following example sets all @@ -290,7 +316,7 @@ PUT my-index-000001/_doc/1 <2> The `count` field is mapped as a `long` field with `doc_values` disabled. [[template-examples]] -==== Template examples +==== Dynamic template examples Here are some examples of potentially useful dynamic templates: diff --git a/docs/reference/mapping/explicit-mapping.asciidoc b/docs/reference/mapping/explicit-mapping.asciidoc new file mode 100644 index 0000000000000..4ef055d5699ce --- /dev/null +++ b/docs/reference/mapping/explicit-mapping.asciidoc @@ -0,0 +1,148 @@ + +[[explicit-mapping]] +== Explicit mapping + +You know more about your data than {es} can guess, so while dynamic +mapping can be useful to get started, at some point you will want to specify +your own explicit mappings. + +You can create field mappings when you <> and +<>. + +[discrete] +[[create-mapping]] +=== Create an index with an explicit mapping + +You can use the <> API to create a new index +with an explicit mapping. + +[source,console] +---- +PUT /my-index-000001 +{ + "mappings": { + "properties": { + "age": { "type": "integer" }, <1> + "email": { "type": "keyword" }, <2> + "name": { "type": "text" } <3> + } + } +} +---- + +<1> Creates `age`, an <> field +<2> Creates `email`, a <> field +<3> Creates `name`, a <> field + +[discrete] +[[add-field-mapping]] +=== Add a field to an existing mapping + +You can use the <> API to add one or more new +fields to an existing index. + +The following example adds `employee-id`, a `keyword` field with an +<> mapping parameter value of `false`. This means values +for the `employee-id` field are stored but not indexed or available for search. + +[source,console] +---- +PUT /my-index-000001/_mapping +{ + "properties": { + "employee-id": { + "type": "keyword", + "index": false + } + } +} +---- +// TEST[continued] + +[discrete] +[[update-mapping]] +=== Update the mapping of a field + +include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=change-field-mapping] + +include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=rename-field] + +[discrete] +[[view-mapping]] +=== View the mapping of an index + +You can use the <> API to view the mapping of +an existing index. + +[source,console] +---- +GET /my-index-000001/_mapping +---- +// TEST[continued] + +The API returns the following response: + +[source,console-result] +---- +{ + "my-index-000001" : { + "mappings" : { + "properties" : { + "age" : { + "type" : "integer" + }, + "email" : { + "type" : "keyword" + }, + "employee-id" : { + "type" : "keyword", + "index" : false + }, + "name" : { + "type" : "text" + } + } + } + } +} +---- + +[discrete] +[[view-field-mapping]] +=== View the mapping of specific fields + +If you only want to view the mapping of one or more specific fields, you can use +the <> API. + +This is useful if you don't need the complete mapping of an index or your index +contains a large number of fields. + +The following request retrieves the mapping for the `employee-id` field. + +[source,console] +---- +GET /my-index-000001/_mapping/field/employee-id +---- +// TEST[continued] + +The API returns the following response: + +[source,console-result] +---- +{ + "my-index-000001" : { + "mappings" : { + "employee-id" : { + "full_name" : "employee-id", + "mapping" : { + "employee-id" : { + "type" : "keyword", + "index" : false + } + } + } + } + } +} + +---- diff --git a/docs/reference/mapping/params/dynamic.asciidoc b/docs/reference/mapping/params/dynamic.asciidoc index 6ded8e4f7809c..1bb845885c43a 100644 --- a/docs/reference/mapping/params/dynamic.asciidoc +++ b/docs/reference/mapping/params/dynamic.asciidoc @@ -39,9 +39,6 @@ PUT my-index-000001/_doc/2 GET my-index-000001/_mapping ---- -TIP: Use the <> to update the `dynamic` -setting on existing fields. - [[dynamic-inner-objects]] ==== Setting `dynamic` on inner objects <> inherit the `dynamic` setting from their parent diff --git a/docs/reference/mapping/removal_of_types.asciidoc b/docs/reference/mapping/removal_of_types.asciidoc index ca028355de6f7..acb521bd79909 100644 --- a/docs/reference/mapping/removal_of_types.asciidoc +++ b/docs/reference/mapping/removal_of_types.asciidoc @@ -3,4 +3,4 @@ Elasticsearch 8.0.0 no longer supports mapping types. For details on how to migrate your clusters away from mapping types, see the -{ref-7x}/removal-of-types.html[removal of types] documentation for the 7.x release +{ref-7x}/removal-of-types.html[removal of types] documentation for the 7.x release. diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index dd1e2404e7a08..9fcfaf4176f66 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -1,9 +1,12 @@ [[runtime]] == Runtime fields -Typically, you index data into {es} to promote faster search. However, indexing -can be slow and requires more disk space, and you have to reindex your data to -add fields to existing documents. With _runtime fields_, you can add -fields to documents already indexed to {es} without reindexing your data. +A _runtime field_ is a field that is evaluated at query time. Runtime fields +enable you to: + +* Add fields to existing documents without reindexing your data +* Start working with your data without understanding how it’s structured +* Override the value returned from an indexed field at query time +* Define fields for a specific use without modifying the underlying schema You access runtime fields from the search API like any other field, and {es} sees runtime fields no differently. You can define runtime fields in the @@ -11,6 +14,11 @@ sees runtime fields no differently. You can define runtime fields in the <>. Your choice, which is part of the inherent flexibility of runtime fields. +Runtime fields are useful when working with log data +(see <>), especially when you're unsure about the +data structure. Your search speed decreases, but your index size is much +smaller and you can more quickly process logs without having to index them. + [discrete] [[runtime-benefits]] === Benefits @@ -34,25 +42,6 @@ front, and can use runtime fields to amend the mapping at any time. Using runtime fields allows for a smaller index and faster ingest time, which combined use less resources and reduce your operating costs. -[discrete] -[[runtime-use-cases]] -=== Use cases -Runtime fields are useful when working with log data -(see <>), especially when you're unsure about the -data structure. Your search speed decreases, but your index size is much -smaller and you can more quickly process logs without having to index them. - -Runtime fields are especially useful in the following contexts: - -* Adding fields to documents that are already indexed without having to reindex -data -* Immediately begin working on a new data stream without fully understanding -the data it contains -* Shadowing an indexed field with a runtime field to fix a mistake after -indexing documents -* Defining fields that are only relevant for a particular context (such as a -visualization in {kib}) without influencing the underlying schema - [discrete] [[runtime-compromises]] === Compromises @@ -80,39 +69,29 @@ to `false`, expensive queries are not allowed and {es} will reject any queries against runtime fields. [[runtime-mapping-fields]] -=== Mapping a runtime field +=== Map a runtime field You map runtime fields by adding a `runtime` section under the mapping definition and defining <>. This script has access to the entire context of a document, including the original `_source` and any mapped -fields plus their values. At search time, the script runs and generates values +fields plus their values. At query time, the script runs and generates values for each scripted field that is required for the query. -NOTE: You can define a runtime field in the mapping definition without a -script. At search time, {es} looks in `_source` for a field with the same name -and returns a value if one exists. If a field with the same name doesn’t -exist, the response doesn't include any values for that runtime field. - -If <> is enabled where the -`dynamic` parameter is set to `runtime`, new fields are automatically added to -the index mapping as runtime fields. - -Runtime fields are similar to the <> parameter -of the `_search` request, but also make the script results available anywhere -in a search request. - -The script in the following request extracts the day of the week from the -`@timestamp` field, which is defined as a `date` type: +When defining a Painless script to use with runtime fields, you must include +`emit` to emit calculated values. For example, the script in the following +request extracts the day of the week from the `@timestamp` field, which is +defined as a `date` type. The script calculates the day of the week based on +the value of `timestamp`, and uses `emit` to return the calculated value. [source,console] ---- -PUT /my-index +PUT my-index/ { "mappings": { - "runtime": { <1> + "runtime": { "day_of_week": { - "type": "keyword", <2> - "script": { <3> + "type": "keyword", + "script": { "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" } } @@ -124,11 +103,6 @@ PUT /my-index } ---- -<1> Runtime fields are defined in the `runtime` section of the mapping -definition. -<2> Each runtime field has its own field type, just like any other field. -<3> The script defines the evaluation to calculate at search time. - The `runtime` section can be any of these data types: * `boolean` @@ -142,6 +116,25 @@ The `runtime` section can be any of these data types: Runtime fields with a `type` of `date` can accept the <> parameter exactly as the `date` field type. +If <> is enabled where the +`dynamic` parameter is set to `runtime`, new fields are automatically added to +the index mapping as runtime fields: + +[source,console] +---- +PUT my-index +{ + "mappings": { + "dynamic": "runtime", + "properties": { + "timestamp": { + "type": "date" + } + } + } +} +---- + [[runtime-updating-scripts]] .Updating runtime scripts **** @@ -158,7 +151,7 @@ to `boolean`. **** [[runtime-search-request]] -=== Defining runtime fields in a search request +=== Define runtime fields in a search request You can specify a `runtime_mappings` section in a search request to create runtime fields that exist only as part of the query. You specify a script as part of the `runtime_mappings` section, just as you would if adding a @@ -195,7 +188,7 @@ GET my-index/_search } } ---- -// TEST[continued] +//TEST[continued] Defining a runtime field in a search request uses the same format as defining a runtime field in the index mapping. That consistency means you can promote a @@ -203,13 +196,34 @@ runtime field from a search request to the index mapping by moving the field definition from `runtime_mappings` in the search request to the `runtime` section of the index mapping. -[[runtime-shadowing-fields]] -=== Shadowing fields +[[runtime-fields-scriptless]] +==== Define runtime fields without a script +You can define a runtime field in the mapping definition without a +script. At query time, {es} looks in `_source` for a field with the same name +and returns a value if one exists. If a field with the same name doesn’t +exist, the response doesn't include any values for that runtime field. + +[source,console] +---- +PUT my-index/ +{ + "mappings": { + "runtime": { + "model_number": { + "type": "keyword" + } + } + } +} +---- + +[[runtime-override-values]] +=== Override field values at query time If you create a runtime field with the same name as a field that already exists in the mapping, the runtime field shadows the mapped field. At -search time, {es} evaluates the runtime field, calculates a value based on the +query time, {es} evaluates the runtime field, calculates a value based on the script, and returns the value as part of the query. Because the runtime field -shadows the mapped field, you can modify the value returned in search without +shadows the mapped field, you can override the value returned in search without modifying the mapped field. For example, let's say you indexed the following documents into `my-index`: @@ -235,7 +249,7 @@ You later realize that the `HG537PU` sensors aren't reporting their true voltage. The indexed values are supposed to be 1.7 times higher than the reported values! Instead of reindexing your data, you can define a script in the `runtime_mappings` section of the `_search` request to shadow the `voltage` -field and calculate a new value at search time. +field and calculate a new value at query time. If you search for documents where the model number matches `HG537PU`: @@ -390,7 +404,7 @@ which still returns in the response: // TESTRESPONSE[s/"_id" : "l02aSXYBkpNf6QRDO62Q"/"_id": $body.hits.hits.1._id/] [[runtime-retrieving-fields]] -=== Retrieving a runtime field +=== Retrieve a runtime field Use the <> parameter on the `_search` API to retrieve the values of runtime fields. Runtime fields won't display in `_source`, but the `fields` API works for all fields, even those that were not sent as part of @@ -398,8 +412,8 @@ the original `_source`. The following request uses the search API to retrieve the `day_of_week` field that <> defined as a runtime field -in the mapping. The value for the `day_of_week` field is calculated dynamically -at search time based on the evaluation of the defined script. +in the mapping. The value for the `day_of_week` field is calculated at query +time based on the evaluation of the defined script. [source,console] ---- @@ -415,18 +429,22 @@ GET my-index/_search // TEST[continued] [[runtime-examples]] -=== Runtime fields examples +=== Explore your data with runtime fields Consider a large set of log data that you want to extract fields from. Indexing the data is time consuming and uses a lot of disk space, and you just want to explore the data structure without committing to a schema up front. You know that your log data contains specific fields that you want to extract. -By using runtime fields, you can define scripts to calculate values at search +In this case, we want to focus on the `@timestamp` and `message` fields. By +using runtime fields, you can define scripts to calculate values at search time for these fields. +[[runtime-examples-define-fields]] +==== Define indexed fields as a starting point + You can start with a simple example by adding the `@timestamp` and `message` -fields to the `my-index` mapping. To remain flexible, use `wildcard` as the -field type for `message`: +fields to the `my-index` mapping as indexed fields. To remain flexible, use +`wildcard` as the field type for `message`: [source,console] ---- @@ -446,6 +464,8 @@ PUT /my-index/ } ---- +[[runtime-examples-ingest-data]] +==== Ingest some data After mapping the fields you want to retrieve, index a few records from your log data into {es}. The following request uses the <> to index raw log data into `my-index`. Instead of indexing all of your log @@ -511,6 +531,8 @@ The mapping contains two fields: `@timestamp` and `message`. ---- // TESTRESPONSE[s/\.\.\./"settings": $body.my-index.settings/] +[[runtime-examples-runtime-field]] +==== Define a runtime field to search by IP address If you want to retrieve results that include `clientip`, you can add that field as a runtime field in the mapping. The runtime script operates on the `clientip` field at runtime to calculate values for that field. @@ -591,6 +613,8 @@ and determine which fields to index. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/] +[[runtime-examples-calculate-value]] +==== Define a runtime field to calculate the day of week You can add the `day_of_week` field to the mapping using the request from <>: @@ -637,6 +661,9 @@ The value for this field is calculated dynamically at runtime without reindexing the document or adding the `day_of_week` field. This flexibility allows you to modify the mapping without changing any field values. +In the following response, the value for `day_of_week` (`Sunday`) was +calculated at query time using the runtime script defined in the mapping. + [source,console-result] ---- { @@ -667,7 +694,7 @@ allows you to modify the mapping without changing any field values. "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0" ], "day_of_week" : [ - "Sunday" <1> + "Sunday" ] } } @@ -678,6 +705,3 @@ allows you to modify the mapping without changing any field values. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/] // TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/] - -<1> This value was calculated at search time using the runtime script defined -in the mapping.