From e3c939d8ab3482143d073d89feee92659bb44d3c Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Wed, 16 Dec 2020 17:07:04 -0500 Subject: [PATCH 01/11] Changes for dynamic templates. --- .../mapping/dynamic/field-mapping.asciidoc | 26 ++++++---- .../mapping/dynamic/templates.asciidoc | 50 +++++++++---------- .../reference/mapping/params/dynamic.asciidoc | 3 -- 3 files changed, 42 insertions(+), 37 deletions(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index bf5a9165d6f28..b2aef225dfc32 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -9,11 +9,22 @@ You can disable dynamic mapping, both at the document and at the `false` ignores new fields, and `strict` rejects the document if {es} encounters an unknown field. -When dynamic field mapping is enabled, {es} uses the following rules to -determine how to map data types for each field. +TIP: Use the <> to update the `dynamic` +setting on existing fields. -NOTE: These are the only <> that {es} detects -dynamically. All other data types must be mapped explicitly. +Dynamic field mapping is enabled by default, but can also be +enabled by setting the `dynamic` parameter to `runtime`. When dynamic field +mapping is enabled, {es} uses the rules in the following table to determine how +to map data types for each field. + +You can customize dynamic field mapping rules for +<> and <>. +To define custom mappings rules that you can apply to additional dynamic +fields, use <>. + +NOTE: The field data types in the following table are the only +<> that {es} detects dynamically. You must +explicitly map all other data types. [cols="3"] |=== @@ -30,9 +41,6 @@ h| JSON data type h| `dynamic:true` h| `dynamic:runtime` 3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | | |=== -Besides the following options, you can customize dynamic field mapping rules -with <>. - [[date-detection]] ==== Date detection @@ -83,9 +91,9 @@ PUT my-index-000001/_doc/1 <1> <1> The `create_date` field has been added as a <> field. -===== Customising detected date formats +===== Customizing detected date formats -Alternatively, the `dynamic_date_formats` can be customised to support your +Alternatively, the `dynamic_date_formats` can be customized to support your own <>: [source,console] diff --git a/docs/reference/mapping/dynamic/templates.asciidoc b/docs/reference/mapping/dynamic/templates.asciidoc index 7f13007bc61f1..ca5446c94546f 100644 --- a/docs/reference/mapping/dynamic/templates.asciidoc +++ b/docs/reference/mapping/dynamic/templates.asciidoc @@ -2,18 +2,20 @@ === Dynamic templates Dynamic templates allow you to define custom mappings that can be applied to -dynamically added fields based on: +dynamically added fields based on the matching condition: -* the <> detected by Elasticsearch, with <>. -* the name of the field, with <> or <>. -* the full dotted path to the field, with <>. +* <> operates on the +<> that {es} detects +* <> use a pattern to match on the field +name +* <> operate on the full +dotted path to the field -The original field name `{name}` and the detected data type -`{dynamic_type}` <> can be used in -the mapping specification as placeholders. +Use the `{name}` and `{dynamic_type}` <> +in the mapping specification as placeholders. IMPORTANT: Dynamic field mappings are only added when a field contains a -concrete value -- not `null` or an empty array. This means that if the +concrete value -- not `null` or an empty array. If the `null_value` option is used in a `dynamic_template`, it will only be applied after the first document with a concrete value for the field has been indexed. @@ -37,6 +39,8 @@ Dynamic templates are specified as an array of named objects: <2> The match conditions can include any of : `match_mapping_type`, `match`, `match_pattern`, `unmatch`, `path_match`, `path_unmatch`. <3> The mapping that the matched field should use. +[[dynamic-templates-validation]] +==== Validating dynamic templates If a provided mapping contains an invalid mapping snippet, a validation error is returned. Validation occurs when applying the dynamic template at index time, and, in most cases, when the dynamic template is updated. Providing an invalid mapping @@ -131,11 +135,21 @@ PUT my-index-000001/_doc/1 The `match` parameter uses a pattern to match on the field name, while `unmatch` uses a pattern to exclude fields matched by `match`. +The `match_pattern` parameter adjusts the behavior of the `match` parameter +to support full Java regular expressions matching on the field name +instead of simple wildcards. For example: + +[source,js] +-------------------------------------------------- + "match_pattern": "regex", + "match": "^profit_\d+$" +-------------------------------------------------- +// NOTCONSOLE + The following example matches all `string` fields whose name starts with `long_` (except for those which end with `_text`) and maps them as `long` fields: - [source,console] -------------------------------------------------- PUT my-index-000001 @@ -166,20 +180,6 @@ PUT my-index-000001/_doc/1 <1> The `long_num` field is mapped as a `long`. <2> The `long_text` field uses the default `string` mapping. -[[match-pattern]] -==== `match_pattern` - -The `match_pattern` parameter adjusts the behavior of the `match` parameter -such that it supports full Java regular expression matching on the field name -instead of simple wildcards, for instance: - -[source,js] --------------------------------------------------- - "match_pattern": "regex", - "match": "^profit_\d+$" --------------------------------------------------- -// NOTCONSOLE - [[path-match-unmatch]] ==== `path_match` and `path_unmatch` @@ -243,7 +243,7 @@ PUT my-index-000001/_doc/2 // TEST[catch:bad_request] [[template-variables]] -==== `{name}` and `{dynamic_type}` +==== Template variables The `{name}` and `{dynamic_type}` placeholders are replaced in the `mapping` with the field name and detected dynamic type. The following example sets all @@ -290,7 +290,7 @@ PUT my-index-000001/_doc/1 <2> The `count` field is mapped as a `long` field with `doc_values` disabled. [[template-examples]] -==== Template examples +==== Dynamic template examples Here are some examples of potentially useful dynamic templates: diff --git a/docs/reference/mapping/params/dynamic.asciidoc b/docs/reference/mapping/params/dynamic.asciidoc index 6ded8e4f7809c..1bb845885c43a 100644 --- a/docs/reference/mapping/params/dynamic.asciidoc +++ b/docs/reference/mapping/params/dynamic.asciidoc @@ -39,9 +39,6 @@ PUT my-index-000001/_doc/2 GET my-index-000001/_mapping ---- -TIP: Use the <> to update the `dynamic` -setting on existing fields. - [[dynamic-inner-objects]] ==== Setting `dynamic` on inner objects <> inherit the `dynamic` setting from their parent From e0368fdf5f361350cbc68fc22f0c079b98bf1282 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Thu, 17 Dec 2020 10:15:52 -0500 Subject: [PATCH 02/11] Clarifying language around dynamic:true and dynamic:runtime. --- .../mapping/dynamic/field-mapping.asciidoc | 36 +++++++++---------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index b2aef225dfc32..33067b76d1384 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -2,25 +2,12 @@ === Dynamic field mapping When {es} detects a new field in a document, it _dynamically_ adds the field to -the type mapping. The <> parameter controls this behavior. +the type mapping by default. The <> parameter controls this behavior. -You can disable dynamic mapping, both at the document and at the -<> level. Setting the `dynamic` parameter to -`false` ignores new fields, and `strict` rejects the document if {es} -encounters an unknown field. - -TIP: Use the <> to update the `dynamic` -setting on existing fields. - -Dynamic field mapping is enabled by default, but can also be -enabled by setting the `dynamic` parameter to `runtime`. When dynamic field -mapping is enabled, {es} uses the rules in the following table to determine how -to map data types for each field. - -You can customize dynamic field mapping rules for -<> and <>. -To define custom mappings rules that you can apply to additional dynamic -fields, use <>. +You can explicitly instruct {es} to dynamically create fields based on incoming +documents by setting the `dynamic` parameter to `true` or `runtime`. When +dynamic field mapping is enabled, {es} uses the rules in the following table to +determine how to map data types for each field. NOTE: The field data types in the following table are the only <> that {es} detects dynamically. You must @@ -41,6 +28,19 @@ h| JSON data type h| `dynamic:true` h| `dynamic:runtime` 3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | | |=== +You can disable dynamic mapping, both at the document and at the +<> level. Setting the `dynamic` parameter to +`false` ignores new fields, and `strict` rejects the document if {es} +encounters an unknown field. + +TIP: Use the <> to update the `dynamic` +setting on existing fields. + +You can customize dynamic field mapping rules for +<> and <>. +To define custom mappings rules that you can apply to additional dynamic +fields, use <>. + [[date-detection]] ==== Date detection From d1a51205f817193bbfddd46c94fa0cc08fac62b3 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Thu, 17 Dec 2020 18:12:08 -0500 Subject: [PATCH 03/11] Clarifying edits and some restructuring. --- .../mapping/dynamic/field-mapping.asciidoc | 3 +- .../mapping/dynamic/templates.asciidoc | 33 +++++- docs/reference/mapping/runtime.asciidoc | 104 ++++++++++-------- 3 files changed, 88 insertions(+), 52 deletions(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index 33067b76d1384..112234e81e617 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -13,12 +13,13 @@ NOTE: The field data types in the following table are the only <> that {es} detects dynamically. You must explicitly map all other data types. +[[dynamic-field-mapping-types]] [cols="3"] |=== h| JSON data type h| `dynamic:true` h| `dynamic:runtime` |`null` 2*| No field added |`true` or `false` 2*| `boolean` - |floating point number | `float` | `double` + |`double` | `float` | `double` |`integer` 2*| `long` |`object`<> 2*| `object` |`array` 2*| Depends on the first non-`null` value in the array diff --git a/docs/reference/mapping/dynamic/templates.asciidoc b/docs/reference/mapping/dynamic/templates.asciidoc index ca5446c94546f..d05a37523774c 100644 --- a/docs/reference/mapping/dynamic/templates.asciidoc +++ b/docs/reference/mapping/dynamic/templates.asciidoc @@ -65,12 +65,12 @@ reordered or deleted after they were initially added. [[match-mapping-type]] ==== `match_mapping_type` -The `match_mapping_type` is the data type detected by the JSON parser. Since +The `match_mapping_type` is the data type detected by the JSON parser. Because JSON doesn't distinguish a `long` from an `integer` or a `double` from -a `float`, it will always choose the wider data type, i.e. `long` for integers +a `float`, it always chooses the wider data type such as `long` for integers and `double` for floating-point numbers. -The following data types may be automatically detected: +{es} automatically detects the following data types: - `boolean` when `true` or `false` are encountered. - `date` when <> is enabled and a string matching @@ -80,7 +80,7 @@ The following data types may be automatically detected: - `object` for objects, also called hashes. - `string` for character strings. -`*` may also be used in order to match all data types. +Use a wildcard (`*`) to match all data types. For example, if we wanted to map all integer fields as `integer` instead of `long`, and all `string` fields as both `text` and `keyword`, we @@ -128,6 +128,31 @@ PUT my-index-000001/_doc/1 <1> The `my_integer` field is mapped as an `integer`. <2> The `my_string` field is mapped as a `text`, with a `keyword` <>. +[[match-mapping-runtime-fields]] +===== Mapping runtime fields in a dynamic template +You can also map runtime fields in a dynamic template. For example, the +following request adds a dynamic template named `dayofweek` that maps all +incoming `string` fields as `keyword` fields. Although the `runtime` definition +is blank, new `string` fields will be mapped as `keyword` fields based on the +<> that {es} uses for adding +field types to the mapping. Any `string` that doesn't pass date detection or +numeric detection is automatically mapped as a `keyword` when `dynamic` is set +to `runtime`. + +[source,console] +---- +PUT my-index/_mappings +{ + "dynamic_templates": [ + { + "dayOfWeek": { + "match_mapping_type": "string", + "runtime": {} + } + } + ] +} +---- [[match-unmatch]] ==== `match` and `unmatch` diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index dd1e2404e7a08..8bdacf95c8aae 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -1,9 +1,12 @@ [[runtime]] == Runtime fields -Typically, you index data into {es} to promote faster search. However, indexing -can be slow and requires more disk space, and you have to reindex your data to -add fields to existing documents. With _runtime fields_, you can add -fields to documents already indexed to {es} without reindexing your data. +A _runtime field_ is a field that is evaluated at query time. Runtime fields +enable you to: + +* Add fields to existing documents without reindexing your data +* Start working with your data without understanding how it’s structured +* Override the value returned from an indexed field at query time +* Define fields for a specific use without modifying the underlying schema You access runtime fields from the search API like any other field, and {es} sees runtime fields no differently. You can define runtime fields in the @@ -44,13 +47,13 @@ smaller and you can more quickly process logs without having to index them. Runtime fields are especially useful in the following contexts: -* Adding fields to documents that are already indexed without having to reindex +* Add fields to documents that are already indexed without having to reindex data -* Immediately begin working on a new data stream without fully understanding -the data it contains -* Shadowing an indexed field with a runtime field to fix a mistake after -indexing documents -* Defining fields that are only relevant for a particular context (such as a +* Start work on a new data stream without fully understanding the data it +contains +* Override the value of an indexed field with a runtime field to fix a mistake +after indexing documents +* Define fields that are only relevant for a particular context (such as a visualization in {kib}) without influencing the underlying schema [discrete] @@ -80,27 +83,14 @@ to `false`, expensive queries are not allowed and {es} will reject any queries against runtime fields. [[runtime-mapping-fields]] -=== Mapping a runtime field +=== Map a runtime field You map runtime fields by adding a `runtime` section under the mapping definition and defining <>. This script has access to the entire context of a document, including the original `_source` and any mapped -fields plus their values. At search time, the script runs and generates values +fields plus their values. At query time, the script runs and generates values for each scripted field that is required for the query. -NOTE: You can define a runtime field in the mapping definition without a -script. At search time, {es} looks in `_source` for a field with the same name -and returns a value if one exists. If a field with the same name doesn’t -exist, the response doesn't include any values for that runtime field. - -If <> is enabled where the -`dynamic` parameter is set to `runtime`, new fields are automatically added to -the index mapping as runtime fields. - -Runtime fields are similar to the <> parameter -of the `_search` request, but also make the script results available anywhere -in a search request. - The script in the following request extracts the day of the week from the `@timestamp` field, which is defined as a `date` type: @@ -109,10 +99,10 @@ The script in the following request extracts the day of the week from the PUT /my-index { "mappings": { - "runtime": { <1> + "runtime": { "day_of_week": { - "type": "keyword", <2> - "script": { <3> + "type": "keyword", + "script": { "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" } } @@ -124,11 +114,6 @@ PUT /my-index } ---- -<1> Runtime fields are defined in the `runtime` section of the mapping -definition. -<2> Each runtime field has its own field type, just like any other field. -<3> The script defines the evaluation to calculate at search time. - The `runtime` section can be any of these data types: * `boolean` @@ -142,6 +127,10 @@ The `runtime` section can be any of these data types: Runtime fields with a `type` of `date` can accept the <> parameter exactly as the `date` field type. +If <> is enabled where the +`dynamic` parameter is set to `runtime`, new fields are automatically added to +the index mapping as runtime fields. + [[runtime-updating-scripts]] .Updating runtime scripts **** @@ -157,8 +146,29 @@ to `boolean`. **** +[[runtime-fields-scriptless]] +==== Define runtime fields without a script +You can define a runtime field in the mapping definition without a +script. At query time, {es} looks in `_source` for a field with the same name +and returns a value if one exists. If a field with the same name doesn’t +exist, the response doesn't include any values for that runtime field. + +[source,console] +---- +PUT my-index/ +{ + "mappings": { + "runtime": { + "newField": { + "type": "keyword" + } + } + } +} +---- + [[runtime-search-request]] -=== Defining runtime fields in a search request +=== Define runtime fields in a search request You can specify a `runtime_mappings` section in a search request to create runtime fields that exist only as part of the query. You specify a script as part of the `runtime_mappings` section, just as you would if adding a @@ -203,13 +213,13 @@ runtime field from a search request to the index mapping by moving the field definition from `runtime_mappings` in the search request to the `runtime` section of the index mapping. -[[runtime-shadowing-fields]] -=== Shadowing fields +[[runtime-override-values]] +=== Override field values at query time If you create a runtime field with the same name as a field that already exists in the mapping, the runtime field shadows the mapped field. At -search time, {es} evaluates the runtime field, calculates a value based on the +query time, {es} evaluates the runtime field, calculates a value based on the script, and returns the value as part of the query. Because the runtime field -shadows the mapped field, you can modify the value returned in search without +shadows the mapped field, you can override the value returned in search without modifying the mapped field. For example, let's say you indexed the following documents into `my-index`: @@ -235,7 +245,7 @@ You later realize that the `HG537PU` sensors aren't reporting their true voltage. The indexed values are supposed to be 1.7 times higher than the reported values! Instead of reindexing your data, you can define a script in the `runtime_mappings` section of the `_search` request to shadow the `voltage` -field and calculate a new value at search time. +field and calculate a new value at query time. If you search for documents where the model number matches `HG537PU`: @@ -390,7 +400,7 @@ which still returns in the response: // TESTRESPONSE[s/"_id" : "l02aSXYBkpNf6QRDO62Q"/"_id": $body.hits.hits.1._id/] [[runtime-retrieving-fields]] -=== Retrieving a runtime field +=== Retrieve a runtime field Use the <> parameter on the `_search` API to retrieve the values of runtime fields. Runtime fields won't display in `_source`, but the `fields` API works for all fields, even those that were not sent as part of @@ -398,8 +408,8 @@ the original `_source`. The following request uses the search API to retrieve the `day_of_week` field that <> defined as a runtime field -in the mapping. The value for the `day_of_week` field is calculated dynamically -at search time based on the evaluation of the defined script. +in the mapping. The value for the `day_of_week` field is calculated at query +time based on the evaluation of the defined script. [source,console] ---- @@ -415,7 +425,7 @@ GET my-index/_search // TEST[continued] [[runtime-examples]] -=== Runtime fields examples +=== Explore your data with runtime fields Consider a large set of log data that you want to extract fields from. Indexing the data is time consuming and uses a lot of disk space, and you just want to explore the data structure without committing to a schema up front. @@ -637,6 +647,9 @@ The value for this field is calculated dynamically at runtime without reindexing the document or adding the `day_of_week` field. This flexibility allows you to modify the mapping without changing any field values. +In the following response, the value for `day_of_week` (`Sunday`) was +calculated at query time using the runtime script defined in the mapping. + [source,console-result] ---- { @@ -667,7 +680,7 @@ allows you to modify the mapping without changing any field values. "211.11.9.0 - - [2020-06-21T15:00:01-05:00] \"GET /english/index.html HTTP/1.0\" 304 0" ], "day_of_week" : [ - "Sunday" <1> + "Sunday" ] } } @@ -678,6 +691,3 @@ allows you to modify the mapping without changing any field values. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/] // TESTRESPONSE[s/"day_of_week" : \[\n\s+"Sunday"\n\s\]/"day_of_week": $body.hits.hits.0.fields.day_of_week/] - -<1> This value was calculated at search time using the runtime script defined -in the mapping. From 245a8cab07b92bdc12fab562fc351463cab047a5 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Mon, 4 Jan 2021 18:27:40 -0500 Subject: [PATCH 04/11] Overhauling the Mapping page. --- docs/reference/mapping.asciidoc | 223 +++--------------- .../mapping/dynamic-mapping.asciidoc | 9 +- .../mapping/dynamic/templates.asciidoc | 18 +- .../mapping/explicit-mapping.asciidoc | 148 ++++++++++++ docs/reference/mapping/runtime.asciidoc | 2 +- 5 files changed, 202 insertions(+), 198 deletions(-) create mode 100644 docs/reference/mapping/explicit-mapping.asciidoc diff --git a/docs/reference/mapping.asciidoc b/docs/reference/mapping.asciidoc index ebea396cd279e..fc657e1d4ea90 100644 --- a/docs/reference/mapping.asciidoc +++ b/docs/reference/mapping.asciidoc @@ -5,14 +5,39 @@ -- Mapping is the process of defining how a document, and the fields it contains, -are stored and indexed. For instance, use mappings to define: - -* which string fields should be treated as full text fields. -* which fields contain numbers, dates, or geolocations. -* the <> of date values. -* custom rules to control the mapping for +are stored and indexed. You can choose how and where to map your data. Each +method provides different benefits based on where you are in your data journey: + +* <> allow you to experiment with +and explore data when you’re just getting started. {es} will add new fields +automatically, just by indexing a document. You can add fields to the top-level +mapping, and to inner <> and <> fields. +* <> are flexible, +allowing you to experiment with different mapping options. +* <> allow you to make +schema changes without reindexing. You can use these fields in conjunction with +indexed fields to balance resource usage and performance. Your index will be +smaller, but with slower search performance. +* <> provide the best search performance +but require more storage space, resulting in a larger index. + +When mapping your data using runtime fields or through explicit mapping, you +choose how to define the mapping definition, such as: + +* Which string fields should be treated as full text fields. +* Which fields contain numbers, dates, or geolocations. +* The <> of date values. +* Custom rules to control the mapping for <>. +[discrete] +[[mapping-definition-fields]] +== Mapping fields +Each document is a collection of fields. Most fields are indexed by default, which makes them searchable. When indexing documents, fields are saved to +`_source`, which contains the original JSON document body that was passed at index time. The inverted index allows queries to look up the search term in unique sorted list of terms, and from that immediately have access to the list of documents that contain the term. + +However, we also need the ability to look up the document and find the terms that it has in a field. Doc values are the on-disk data structure, built at document index time, which makes this data access pattern possible. They store the same values as the `_source` but in a column-oriented fashion that is way more efficient for sorting and aggregations. + A mapping definition includes metadata fields and fields: <>:: @@ -27,8 +52,8 @@ treated. Examples of metadata fields include the document's A mapping contains a list of fields or `properties` pertinent to the document. Each field has its own <>. -NOTE: Before 7.0.0, the 'mappings' definition used to include a type name. -For more details, please see <>. +NOTE: Before 7.0.0, the 'mappings' definition included a type name. +{es} 7.0.0 and later no longer accept a _default_ mapping. See <>. [discrete] [[mapping-limit-settings]] @@ -45,184 +70,8 @@ Use the <> to limit the number of field mappings (created manually or dynamically) and prevent documents from causing a mapping explosion. -[discrete] -[[runtime-fields]] -== Runtime fields -Typically, you index data into {es} to promote faster search. However, indexing -can be slow and requires more disk space, and you have to reindex your data to -add fields to existing documents. - -<> are not indexed, which saves disk space and makes -data ingest faster. You can add runtime fields to existing documents without -reindexing your data and calculate field values dynamically at search time. - -[discrete] -[[dynamic-mapping-intro]] -== Dynamic mapping - -Fields and mapping types do not need to be defined before being used. Thanks -to _dynamic mapping_, new field names will be added automatically, just by -indexing a document. New fields can be added both to the top-level mapping -type, and to inner <> and <> fields. - -The <> rules can be configured to customise -the mapping that is used for new fields. - -[discrete] -== Explicit mappings - -You know more about your data than Elasticsearch can guess, so while dynamic -mapping can be useful to get started, at some point you will want to specify -your own explicit mappings. - -You can create field mappings when you <> and -<>. - -[discrete] -[[create-mapping]] -=== Create an index with an explicit mapping - -You can use the <> API to create a new index -with an explicit mapping. - -[source,console] ----- -PUT /my-index-000001 -{ - "mappings": { - "properties": { - "age": { "type": "integer" }, <1> - "email": { "type": "keyword" }, <2> - "name": { "type": "text" } <3> - } - } -} ----- - -<1> Creates `age`, an <> field -<2> Creates `email`, a <> field -<3> Creates `name`, a <> field - -[discrete] -[[add-field-mapping]] -== Add a field to an existing mapping - -You can use the <> API to add one or more new -fields to an existing index. - -The following example adds `employee-id`, a `keyword` field with an -<> mapping parameter value of `false`. This means values -for the `employee-id` field are stored but not indexed or available for search. - -[source,console] ----- -PUT /my-index-000001/_mapping -{ - "properties": { - "employee-id": { - "type": "keyword", - "index": false - } - } -} ----- -// TEST[continued] - -[discrete] -[[update-mapping]] -=== Update the mapping of a field - -include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=change-field-mapping] - -include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=rename-field] - -[discrete] -[[view-mapping]] -== View the mapping of an index - -You can use the <> API to view the mapping of -an existing index. - -[source,console] ----- -GET /my-index-000001/_mapping ----- -// TEST[continued] - -The API returns the following response: - -[source,console-result] ----- -{ - "my-index-000001" : { - "mappings" : { - "properties" : { - "age" : { - "type" : "integer" - }, - "email" : { - "type" : "keyword" - }, - "employee-id" : { - "type" : "keyword", - "index" : false - }, - "name" : { - "type" : "text" - } - } - } - } -} ----- - - -[discrete] -[[view-field-mapping]] -== View the mapping of specific fields - -If you only want to view the mapping of one or more specific fields, you can use -the <> API. - -This is useful if you don't need the complete mapping of an index or your index -contains a large number of fields. - -The following request retrieves the mapping for the `employee-id` field. - -[source,console] ----- -GET /my-index-000001/_mapping/field/employee-id ----- -// TEST[continued] - -The API returns the following response: - -[source,console-result] ----- -{ - "my-index-000001" : { - "mappings" : { - "employee-id" : { - "full_name" : "employee-id", - "mapping" : { - "employee-id" : { - "type" : "keyword", - "index" : false - } - } - } - } - } -} - ----- - -- -include::mapping/removal_of_types.asciidoc[] - -include::mapping/mapping-settings-limit.asciidoc[] - include::mapping/types.asciidoc[] include::mapping/runtime.asciidoc[] @@ -232,3 +81,9 @@ include::mapping/fields.asciidoc[] include::mapping/params.asciidoc[] include::mapping/dynamic-mapping.asciidoc[] + +include::mapping/explicit-mapping.asciidoc[] + +include::mapping/mapping-settings-limit.asciidoc[] + +include::mapping/removal_of_types.asciidoc[] diff --git a/docs/reference/mapping/dynamic-mapping.asciidoc b/docs/reference/mapping/dynamic-mapping.asciidoc index e62f243b669eb..875e879495864 100644 --- a/docs/reference/mapping/dynamic-mapping.asciidoc +++ b/docs/reference/mapping/dynamic-mapping.asciidoc @@ -1,11 +1,11 @@ [[dynamic-mapping]] -== Dynamic Mapping +== Dynamic mapping -One of the most important features of Elasticsearch is that it tries to get +One of the most important features of {es} is that it tries to get out of your way and let you start exploring your data as quickly as possible. To index a document, you don't have to first create an index, define a mapping type, and define your fields -- you can just index a document and the index, -type, and fields will spring to life automatically: +type, and fields will display automatically: [source,console] -------------------------------------------------- @@ -17,7 +17,7 @@ PUT data/_doc/1 <1> called `count` with data type `long`. The automatic detection and addition of new fields is called -_dynamic mapping_. The dynamic mapping rules can be customised to suit your +_dynamic mapping_. The dynamic mapping rules can be customized to suit your purposes with: <>:: @@ -35,4 +35,3 @@ automatically or explicitly. include::dynamic/field-mapping.asciidoc[] include::dynamic/templates.asciidoc[] - diff --git a/docs/reference/mapping/dynamic/templates.asciidoc b/docs/reference/mapping/dynamic/templates.asciidoc index d05a37523774c..33ed41589cf53 100644 --- a/docs/reference/mapping/dynamic/templates.asciidoc +++ b/docs/reference/mapping/dynamic/templates.asciidoc @@ -141,16 +141,18 @@ to `runtime`. [source,console] ---- -PUT my-index/_mappings +PUT my-index-000001/ { - "dynamic_templates": [ - { - "dayOfWeek": { - "match_mapping_type": "string", - "runtime": {} + "mappings": { + "dynamic_templates": [ + { + "dayOfWeek": { + "match_mapping_type": "string", + "runtime": {} + } } - } - ] + ] + } } ---- diff --git a/docs/reference/mapping/explicit-mapping.asciidoc b/docs/reference/mapping/explicit-mapping.asciidoc new file mode 100644 index 0000000000000..b056a263a8408 --- /dev/null +++ b/docs/reference/mapping/explicit-mapping.asciidoc @@ -0,0 +1,148 @@ + +[[explicit-mapping]] +== Explicit field mapping + +You know more about your data than {es} can guess, so while dynamic +mapping can be useful to get started, at some point you will want to specify +your own explicit mappings. + +You can create field mappings when you <> and +<>. + +[discrete] +[[create-mapping]] +=== Create an index with an explicit mapping + +You can use the <> API to create a new index +with an explicit mapping. + +[source,console] +---- +PUT /my-index-000001 +{ + "mappings": { + "properties": { + "age": { "type": "integer" }, <1> + "email": { "type": "keyword" }, <2> + "name": { "type": "text" } <3> + } + } +} +---- + +<1> Creates `age`, an <> field +<2> Creates `email`, a <> field +<3> Creates `name`, a <> field + +[discrete] +[[add-field-mapping]] +=== Add a field to an existing mapping + +You can use the <> API to add one or more new +fields to an existing index. + +The following example adds `employee-id`, a `keyword` field with an +<> mapping parameter value of `false`. This means values +for the `employee-id` field are stored but not indexed or available for search. + +[source,console] +---- +PUT /my-index-000001/_mapping +{ + "properties": { + "employee-id": { + "type": "keyword", + "index": false + } + } +} +---- +// TEST[continued] + +[discrete] +[[update-mapping]] +=== Update the mapping of a field + +include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=change-field-mapping] + +include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=rename-field] + +[discrete] +[[view-mapping]] +=== View the mapping of an index + +You can use the <> API to view the mapping of +an existing index. + +[source,console] +---- +GET /my-index-000001/_mapping +---- +// TEST[continued] + +The API returns the following response: + +[source,console-result] +---- +{ + "my-index-000001" : { + "mappings" : { + "properties" : { + "age" : { + "type" : "integer" + }, + "email" : { + "type" : "keyword" + }, + "employee-id" : { + "type" : "keyword", + "index" : false + }, + "name" : { + "type" : "text" + } + } + } + } +} +---- + +[discrete] +[[view-field-mapping]] +=== View the mapping of specific fields + +If you only want to view the mapping of one or more specific fields, you can use +the <> API. + +This is useful if you don't need the complete mapping of an index or your index +contains a large number of fields. + +The following request retrieves the mapping for the `employee-id` field. + +[source,console] +---- +GET /my-index-000001/_mapping/field/employee-id +---- +// TEST[continued] + +The API returns the following response: + +[source,console-result] +---- +{ + "my-index-000001" : { + "mappings" : { + "employee-id" : { + "full_name" : "employee-id", + "mapping" : { + "employee-id" : { + "type" : "keyword", + "index" : false + } + } + } + } + } +} + +---- diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index 8bdacf95c8aae..d62a51c7b4aa8 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -96,7 +96,7 @@ The script in the following request extracts the day of the week from the [source,console] ---- -PUT /my-index +PUT my-index/ { "mappings": { "runtime": { From 1f30c4590dba928b3be7d16ee86a768a71614d64 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Tue, 5 Jan 2021 12:29:24 -0500 Subject: [PATCH 05/11] Incorporating changes from #66911. --- docs/reference/mapping/dynamic/field-mapping.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/mapping/dynamic/field-mapping.asciidoc b/docs/reference/mapping/dynamic/field-mapping.asciidoc index 112234e81e617..ae7e0e0b7b0cc 100644 --- a/docs/reference/mapping/dynamic/field-mapping.asciidoc +++ b/docs/reference/mapping/dynamic/field-mapping.asciidoc @@ -24,7 +24,7 @@ h| JSON data type h| `dynamic:true` h| `dynamic:runtime` |`object`<> 2*| `object` |`array` 2*| Depends on the first non-`null` value in the array |`string` that passes <> 2*| `date` - |`string` that passes <> | `double` or `long` | `double` + |`string` that passes <> | `float` or `long` | `double` or `long` |`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword` 3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | | |=== From 901b78df48d01a59a26dd4b6a0ba2ae1c120983f Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Tue, 5 Jan 2021 16:30:45 -0500 Subject: [PATCH 06/11] Reworking mapping page to focus on dynamic vs. explicit mapping. --- docs/reference/mapping.asciidoc | 69 +++++++------- .../mapping/explicit-mapping.asciidoc | 2 +- docs/reference/mapping/runtime.asciidoc | 92 +++++++++---------- 3 files changed, 84 insertions(+), 79 deletions(-) diff --git a/docs/reference/mapping.asciidoc b/docs/reference/mapping.asciidoc index fc657e1d4ea90..37ef6d314a32b 100644 --- a/docs/reference/mapping.asciidoc +++ b/docs/reference/mapping.asciidoc @@ -6,23 +6,29 @@ Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. You can choose how and where to map your data. Each -method provides different benefits based on where you are in your data journey: +method provides different benefits based on where you are in your data journey. -* <> allow you to experiment with -and explore data when you’re just getting started. {es} will add new fields +[discrete] +[[mapping-dynamic]] +== Dynamic mapping +<> allows you to experiment with +and explore data when you’re just getting started. {es} adds new fields automatically, just by indexing a document. You can add fields to the top-level mapping, and to inner <> and <> fields. -* <> are flexible, -allowing you to experiment with different mapping options. -* <> allow you to make -schema changes without reindexing. You can use these fields in conjunction with -indexed fields to balance resource usage and performance. Your index will be -smaller, but with slower search performance. -* <> provide the best search performance -but require more storage space, resulting in a larger index. - -When mapping your data using runtime fields or through explicit mapping, you -choose how to define the mapping definition, such as: + +You can also dynamically map runtime fields: + +* <> so +that new fields are automatically added to the index mapping as runtime fields. +* <> to +experiment with different mapping options, and also fix mistakes in your index +mapping values by overriding values in the mapping during the search request. + +[discrete] +[[mapping-explicit]] +== Explicit mapping +<> allows you to precisely choose how to +define the mapping definition, such as: * Which string fields should be treated as full text fields. * Which fields contain numbers, dates, or geolocations. @@ -30,29 +36,28 @@ choose how to define the mapping definition, such as: * Custom rules to control the mapping for <>. +You can <>, +which allows you to make schema changes without reindexing. You can use runtime +fields in conjunction with indexed fields to balance resource usage and +performance. Your index will be smaller, but with slower search performance. + [discrete] [[mapping-definition-fields]] -== Mapping fields -Each document is a collection of fields. Most fields are indexed by default, which makes them searchable. When indexing documents, fields are saved to -`_source`, which contains the original JSON document body that was passed at index time. The inverted index allows queries to look up the search term in unique sorted list of terms, and from that immediately have access to the list of documents that contain the term. - -However, we also need the ability to look up the document and find the terms that it has in a field. Doc values are the on-disk data structure, built at document index time, which makes this data access pattern possible. They store the same values as the `_source` but in a column-oriented fashion that is way more efficient for sorting and aggregations. - -A mapping definition includes metadata fields and fields: - -<>:: - -Metadata fields are used to customize how a document's associated metadata is -treated. Examples of metadata fields include the document's -<>, <>, and -<> fields. +== Fields in the mapping definition +Each document is a collection of fields, which each have +their own <>. A mapping definition contains a list of +fields or `properties` pertinent to the document. -<>:: +Most fields are indexed by default, which makes them searchable. When indexing +documents, fields are stored in <>, which +contains the original JSON document body that was passed at index time. -A mapping contains a list of fields or `properties` pertinent to the -document. Each field has its own <>. +<>, like the `_source` field, are used to +customize how a document's associated metadata is handled. Your mapping +definition uses a combination of metadata fields and fields to outline the +document schema. -NOTE: Before 7.0.0, the 'mappings' definition included a type name. +NOTE: Before 7.0.0, the mapping definition included a type name. {es} 7.0.0 and later no longer accept a _default_ mapping. See <>. [discrete] diff --git a/docs/reference/mapping/explicit-mapping.asciidoc b/docs/reference/mapping/explicit-mapping.asciidoc index b056a263a8408..4ef055d5699ce 100644 --- a/docs/reference/mapping/explicit-mapping.asciidoc +++ b/docs/reference/mapping/explicit-mapping.asciidoc @@ -1,6 +1,6 @@ [[explicit-mapping]] -== Explicit field mapping +== Explicit mapping You know more about your data than {es} can guess, so while dynamic mapping can be useful to get started, at some point you will want to specify diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index d62a51c7b4aa8..0a00c12c95696 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -82,6 +82,52 @@ IMPORTANT: Queries against runtime fields are considered expensive. If to `false`, expensive queries are not allowed and {es} will reject any queries against runtime fields. +[[runtime-search-request]] +=== Define runtime fields in a search request +You can specify a `runtime_mappings` section in a search request to create +runtime fields that exist only as part of the query. You specify a script +as part of the `runtime_mappings` section, just as you would if adding a +runtime field to the mappings. + +Fields defined in the search request take precedence over fields defined with +the same name in the index mappings. This flexibility allows you to shadow +existing fields and calculate a different value in the search request, without +modifying the field itself. If you made a mistake in your index mapping, you +can use runtime fields to calculate values that override values in the mapping +during the search request. + +In the following request, the values for the `day_of_week` field are calculated +dynamically, and only within the context of this search request: + +[source,console] +---- +GET my-index/_search +{ + "runtime_mappings": { + "day_of_week": { + "type": "keyword", + "script": { + "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" + } + } + }, + "aggs": { + "day_of_week": { + "terms": { + "field": "day_of_week" + } + } + } +} +---- +// TEST[continued] + +Defining a runtime field in a search request uses the same format as defining +a runtime field in the index mapping. That consistency means you can promote a +runtime field from a search request to the index mapping by moving the field +definition from `runtime_mappings` in the search request to the `runtime` +section of the index mapping. + [[runtime-mapping-fields]] === Map a runtime field You map runtime fields by adding a `runtime` section under the mapping @@ -167,52 +213,6 @@ PUT my-index/ } ---- -[[runtime-search-request]] -=== Define runtime fields in a search request -You can specify a `runtime_mappings` section in a search request to create -runtime fields that exist only as part of the query. You specify a script -as part of the `runtime_mappings` section, just as you would if adding a -runtime field to the mappings. - -Fields defined in the search request take precedence over fields defined with -the same name in the index mappings. This flexibility allows you to shadow -existing fields and calculate a different value in the search request, without -modifying the field itself. If you made a mistake in your index mapping, you -can use runtime fields to calculate values that override values in the mapping -during the search request. - -In the following request, the values for the `day_of_week` field are calculated -dynamically, and only within the context of this search request: - -[source,console] ----- -GET my-index/_search -{ - "runtime_mappings": { - "day_of_week": { - "type": "keyword", - "script": { - "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" - } - } - }, - "aggs": { - "day_of_week": { - "terms": { - "field": "day_of_week" - } - } - } -} ----- -// TEST[continued] - -Defining a runtime field in a search request uses the same format as defining -a runtime field in the index mapping. That consistency means you can promote a -runtime field from a search request to the index mapping by moving the field -definition from `runtime_mappings` in the search request to the `runtime` -section of the index mapping. - [[runtime-override-values]] === Override field values at query time If you create a runtime field with the same name as a field that From a6adf7b786fa6cfea548abf10459d47c1f7215a4 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Wed, 6 Jan 2021 09:29:22 -0500 Subject: [PATCH 07/11] Reordering to fix test failure. --- docs/reference/mapping/runtime.asciidoc | 92 ++++++++++++------------- 1 file changed, 46 insertions(+), 46 deletions(-) diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index 0a00c12c95696..ef70a046f2ae7 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -82,52 +82,6 @@ IMPORTANT: Queries against runtime fields are considered expensive. If to `false`, expensive queries are not allowed and {es} will reject any queries against runtime fields. -[[runtime-search-request]] -=== Define runtime fields in a search request -You can specify a `runtime_mappings` section in a search request to create -runtime fields that exist only as part of the query. You specify a script -as part of the `runtime_mappings` section, just as you would if adding a -runtime field to the mappings. - -Fields defined in the search request take precedence over fields defined with -the same name in the index mappings. This flexibility allows you to shadow -existing fields and calculate a different value in the search request, without -modifying the field itself. If you made a mistake in your index mapping, you -can use runtime fields to calculate values that override values in the mapping -during the search request. - -In the following request, the values for the `day_of_week` field are calculated -dynamically, and only within the context of this search request: - -[source,console] ----- -GET my-index/_search -{ - "runtime_mappings": { - "day_of_week": { - "type": "keyword", - "script": { - "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" - } - } - }, - "aggs": { - "day_of_week": { - "terms": { - "field": "day_of_week" - } - } - } -} ----- -// TEST[continued] - -Defining a runtime field in a search request uses the same format as defining -a runtime field in the index mapping. That consistency means you can promote a -runtime field from a search request to the index mapping by moving the field -definition from `runtime_mappings` in the search request to the `runtime` -section of the index mapping. - [[runtime-mapping-fields]] === Map a runtime field You map runtime fields by adding a `runtime` section under the mapping @@ -192,6 +146,52 @@ to `boolean`. **** +[[runtime-search-request]] +=== Define runtime fields in a search request +You can specify a `runtime_mappings` section in a search request to create +runtime fields that exist only as part of the query. You specify a script +as part of the `runtime_mappings` section, just as you would if adding a +runtime field to the mappings. + +Fields defined in the search request take precedence over fields defined with +the same name in the index mappings. This flexibility allows you to shadow +existing fields and calculate a different value in the search request, without +modifying the field itself. If you made a mistake in your index mapping, you +can use runtime fields to calculate values that override values in the mapping +during the search request. + +In the following request, the values for the `day_of_week` field are calculated +dynamically, and only within the context of this search request: + +[source,console] +---- +GET my-index/_search +{ + "runtime_mappings": { + "day_of_week": { + "type": "keyword", + "script": { + "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))" + } + } + }, + "aggs": { + "day_of_week": { + "terms": { + "field": "day_of_week" + } + } + } +} +---- +//TEST[continued] + +Defining a runtime field in a search request uses the same format as defining +a runtime field in the index mapping. That consistency means you can promote a +runtime field from a search request to the index mapping by moving the field +definition from `runtime_mappings` in the search request to the `runtime` +section of the index mapping. + [[runtime-fields-scriptless]] ==== Define runtime fields without a script You can define a runtime field in the mapping definition without a From 97f98ec3e8a488e1ccbef67fa026ee919d1b2845 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Wed, 6 Jan 2021 11:08:55 -0500 Subject: [PATCH 08/11] Further clarifying mapping page. --- docs/reference/mapping.asciidoc | 36 ++++++++----------- .../mapping/removal_of_types.asciidoc | 2 +- 2 files changed, 16 insertions(+), 22 deletions(-) diff --git a/docs/reference/mapping.asciidoc b/docs/reference/mapping.asciidoc index 37ef6d314a32b..cc4197b0e4331 100644 --- a/docs/reference/mapping.asciidoc +++ b/docs/reference/mapping.asciidoc @@ -5,8 +5,21 @@ -- Mapping is the process of defining how a document, and the fields it contains, -are stored and indexed. You can choose how and where to map your data. Each -method provides different benefits based on where you are in your data journey. +are stored and indexed. + +Each document is a collection of fields, which each have their own +<>. When mapping your data, you create a mapping +definition, which contains a list of fields or `properties` that are pertinent +to the document. A mapping definition also includes +<>, like the `_source` field, which customize +how a document's associated metadata is handled. + +You can choose a _dynamic mapping_ or an _explicit mapping_ to define your data. +Each method provides different benefits based on where you are in your data +journey. + +NOTE: Before 7.0.0, the mapping definition included a type name. +{es} 7.0.0 and later no longer accept a _default_ mapping. See <>. [discrete] [[mapping-dynamic]] @@ -41,25 +54,6 @@ which allows you to make schema changes without reindexing. You can use runtime fields in conjunction with indexed fields to balance resource usage and performance. Your index will be smaller, but with slower search performance. -[discrete] -[[mapping-definition-fields]] -== Fields in the mapping definition -Each document is a collection of fields, which each have -their own <>. A mapping definition contains a list of -fields or `properties` pertinent to the document. - -Most fields are indexed by default, which makes them searchable. When indexing -documents, fields are stored in <>, which -contains the original JSON document body that was passed at index time. - -<>, like the `_source` field, are used to -customize how a document's associated metadata is handled. Your mapping -definition uses a combination of metadata fields and fields to outline the -document schema. - -NOTE: Before 7.0.0, the mapping definition included a type name. -{es} 7.0.0 and later no longer accept a _default_ mapping. See <>. - [discrete] [[mapping-limit-settings]] == Settings to prevent mapping explosion diff --git a/docs/reference/mapping/removal_of_types.asciidoc b/docs/reference/mapping/removal_of_types.asciidoc index ca028355de6f7..acb521bd79909 100644 --- a/docs/reference/mapping/removal_of_types.asciidoc +++ b/docs/reference/mapping/removal_of_types.asciidoc @@ -3,4 +3,4 @@ Elasticsearch 8.0.0 no longer supports mapping types. For details on how to migrate your clusters away from mapping types, see the -{ref-7x}/removal-of-types.html[removal of types] documentation for the 7.x release +{ref-7x}/removal-of-types.html[removal of types] documentation for the 7.x release. From d99eec8f2db97e0031ec152a37cc48a66a834b89 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Wed, 6 Jan 2021 16:57:47 -0500 Subject: [PATCH 09/11] Reordering sections, adding headings to examples, and other clarifications. --- docs/reference/mapping.asciidoc | 10 +++++----- .../reference/mapping/dynamic/templates.asciidoc | 14 +++++++------- docs/reference/mapping/runtime.asciidoc | 16 +++++++++++++--- 3 files changed, 25 insertions(+), 15 deletions(-) diff --git a/docs/reference/mapping.asciidoc b/docs/reference/mapping.asciidoc index cc4197b0e4331..1a2441389fccd 100644 --- a/docs/reference/mapping.asciidoc +++ b/docs/reference/mapping.asciidoc @@ -71,18 +71,18 @@ causing a mapping explosion. -- -include::mapping/types.asciidoc[] +include::mapping/dynamic-mapping.asciidoc[] + +include::mapping/explicit-mapping.asciidoc[] include::mapping/runtime.asciidoc[] +include::mapping/types.asciidoc[] + include::mapping/fields.asciidoc[] include::mapping/params.asciidoc[] -include::mapping/dynamic-mapping.asciidoc[] - -include::mapping/explicit-mapping.asciidoc[] - include::mapping/mapping-settings-limit.asciidoc[] include::mapping/removal_of_types.asciidoc[] diff --git a/docs/reference/mapping/dynamic/templates.asciidoc b/docs/reference/mapping/dynamic/templates.asciidoc index 33ed41589cf53..d1252b8c5b13f 100644 --- a/docs/reference/mapping/dynamic/templates.asciidoc +++ b/docs/reference/mapping/dynamic/templates.asciidoc @@ -131,13 +131,13 @@ PUT my-index-000001/_doc/1 [[match-mapping-runtime-fields]] ===== Mapping runtime fields in a dynamic template You can also map runtime fields in a dynamic template. For example, the -following request adds a dynamic template named `dayofweek` that maps all -incoming `string` fields as `keyword` fields. Although the `runtime` definition -is blank, new `string` fields will be mapped as `keyword` fields based on the -<> that {es} uses for adding -field types to the mapping. Any `string` that doesn't pass date detection or -numeric detection is automatically mapped as a `keyword` when `dynamic` is set -to `runtime`. +following request adds a dynamic template named `dayOfWeek` that maps all +incoming `string` fields as `keyword` runtime fields. Although the `runtime` +definition is blank, new `string` fields will be mapped as `keyword` runtime +fields based on the <> that +{es} uses for adding field types to the mapping. Any `string` that doesn't pass +date detection or numeric detection is automatically mapped as a `keyword` when +`dynamic` is set to `runtime`. [source,console] ---- diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index ef70a046f2ae7..37da90aff7121 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -431,12 +431,16 @@ Indexing the data is time consuming and uses a lot of disk space, and you just want to explore the data structure without committing to a schema up front. You know that your log data contains specific fields that you want to extract. -By using runtime fields, you can define scripts to calculate values at search +In this case, we want to focus on the `@timestamp` and `message` fields. By +using runtime fields, you can define scripts to calculate values at search time for these fields. +[[runtime-examples-define-fields]] +==== Define indexed fields as a starting point + You can start with a simple example by adding the `@timestamp` and `message` -fields to the `my-index` mapping. To remain flexible, use `wildcard` as the -field type for `message`: +fields to the `my-index` mapping as indexed fields. To remain flexible, use +`wildcard` as the field type for `message`: [source,console] ---- @@ -456,6 +460,8 @@ PUT /my-index/ } ---- +[[runtime-examples-ingest-data]] +==== Ingest some data After mapping the fields you want to retrieve, index a few records from your log data into {es}. The following request uses the <> to index raw log data into `my-index`. Instead of indexing all of your log @@ -521,6 +527,8 @@ The mapping contains two fields: `@timestamp` and `message`. ---- // TESTRESPONSE[s/\.\.\./"settings": $body.my-index.settings/] +[[runtime-examples-runtime-field]] +==== Define a runtime field to search by IP address If you want to retrieve results that include `clientip`, you can add that field as a runtime field in the mapping. The runtime script operates on the `clientip` field at runtime to calculate values for that field. @@ -601,6 +609,8 @@ and determine which fields to index. // TESTRESPONSE[s/\.\.\./"took" : $body.took,"timed_out" : $body.timed_out,"_shards" : $body._shards,/] // TESTRESPONSE[s/"_id" : "oWs5KXYB-XyJbifr9mrz"/"_id": $body.hits.hits.0._id/] +[[runtime-examples-calculate-value]] +==== Define a runtime field to calculate the day of week You can add the `day_of_week` field to the mapping using the request from <>: From 2323993ff4e0b3909839660bf93d38c086f053a9 Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Thu, 7 Jan 2021 16:38:15 -0500 Subject: [PATCH 10/11] Incorporating review feedback. --- docs/reference/mapping.asciidoc | 43 +++++++++++-------- .../mapping/dynamic/templates.asciidoc | 11 +++-- docs/reference/mapping/runtime.asciidoc | 43 ++++++++++--------- 3 files changed, 52 insertions(+), 45 deletions(-) diff --git a/docs/reference/mapping.asciidoc b/docs/reference/mapping.asciidoc index 1a2441389fccd..af79ceea80c74 100644 --- a/docs/reference/mapping.asciidoc +++ b/docs/reference/mapping.asciidoc @@ -9,18 +9,27 @@ are stored and indexed. Each document is a collection of fields, which each have their own <>. When mapping your data, you create a mapping -definition, which contains a list of fields or `properties` that are pertinent -to the document. A mapping definition also includes -<>, like the `_source` field, which customize -how a document's associated metadata is handled. +definition, which contains a list of fields that are pertinent to the document. +A mapping definition also includes <>, like the +`_source` field, which customize how a document's associated metadata is +handled. -You can choose a _dynamic mapping_ or an _explicit mapping_ to define your data. -Each method provides different benefits based on where you are in your data -journey. +Use _dynamic mapping_ and _explicit mapping_ to define your data. Each method +provides different benefits based on where you are in your data journey. For +example, explicitly map fields where you don't want to use the defaults, or to +gain greater control over which fields are created. You can then allow {es} to +add other fields dynamically. NOTE: Before 7.0.0, the mapping definition included a type name. {es} 7.0.0 and later no longer accept a _default_ mapping. See <>. +.Experiment with mapping options +**** +<> to +experiment with different mapping options, and also fix mistakes in your index +mapping values by overriding values in the mapping during the search request. +**** + [discrete] [[mapping-dynamic]] == Dynamic mapping @@ -29,13 +38,11 @@ and explore data when you’re just getting started. {es} adds new fields automatically, just by indexing a document. You can add fields to the top-level mapping, and to inner <> and <> fields. -You can also dynamically map runtime fields: - -* <> so -that new fields are automatically added to the index mapping as runtime fields. -* <> to -experiment with different mapping options, and also fix mistakes in your index -mapping values by overriding values in the mapping during the search request. +Use <> to define custom mappings that are +applied to dynamically added fields based on the matching condition. You can +<> +so that new fields are automatically added to the index mapping as runtime +fields. [discrete] [[mapping-explicit]] @@ -49,10 +56,10 @@ define the mapping definition, such as: * Custom rules to control the mapping for <>. -You can <>, -which allows you to make schema changes without reindexing. You can use runtime -fields in conjunction with indexed fields to balance resource usage and -performance. Your index will be smaller, but with slower search performance. +<> allows you to make +schema changes without reindexing. You can use runtime fields in conjunction +with indexed fields to balance resource usage and performance. Your index will +be smaller, but with slower search performance. [discrete] [[mapping-limit-settings]] diff --git a/docs/reference/mapping/dynamic/templates.asciidoc b/docs/reference/mapping/dynamic/templates.asciidoc index d1252b8c5b13f..ce13b2a02bd83 100644 --- a/docs/reference/mapping/dynamic/templates.asciidoc +++ b/docs/reference/mapping/dynamic/templates.asciidoc @@ -132,12 +132,11 @@ PUT my-index-000001/_doc/1 ===== Mapping runtime fields in a dynamic template You can also map runtime fields in a dynamic template. For example, the following request adds a dynamic template named `dayOfWeek` that maps all -incoming `string` fields as `keyword` runtime fields. Although the `runtime` -definition is blank, new `string` fields will be mapped as `keyword` runtime -fields based on the <> that -{es} uses for adding field types to the mapping. Any `string` that doesn't pass -date detection or numeric detection is automatically mapped as a `keyword` when -`dynamic` is set to `runtime`. +incoming `string` fields as `keyword` fields in the `runtime` section of the +mapping. Although the `runtime` definition is blank, new `string` fields will +be mapped as `keyword` runtime fields based on the <> that {es} uses for adding field types to the +mapping. Any `string` that doesn't pass date detection or numeric detection is +automatically mapped as a `keyword` when `dynamic` is set to `runtime`. [source,console] ---- diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index 37da90aff7121..ec65a480e095a 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -14,6 +14,11 @@ sees runtime fields no differently. You can define runtime fields in the <>. Your choice, which is part of the inherent flexibility of runtime fields. +Runtime fields are useful when working with log data +(see <>), especially when you're unsure about the +data structure. Your search speed decreases, but your index size is much +smaller and you can more quickly process logs without having to index them. + [discrete] [[runtime-benefits]] === Benefits @@ -37,25 +42,6 @@ front, and can use runtime fields to amend the mapping at any time. Using runtime fields allows for a smaller index and faster ingest time, which combined use less resources and reduce your operating costs. -[discrete] -[[runtime-use-cases]] -=== Use cases -Runtime fields are useful when working with log data -(see <>), especially when you're unsure about the -data structure. Your search speed decreases, but your index size is much -smaller and you can more quickly process logs without having to index them. - -Runtime fields are especially useful in the following contexts: - -* Add fields to documents that are already indexed without having to reindex -data -* Start work on a new data stream without fully understanding the data it -contains -* Override the value of an indexed field with a runtime field to fix a mistake -after indexing documents -* Define fields that are only relevant for a particular context (such as a -visualization in {kib}) without influencing the underlying schema - [discrete] [[runtime-compromises]] === Compromises @@ -129,7 +115,22 @@ Runtime fields with a `type` of `date` can accept the If <> is enabled where the `dynamic` parameter is set to `runtime`, new fields are automatically added to -the index mapping as runtime fields. +the index mapping as runtime fields: + +[source,console] +---- +PUT my-index +{ + "mappings": { + "dynamic": "runtime", + "properties": { + "timestamp": { + "type": "date" + } + } + } +} +---- [[runtime-updating-scripts]] .Updating runtime scripts @@ -205,7 +206,7 @@ PUT my-index/ { "mappings": { "runtime": { - "newField": { + "model_number": { "type": "keyword" } } From f692f87cb71f8e2d5e6f658af46dd3989d0d089c Mon Sep 17 00:00:00 2001 From: Adam Locke Date: Thu, 7 Jan 2021 16:53:44 -0500 Subject: [PATCH 11/11] Adding description of for Painless script. --- docs/reference/mapping/runtime.asciidoc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/reference/mapping/runtime.asciidoc b/docs/reference/mapping/runtime.asciidoc index ec65a480e095a..9fcfaf4176f66 100644 --- a/docs/reference/mapping/runtime.asciidoc +++ b/docs/reference/mapping/runtime.asciidoc @@ -77,8 +77,11 @@ entire context of a document, including the original `_source` and any mapped fields plus their values. At query time, the script runs and generates values for each scripted field that is required for the query. -The script in the following request extracts the day of the week from the -`@timestamp` field, which is defined as a `date` type: +When defining a Painless script to use with runtime fields, you must include +`emit` to emit calculated values. For example, the script in the following +request extracts the day of the week from the `@timestamp` field, which is +defined as a `date` type. The script calculates the day of the week based on +the value of `timestamp`, and uses `emit` to return the calculated value. [source,console] ----