Skip to content

[DOCS] Runtime fields improvements #66530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lockewritesdocs merged 13 commits into from
Jan 11, 2021
Merged

[DOCS] Runtime fields improvements #66530

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
e3c939d
Changes for dynamic templates.
Dec 16, 2020
e0368fd
Clarifying language around dynamic:true and dynamic:runtime.
Dec 17, 2020
d1a5120
Clarifying edits and some restructuring.
Dec 17, 2020
581946e
Merge branch 'master' into docs__runtime-fields-improvements
elasticmachine Jan 4, 2021
245a8ca
Overhauling the Mapping page.
Jan 4, 2021
1f30c45
Incorporating changes from #66911.
Jan 5, 2021
901b78d
Reworking mapping page to focus on dynamic vs. explicit mapping.
Jan 5, 2021
176e3a5
Merge branch 'master' into docs__runtime-fields-improvements
elasticmachine Jan 6, 2021
a6adf7b
Reordering to fix test failure.
Jan 6, 2021
97f98ec
Further clarifying mapping page.
Jan 6, 2021
d99eec8
Reordering sections, adding headings to examples, and other clarifica…
Jan 6, 2021
2323993
Incorporating review feedback.
Jan 7, 2021
f692f87
Adding description of for Painless script.
Jan 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
255 changes: 58 additions & 197 deletions docs/reference/mapping.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,61 @@
--

Mapping is the process of defining how a document, and the fields it contains,
are stored and indexed. For instance, use mappings to define:
are stored and indexed.

Each document is a collection of fields, which each have their own
<<mapping-types,data type>>. When mapping your data, you create a mapping
definition, which contains a list of fields that are pertinent to the document.
A mapping definition also includes <<mapping-fields,metadata fields>>, like the
`_source` field, which customize how a document's associated metadata is
handled.

Use _dynamic mapping_ and _explicit mapping_ to define your data. Each method
provides different benefits based on where you are in your data journey. For
example, explicitly map fields where you don't want to use the defaults, or to
gain greater control over which fields are created. You can then allow {es} to
add other fields dynamically.

NOTE: Before 7.0.0, the mapping definition included a type name.
{es} 7.0.0 and later no longer accept a _default_ mapping. See <<removal-of-types>>.

.Experiment with mapping options
****
<<runtime-search-request,Define runtime fields in a search request>> to
experiment with different mapping options, and also fix mistakes in your index
mapping values by overriding values in the mapping during the search request.
****

* which string fields should be treated as full text fields.
* which fields contain numbers, dates, or geolocations.
* the <<mapping-date-format,format>> of date values.
* custom rules to control the mapping for
<<dynamic-mapping,dynamically added fields>>.

A mapping definition includes metadata fields and fields:

<<mapping-fields,Metadata fields>>::

Metadata fields are used to customize how a document's associated metadata is
treated. Examples of metadata fields include the document's
<<mapping-index-field,`_index`>>, <<mapping-id-field,`_id`>>, and
<<mapping-source-field,`_source`>> fields.
[discrete]
[[mapping-dynamic]]
== Dynamic mapping
<<dynamic-field-mapping,Dynamic mapping>> allows you to experiment with
and explore data when you’re just getting started. {es} adds new fields
automatically, just by indexing a document. You can add fields to the top-level
mapping, and to inner <<object,`object`>> and <<nested,`nested`>> fields.

<<mapping-types,Fields>>::
Use <<dynamic-templates,dynamic templates>> to define custom mappings that are
applied to dynamically added fields based on the matching condition. You can
<<match-mapping-runtime-fields,map runtime fields in a dynamic template>>
so that new fields are automatically added to the index mapping as runtime
fields.

A mapping contains a list of fields or `properties` pertinent to the
document. Each field has its own <<mapping-types, data type>>.
[discrete]
[[mapping-explicit]]
== Explicit mapping
<<explicit-mapping,Explicit mapping>> allows you to precisely choose how to
define the mapping definition, such as:

* Which string fields should be treated as full text fields.
* Which fields contain numbers, dates, or geolocations.
* The <<mapping-date-format,format>> of date values.
* Custom rules to control the mapping for
<<dynamic-mapping,dynamically added fields>>.

NOTE: Before 7.0.0, the 'mappings' definition used to include a type name.
For more details, please see <<removal-of-types>>.
<<runtime-mapping-fields,Explicitly mapping runtime fields>> allows you to make
schema changes without reindexing. You can use runtime fields in conjunction
with indexed fields to balance resource usage and performance. Your index will
be smaller, but with slower search performance.

[discrete]
[[mapping-limit-settings]]
Expand All @@ -45,190 +76,20 @@ Use the <<mapping-settings-limit,mapping limit settings>> to limit the number
of field mappings (created manually or dynamically) and prevent documents from
causing a mapping explosion.

[discrete]
[[runtime-fields]]
== Runtime fields
Typically, you index data into {es} to promote faster search. However, indexing
can be slow and requires more disk space, and you have to reindex your data to
add fields to existing documents.

<<runtime,Runtime fields>> are not indexed, which saves disk space and makes
data ingest faster. You can add runtime fields to existing documents without
reindexing your data and calculate field values dynamically at search time.

[discrete]
[[dynamic-mapping-intro]]
== Dynamic mapping

Fields and mapping types do not need to be defined before being used. Thanks
to _dynamic mapping_, new field names will be added automatically, just by
indexing a document. New fields can be added both to the top-level mapping
type, and to inner <<object,`object`>> and <<nested,`nested`>> fields.

The <<dynamic-mapping,dynamic mapping>> rules can be configured to customise
the mapping that is used for new fields.

[discrete]
== Explicit mappings

You know more about your data than Elasticsearch can guess, so while dynamic
mapping can be useful to get started, at some point you will want to specify
your own explicit mappings.

You can create field mappings when you <<create-mapping,create an index>> and
<<add-field-mapping,add fields to an existing index>>.

[discrete]
[[create-mapping]]
=== Create an index with an explicit mapping

You can use the <<indices-create-index,create index>> API to create a new index
with an explicit mapping.

[source,console]
----
PUT /my-index-000001
{
"mappings": {
"properties": {
"age": { "type": "integer" }, <1>
"email": { "type": "keyword" }, <2>
"name": { "type": "text" } <3>
}
}
}
----

<1> Creates `age`, an <<number,`integer`>> field
<2> Creates `email`, a <<keyword,`keyword`>> field
<3> Creates `name`, a <<text,`text`>> field

[discrete]
[[add-field-mapping]]
== Add a field to an existing mapping

You can use the <<indices-put-mapping, put mapping>> API to add one or more new
fields to an existing index.

The following example adds `employee-id`, a `keyword` field with an
<<mapping-index,`index`>> mapping parameter value of `false`. This means values
for the `employee-id` field are stored but not indexed or available for search.

[source,console]
----
PUT /my-index-000001/_mapping
{
"properties": {
"employee-id": {
"type": "keyword",
"index": false
}
}
}
----
// TEST[continued]

[discrete]
[[update-mapping]]
=== Update the mapping of a field

include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=change-field-mapping]

include::{es-repo-dir}/indices/put-mapping.asciidoc[tag=rename-field]

[discrete]
[[view-mapping]]
== View the mapping of an index

You can use the <<indices-get-mapping, get mapping>> API to view the mapping of
an existing index.

[source,console]
----
GET /my-index-000001/_mapping
----
// TEST[continued]

The API returns the following response:

[source,console-result]
----
{
"my-index-000001" : {
"mappings" : {
"properties" : {
"age" : {
"type" : "integer"
},
"email" : {
"type" : "keyword"
},
"employee-id" : {
"type" : "keyword",
"index" : false
},
"name" : {
"type" : "text"
}
}
}
}
}
----


[discrete]
[[view-field-mapping]]
== View the mapping of specific fields

If you only want to view the mapping of one or more specific fields, you can use
the <<indices-get-field-mapping, get field mapping>> API.

This is useful if you don't need the complete mapping of an index or your index
contains a large number of fields.

The following request retrieves the mapping for the `employee-id` field.

[source,console]
----
GET /my-index-000001/_mapping/field/employee-id
----
// TEST[continued]

The API returns the following response:

[source,console-result]
----
{
"my-index-000001" : {
"mappings" : {
"employee-id" : {
"full_name" : "employee-id",
"mapping" : {
"employee-id" : {
"type" : "keyword",
"index" : false
}
}
}
}
}
}

----

--

include::mapping/removal_of_types.asciidoc[]

include::mapping/mapping-settings-limit.asciidoc[]
include::mapping/dynamic-mapping.asciidoc[]

include::mapping/types.asciidoc[]
include::mapping/explicit-mapping.asciidoc[]

include::mapping/runtime.asciidoc[]

include::mapping/types.asciidoc[]

include::mapping/fields.asciidoc[]

include::mapping/params.asciidoc[]

include::mapping/dynamic-mapping.asciidoc[]
include::mapping/mapping-settings-limit.asciidoc[]

include::mapping/removal_of_types.asciidoc[]
9 changes: 4 additions & 5 deletions docs/reference/mapping/dynamic-mapping.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
[[dynamic-mapping]]
== Dynamic Mapping
== Dynamic mapping

One of the most important features of Elasticsearch is that it tries to get
One of the most important features of {es} is that it tries to get
out of your way and let you start exploring your data as quickly as possible.
To index a document, you don't have to first create an index, define a mapping
type, and define your fields -- you can just index a document and the index,
type, and fields will spring to life automatically:
type, and fields will display automatically:

[source,console]
--------------------------------------------------
Expand All @@ -17,7 +17,7 @@ PUT data/_doc/1 <1>
called `count` with data type `long`.

The automatic detection and addition of new fields is called
_dynamic mapping_. The dynamic mapping rules can be customised to suit your
_dynamic mapping_. The dynamic mapping rules can be customized to suit your
purposes with:

<<dynamic-field-mapping,Dynamic field mappings>>::
Expand All @@ -35,4 +35,3 @@ automatically or explicitly.
include::dynamic/field-mapping.asciidoc[]

include::dynamic/templates.asciidoc[]

39 changes: 24 additions & 15 deletions docs/reference/mapping/dynamic/field-mapping.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,36 +2,45 @@
=== Dynamic field mapping

When {es} detects a new field in a document, it _dynamically_ adds the field to
the type mapping. The <<dynamic,`dynamic`>> parameter controls this behavior.
the type mapping by default. The <<dynamic,`dynamic`>> parameter controls this behavior.

You can disable dynamic mapping, both at the document and at the
<<object,`object`>> level. Setting the `dynamic` parameter to
`false` ignores new fields, and `strict` rejects the document if {es}
encounters an unknown field.

When dynamic field mapping is enabled, {es} uses the following rules to
You can explicitly instruct {es} to dynamically create fields based on incoming
documents by setting the `dynamic` parameter to `true` or `runtime`. When
dynamic field mapping is enabled, {es} uses the rules in the following table to
determine how to map data types for each field.

NOTE: These are the only <<mapping-types,field data types>> that {es} detects
dynamically. All other data types must be mapped explicitly.
NOTE: The field data types in the following table are the only
<<mapping-types,field data types>> that {es} detects dynamically. You must
explicitly map all other data types.

[[dynamic-field-mapping-types]]
[cols="3"]
|===
h| JSON data type h| `dynamic:true` h| `dynamic:runtime`
|`null` 2*| No field added
|`true` or `false` 2*| `boolean`
|floating point number | `float` | `double`
|`double` | `float` | `double`
|`integer` 2*| `long`
|`object`<<dynamic-object-footnote,^1^>> 2*| `object`
|`array` 2*| Depends on the first non-`null` value in the array
|`string` that passes <<date-detection,date detection>> 2*| `date`
|`string` that passes <<numeric-detection,numeric detection>> | `double` or `long` | `double`
|`string` that passes <<numeric-detection,numeric detection>> | `float` or `long` | `double` or `long`
|`string` that doesn't pass `date` detection or `numeric` detection | `text` with a `.keyword` sub-field | `keyword`
3+| [[dynamic-object-footnote]] ^1^Objects are always mapped as part of the `properties` section, even when the `dynamic` parameter is set to `runtime`. | |
|===

Besides the following options, you can customize dynamic field mapping rules
with <<dynamic-templates,`dynamic_templates`>>.
You can disable dynamic mapping, both at the document and at the
<<object,`object`>> level. Setting the `dynamic` parameter to
`false` ignores new fields, and `strict` rejects the document if {es}
encounters an unknown field.

TIP: Use the <<indices-put-mapping,PUT mapping API>> to update the `dynamic`
setting on existing fields.

You can customize dynamic field mapping rules for
<<date-detection,date detection>> and <<numeric-detection,numeric detection>>.
To define custom mappings rules that you can apply to additional dynamic
fields, use <<dynamic-templates,`dynamic_templates`>>.

[[date-detection]]
==== Date detection
Expand Down Expand Up @@ -83,9 +92,9 @@ PUT my-index-000001/_doc/1 <1>

<1> The `create_date` field has been added as a <<text,`text`>> field.

===== Customising detected date formats
===== Customizing detected date formats

Alternatively, the `dynamic_date_formats` can be customised to support your
Alternatively, the `dynamic_date_formats` can be customized to support your
own <<mapping-date-format,date formats>>:

[source,console]
Expand Down
Loading