Default to @timestamp in composable template datastream definition by andreidan · Pull Request #59317 · elastic/elasticsearch

andreidan · 2020-07-09T16:20:41Z

This removes the data_stream timestamp field specification when
defining a composable template.
The timestamp field is locked to @timestamp for which we provide a default
mapping of type date with default options (added with #59244 ), but the user
can still provide a custom mapping in the template.

The data_stream definition in the composable template is now an empty object:

data_stream: {}

Relates to #53100

This makes the data_stream timestamp field specification optional when defining a composable template. When there isn't one specified it will default to `@timestamp`.

elasticmachine · 2020-07-09T16:20:43Z

Pinging @elastic/es-core-features (:Core/Features/Data streams)

elasticmachine · 2020-07-09T16:20:45Z

Pinging @elastic/es-docs (>docs)

jrodewig

Docs LGTM.

Since @timestamp will soon be both the required and default timestamp field, I don't see much point in telling users to supply a non-empty data_stream object body, even optionally. Let me know if there is some context I'm missing tho. Thanks!

docs/reference/data-streams/set-up-a-data-stream.asciidoc

jrodewig · 2020-07-09T16:35:26Z

docs/reference/indices/index-templates.asciidoc

+(Optional, string) The name of the timestamp field. This field must be present
 in all documents indexed into the data stream and must be of type
-<<date, `date`>> or <<date_nanos, `date_nanos`>>.
+<<date, `date`>> or <<date_nanos, `date_nanos`>>. Defaults to `@timestamp`.


I'd change the last sentence to:

Must be @timestamp, which is the default.

dakrone · 2020-07-09T16:43:39Z

I don't see much point in telling users to supply a data_stream object body, even optionally. Let me know if there is some context I'm missing tho. Thanks!

@jrodewig if a user doesn't supply the empty data_stream: {} object, the template will create regular indices, instead of creating a data stream when documents are indexed that are matched

jrodewig · 2020-07-09T16:49:51Z

@dakrone Sorry for the confusion. I wasn't clear in my original comment. I meant that we should just tell users to supply:

  "data_stream": { }

Instead of:

  "data_stream": {
    "timestamp_field": "@timestamp"
  }

The defs produce the same results now so there is no point of even telling them about the timestamp_field param, even as an option.

andreidan · 2020-07-09T17:19:05Z

The defs produce the same results now so there is no point of even telling them about the timestamp_field param, even as an option.

@jrodewig I agree it's rather redundant. However I believe the plan is to eventually allow a custom timestamp field configuration.

Happy to keep it simple for now and not mention it if that's your preference though.

@martijnvg what do you think?

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

andreidan · 2020-07-09T18:00:21Z

@elasticmachine update branch

jtibshirani · 2020-07-09T21:04:38Z

I meant that we should just tell users to supply: "data_stream": { }

+1, and I think we shouldn't even allow the timestamp_field to be specified in the API. We decided to postpone support for a custom timestamp field name, and we don't yet know exactly how it will look. For example, one option is that instead of specifying timestamp_field, users just specify @timestamp as a field alias to the concrete timestamp field.

andreidan · 2020-07-13T08:47:02Z

@elasticmachine update branch

martijnvg

Looks good. I left a question

martijnvg · 2020-07-13T08:58:26Z

server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java


        static {
-            PARSER.declareString(ConstructingObjectParser.constructorArg(), DataStream.TIMESTAMP_FIELD_FIELD);
+            PARSER.declareString(ConstructingObjectParser.optionalConstructorArg(), DataStream.TIMESTAMP_FIELD_FIELD);


maybe remove this field from the xcontent format? So that the only valid format is an empty json object?

Yes, good point. i'll drop the field completely in the POJO but keep the data_stream empty JSON object

@timestamp

This will always be @timestamp

…lastic#59317) This makes the data_stream timestamp field specification optional when defining a composable template. When there isn't one specified it will default to `@timestamp`. (cherry picked from commit 5609353) Signed-off-by: Andrei Dan <andrei.dan@elastic.co>

…59317) (#59516) This makes the data_stream timestamp field specification optional when defining a composable template. When there isn't one specified it will default to `@timestamp`. (cherry picked from commit 5609353) Signed-off-by: Andrei Dan <andrei.dan@elastic.co>

This reverts commit 909de77.

cjcenizal · 2020-07-14T16:05:37Z

@andreidan The PR description states that this PR "makes the data_stream timestamp field specification optional", but it looks like it actually removes it completely. Could you please clarify or update the PR description if it's incorrect?

Removes the `@timestamp` field mapping from several data stream index template snippets. With #59317, the `@timestamp` field defaults to a `date` field data type for data streams.

andreidan · 2020-07-14T16:19:37Z

@cjcenizal thanks for pointing it out. changed the description to reflect the actual/final behaviour.

elastic#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

…9553) Removes the `@timestamp` field mapping from several data stream index template snippets. With #59317, the `@timestamp` field defaults to a `date` field data type for data streams.

elastic#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

Default to @timestamp in composable template datastream definition

8f6c5da

This makes the data_stream timestamp field specification optional when defining a composable template. When there isn't one specified it will default to `@timestamp`.

andreidan added >docs General docs changes v8.0.0 :StorageEngine/Data streams Data streams and their lifecycles v7.9.0 labels Jul 9, 2020

elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Jul 9, 2020

elasticmachine added the Team:Docs Meta label for docs team label Jul 9, 2020

jrodewig approved these changes Jul 9, 2020

View reviewed changes

Fix yaml test

fb6b62a

andreidan requested a review from martijnvg July 9, 2020 17:17

andreidan and others added 2 commits July 9, 2020 18:59

Update index templates doc

65c1d5d

Mention only the empty body data_stream

9646bdc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

elasticmachine and others added 2 commits July 9, 2020 12:00

Merge branch 'master' into default-ds-timestamp-definition

c920a53

Use the empty data_stream definition in yaml tests

cce093d

jtibshirani mentioned this pull request Jul 9, 2020

Fix the timestamp field of a data stream to @timestamp #59076

Merged

Disablle more yaml tests in 7.9

8d1d819

Merge branch 'master' into default-ds-timestamp-definition

69a34f7

martijnvg reviewed Jul 13, 2020

View reviewed changes

andreidan added 4 commits July 13, 2020 14:00

Remove timestamp_field from data_stream definition

d55060d

This will always be @timestamp

Handle equals and hashcode for empty DataStreamTemplate

98b23e5

Update xpack templates to not specify the timestamp_field

1a73aaf

Merge branch 'master' into default-ds-timestamp-definition

0bc0ae4

andreidan added 3 commits July 14, 2020 09:15

Fix DataStreamsUpgradeIT

3c737c2

Merge branch 'master' into default-ds-timestamp-definition

fc7fb1e

Fix merge

d75db9e

andreidan merged commit 5609353 into elastic:master Jul 14, 2020

This was referenced Jul 14, 2020

[7.x] Default to @timestamp in composable template datastream definition (#59317) #59516

Merged

Fix sentence in data stream docs #59518

Merged

andreidan added a commit to andreidan/elasticsearch that referenced this pull request Jul 14, 2020

Re-enable bwc after backport of elastic#59317

909de77

andreidan mentioned this pull request Jul 14, 2020

Re-enable bwc after backport of #59317 #59519

Merged

andreidan added a commit to danhermann/elasticsearch that referenced this pull request Jul 14, 2020

Revert "Re-enable bwc after backport of elastic#59317"

8522c95

This reverts commit 909de77.

andreidan mentioned this pull request Jul 14, 2020

Data Streams: fix bwc test #59528

Merged

jrodewig mentioned this pull request Jul 14, 2020

[DOCS] Simplify index template snippets for data streams #59533

Merged

andreidan added the Team:Deployment Management Meta label for Management Experience - Deployment Management team label Jul 14, 2020

tylersmalley mentioned this pull request Jul 14, 2020

Remove support for timestamp_field in data stream elastic/kibana#71670

Closed

danhermann pushed a commit that referenced this pull request Jul 14, 2020

Re-enable bwc after backport of #59317 (#59519)

4f47227

tylersmalley added a commit to tylersmalley/kibana that referenced this pull request Jul 14, 2020

Removes timestamp_field from data_stream

1ebfa62

elastic#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley mentioned this pull request Jul 14, 2020

Removes timestamp_field from data_stream elastic/kibana#71727

Merged

elasticmachine pushed a commit to mistic/kibana that referenced this pull request Jul 15, 2020

Removes timestamp_field from data_stream (elastic#71727)

6068285

elastic#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley added a commit to tylersmalley/kibana that referenced this pull request Jul 15, 2020

Removes timestamp_field from data_stream (elastic#71727)

9904aaf

elastic#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley added a commit to tylersmalley/kibana that referenced this pull request Jul 15, 2020

Removes timestamp_field from data_stream (elastic#71727)

3f3cc07

elastic#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley added a commit to elastic/kibana that referenced this pull request Jul 15, 2020

Removes timestamp_field from data_stream (#71727) (#71896)

300458c

#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

tylersmalley added a commit to elastic/kibana that referenced this pull request Jul 15, 2020

Removes timestamp_field from data_stream (#71727) (#71895)

8b9c580

#71670 Caused by elastic/elasticsearch#59317 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Conversation

andreidan commented Jul 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Jul 9, 2020

Uh oh!

elasticmachine commented Jul 9, 2020

Uh oh!

jrodewig left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jrodewig Jul 9, 2020

Choose a reason for hiding this comment

Uh oh!

dakrone commented Jul 9, 2020

Uh oh!

jrodewig commented Jul 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreidan commented Jul 9, 2020

Uh oh!

andreidan commented Jul 9, 2020

Uh oh!

jtibshirani commented Jul 9, 2020

Uh oh!

andreidan commented Jul 13, 2020

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

martijnvg Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

andreidan Jul 13, 2020

Choose a reason for hiding this comment

Uh oh!

cjcenizal commented Jul 14, 2020

Uh oh!

andreidan commented Jul 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

andreidan commented Jul 9, 2020 •

edited

Loading

jrodewig left a comment •

edited

Loading

jrodewig commented Jul 9, 2020 •

edited

Loading