Set index template for ingest_mode: data_stream

[felixbarny]

Rolls back changes in #722 that broke the ingest_mode: data_stream.

Hopefully resolves #848

[felixbarny]

What's the best way to test this locally? Could someone recommend a rally command to invoke this?

This is a snippet of how I would start it with esbench but I'm having trouble converting this to a rally invocation for local execution:

  "elasticsearch.node.count": 1,
  "kibana.node.count": 0,
  "track.name": "tsdb",
  "include.tasks": [
    "tag:setup",
    "tag:health",
    "tag:index"
  ],
  "track.params": {
    "force_merge_max_num_segments": 1,
    "ingest_mode": "data_stream",
    "index_mode": "time_series",
    "source_mode": "synthetic",
    "bulk_indexing_clients": 16,
    "corpus": "split16",
    "run_esql_aggs": true
  },
  "stats.telemetry": [
    "disk-usage-stats"
  ]

[felixbarny]

I found out how to validate the changes locally:

esrally race --track-path=tsdb \
  --distribution-version=9.1.0 \
  --test-mode \
  --kill-running-processes \
  --track-params="force_merge_max_num_segments:1,ingest_mode:data_stream,index_mode:time_series,source_mode:synthetic,bulk_indexing_clients:16,corpus:split16,run_esql_aggs:true" 

This fails without this change and succeeds with it.

[martijnvg]

LGTM

[felixbarny]

There's a test failure for the tsdb track:

-------------------------------- live log call ---------------------------------
INFO     pytest_rally.rally:rally.py:144 Running command: [esrally race --track="tsdb" --track-repository="/home/runner/work/rally-tracks/rally-tracks" --track-revision="813c9815a440ef2b05240a4931e0e9caf7326994" --configuration-name="pytest" --enable-assertions --kill-running-processes --on-error="abort" --pipeline="benchmark-only" --target-hosts="127.0.0.1:19200" --test-mode --track-params="run_esql_aggs:True,index_mode:time_series"]
FAILED                                                                   [ 86%]

Not sure if this is related to my changes or not. Maybe it's due to recent changes in the TS command as run_esql_aggs:True is enabled. I'll create a dummy test PR to check if it fails for the same reason.

[felixbarny]

I looked at other recent PRs and they all fail for the same reason.

@kkrik-es @dnhatn any recent changes to TS that may have caused this?

[kkrik-es]

There have been a few.. do we have a stack trace or something?

[felixbarny]

think this has been resolved by elastic/elasticsearch#134603.

I've re-started the run but I already saw it succeeding in the build of another PR.

[gareth-ellis]

LGTM

[esbenchmachine]

@felixbarny
A backport is pending for this PR. Please add all required vX.Y version labels.

• If it is intended for the current Elasticsearch release version, apply the corresponding version label.
• If it also supports past released versions, add those labels too.
• If it only targets a future version, wait until that version label exists and then add it.
  (Each rally-tracks version label is created during the feature freeze of a new Elasticsearch branch).

Backporting entails:

1. Ensure the correct version labels exist in this PR.
2. Ensure backport PRs have backport label and are passing tests.
3. Merge backport PRs (you can approve yourself and enable auto-merge).
4. Remove backport pending label from this PR once all backport PRs are merged.

Thank you!