Skip to content

Add synthetic_source_keep param to tsdb and http_logs tracks #683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
martijnvg merged 2 commits into from
Oct 2, 2024
Merged

Add synthetic_source_keep param to tsdb and http_logs tracks #683

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
897d032
Add synthetic_source_keep param to tsdb and http_logs tracks
martijnvg Sep 30, 2024
b202a94
Fix small mistake in elastic/logs template when no synthetic_source_k…
martijnvg Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions elastic/logs/templates/component/track-shared-logsdb-mode.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
"settings": {
{% if index_mode %}
"index": {
"mode": {{ index_mode | tojson }},
"mode": {{ index_mode | tojson }}
{% if synthetic_source_keep and synthetic_source_keep != 'none' %}
"mapping": {
,"mapping": {
"synthetic_source_keep": "{{ synthetic_source_keep }}"
}
{% endif %}
Expand Down
1 change: 1 addition & 0 deletions http_logs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ node pipeline to run. Valid options are `'baseline'` (default), `'grok'` and `'
* `post_ingest_sleep_duration` (default: 30): Sleep duration in seconds.
* `enable_logsdb` (default: false) Determines whether the logsdb index mode gets used. If set then index sorting is configured to only use `@timestamp` field and the `source_enabled` parameter will have no effect.
* `force_merge_max_num_segments` (default: unset): An integer specifying the max amount of segments the force-merge operation should use.
* `synthetic_source_keep` (default: unset): If specified, configures the `index.mapping.synthetic_source_keep` index setting.

### License

Expand Down
4 changes: 3 additions & 1 deletion http_logs/index.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
{%- endif -%}{# non-serverless-index-settings-marker-end #}
{%- if enable_logsdb %},
"index.mode": "logsdb",
"index.mapping.synthetic_source_keep": "arrays",
{% if synthetic_source_keep %}
"index.mapping.synthetic_source_keep": "{{synthetic_source_keep}}",
{% endif %}
"index.sort.field": ["@timestamp"],
"index.sort.order":["desc"]
{%- endif %}
Expand Down
1 change: 1 addition & 0 deletions tsdb/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,7 @@ This track allows to overwrite the following parameters using `--track-params`:
* `post_ingest_sleep` (default: false): Whether to pause after ingest and prior to subsequent operations.
* `post_ingest_sleep_duration` (default: 30): Sleep duration in seconds.
* `document_ids`: documentd IDs to use for search, get and mget apis in the `low-latency` challenge. If empty, a default set of 4 values is used.
* `synthetic_source_keep` (default: unset): If specified, configures the `index.mapping.synthetic_source_keep` index setting.

### License

Expand Down
7 changes: 6 additions & 1 deletion tsdb/index-template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,12 @@
"end_time": "2021-05-01T00:00:00.000Z"
},
{% endif %}
"mapping.total_fields.limit": 10000
"mapping": {
{% if synthetic_source_keep %}
"synthetic_source_keep": "{{synthetic_source_keep}}",
{% endif %}
"total_fields.limit": 10000
}
}
},
"mappings": {
Expand Down
7 changes: 6 additions & 1 deletion tsdb/index.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,12 @@
"end_time": "9999-01-01T00:00:00.000Z"
},
{% endif %}
"mapping.total_fields.limit": 10000
"mapping": {
{% if synthetic_source_keep %}
"synthetic_source_keep": "{{synthetic_source_keep}}",
{% endif %}
"total_fields.limit": 10000
}
}
},
"mappings": {
Expand Down