Add warmup settings documentation for pull-based ingestion - #12187
Conversation
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). |
kolchfa-aws
added
Doc review
kolchfa-aws
left a comment
kolchfa-aws
left a comment
There was a problem hiding this comment.
Thank you, @kaustubhbutte17! Please answer the clarifying questions so we can proceed with the doc review.
| | `internal_queue_size` | The size of the internal blocking queue for advanced tuning. Valid values are from 1 to 100,000, inclusive. Optional. Default is 100. | | ||
| | `all_active` | Whether to enable the all-active ingestion mode. Cannot be enabled for indexes that use segment replication mode. Default is `false`. See [Ingestion modes](#ingestion-modes). | | ||
| | `pointer_based_lag_update_interval` | The interval at which pointer-based lag is calculated. Accepts time units. Default is `10s`. Setting this value to `0` disables pointer-based lag calculation. | | ||
| | `warmup.timeout` | The maximum time to wait for the shard to catch up with the streaming source during the warmup phase after node restart or shard relocation. Shards will not serve queries until warmup completes or times out. A value of `-1` disables warmup. Accepts time units. Optional. Dynamic. Default is `-1` (disabled). | |
There was a problem hiding this comment.
Are other parameters in this table (except for the newly added ones) static (not updateable)? To update the newly added parameters, users should use the Update Settings API?
There was a problem hiding this comment.
Some entries in the table like error_strategy are dynamic, and some are static.
There was a problem hiding this comment.
Can you specify which ones are static so I can update all of them with static/dynamic?
There was a problem hiding this comment.
Here is the breakdown of static and dynamic properties
Dynamic:
error_strategypoll.max_batch_sizepoll.timeoutparamwarmup.timeoutwarmup.lag_threshold
Static:
typepointer.init.resetpointer.init.reset.valuenum_processor_threadsinternal_queue_sizeall_activepointer_based_lag_update_intervalmapper_type
Signed-off-by: Kaustubh Butte <kaustubhbutte17@gmail.com>
f9787f6 to
b0f6b09
Compare
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
kolchfa-aws
left a comment
kolchfa-aws
left a comment
There was a problem hiding this comment.
Thank you, @kaustubhbutte17! PR updated based on your comments.
…h-project#12187) * Add warmup settings documentation for pull-based ingestion Signed-off-by: Kaustubh Butte <kaustubhbutte17@gmail.com> * Doc review Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Kaustubh Butte <kaustubhbutte17@gmail.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Arya Soni <aryasoni98@gmail.com>
3 participants
Summary
warmup.timeoutandwarmup.lag_thresholdparameters to the ingestion source parameters table in pull-based ingestion docs.Description
The warmup phase prevents shards from serving queries until they have caught up with the streaming source after node restart or shard relocation. Two new dynamic settings control this behavior:
warmup.timeout: Maximum time to wait for lag to catch up. Default is-1(disabled).warmup.lag_threshold: Acceptable lag threshold for warmup completion. Default is100.Test plan
Tested locally
Signed-off-by: Kaustubh Butte kaustubhbutte17@gmail.com