Add metric for shard snapshot queue time by ywangd · Pull Request #143658 · elastic/elasticsearch

ywangd · 2026-03-05T06:55:20Z

Add a new histogram metric
es.repositories.snapshots.shards.queue_time.histogram that reports how long each shard snapshot spent waiting in queues before its actual operation began. The creation time is stored as a new field. Queue time is computed when the shard is dequeued in BlobStoreRepository.doSnapshotShard. This new field is an internal stat and not exposed to end users, i.e. not available in IndexShardSnapshotStatus.Copy.

Closes ES-14292

Add a new histogram metric es.repositories.snapshots.shards.queue_time.histogram that reports how long each shard snapshot spent waiting in queues before its actual operation began. The creation time is stored as a negated value in the existing startTimeMillis field of IndexShardSnapshotStatus, which is then overwritten by moveToStarted. This lets startTimeMillis > 0 reliably indicate that the snapshot has started, replacing the previous != 0 check. Queue time is computed when the shard is dequeued in BlobStoreRepository.doSnapshotShard. Closes ES-14292

elasticsearchmachine · 2026-03-05T06:55:44Z

Pinging @elastic/es-distributed (Team:Distributed)

ywangd · 2026-03-05T07:15:40Z

server/src/main/java/org/elasticsearch/index/snapshots/IndexShardSnapshotStatus.java

+    /**
+     * @param creationTimeMillis the time this status was created, used to compute queue time until the snapshot starts.
+     *                           Stored as a negative value in {@code startTimeMillis} so that {@code startTimeMillis > 0}
+     *                           reliably indicates that {@link #moveToStarted} has been called.
+     */
+    public static IndexShardSnapshotStatus newInitializing(ShardGeneration generation, long creationTimeMillis) {
+        return new IndexShardSnapshotStatus(Stage.INIT, -creationTimeMillis, 0L, 0, 0, 0, 0, 0, 0, null, generation, "initializing");


This avoids adding a new field to just track the queue time. Please let me know if you'd rather prefer a separate field.

I think we should add a new field, I think it's preferable from a maintainability perspective

ywangd · 2026-03-05T07:17:11Z

server/src/main/java/org/elasticsearch/index/snapshots/IndexShardSnapshotStatus.java

-            startTimeMillis,
+            startTimeMillis < 0 ? 0 : startTimeMillis,


The Copy ends up being visible to the users and can go across the wire. So maintaining the field being non-negative seems right. It really is an implementation detail.

coderabbitai · 2026-03-06T07:34:35Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (2)

Team:Delivery
Team:Search - Inference

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 503731d8-8a43-4dbc-aa35-6b4959eb9b07

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

nicktindall

Sorry, I think we should have a separate field. I assume that has BWC implications? but still I think it's worthwhile for readability and perhaps it'll become useful for more one day

nicktindall · 2026-03-11T05:24:26Z

server/src/main/java/org/elasticsearch/index/snapshots/IndexShardSnapshotStatus.java

+    /**
+     * @param creationTimeMillis the time this status was created, used to compute queue time until the snapshot starts.
+     *                           Stored as a negative value in {@code startTimeMillis} so that {@code startTimeMillis > 0}
+     *                           reliably indicates that {@link #moveToStarted} has been called.
+     */
+    public static IndexShardSnapshotStatus newInitializing(ShardGeneration generation, long creationTimeMillis) {
+        return new IndexShardSnapshotStatus(Stage.INIT, -creationTimeMillis, 0L, 0, 0, 0, 0, 0, 0, null, generation, "initializing");


I think we should add a new field, I think it's preferable from a maintainability perspective

ywangd · 2026-03-11T06:04:41Z

Yeah no problem. I updated with a separate field in 2951f9c

nicktindall

This LGTM, is zero a good default to use for creationTimeMillis in newDone? I know we won't use the creation time value for any metrics but I don't know what newDone statuses are used for (just wondering if startTime might make more sense?)

ywangd · 2026-03-11T08:29:23Z

is zero a good default to use for creationTimeMillis in newDone?

newDone creates a Copy used for transport actions. We don't want the new field to be visible to users. It's an internal tracking and also avoids any BWC by not showing it.

Add a new histogram metric `es.repositories.snapshots.shards.queue_time.histogram` that reports how long each shard snapshot spent waiting in queues before its actual operation began. The creation time is stored as a new field. Queue time is computed when the shard is dequeued in `BlobStoreRepository.doSnapshotShard`. This new field is an internal stat and not exposed to end users, i.e. not available in `IndexShardSnapshotStatus.Copy`. Closes ES-14292

ywangd requested a review from nicktindall March 5, 2026 06:55

ywangd added >non-issue :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v9.4.0 labels Mar 5, 2026

elasticsearchmachine added the Team:Distributed Meta label for distributed team. label Mar 5, 2026

asCopy

9ca28f0

ywangd commented Mar 5, 2026

View reviewed changes

ywangd added 4 commits March 5, 2026 18:41

Merge branch 'main' into ES-14292-snapshot-queue-time-metrics

ef99a6c

fix tests

da982f5

Merge branch 'main' into ES-14292-snapshot-queue-time-metrics

a58bd8a

Merge branch 'main' into ES-14292-snapshot-queue-time-metrics

f73dee9

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Mar 6, 2026

nicktindall reviewed Mar 11, 2026

View reviewed changes

ywangd added 2 commits March 11, 2026 16:41

Merge branch 'main' into ES-14292-snapshot-queue-time-metrics

249fcb1

separate field

2951f9c

ywangd requested a review from nicktindall March 11, 2026 06:04

nicktindall approved these changes Mar 11, 2026

View reviewed changes

Merge branch 'main' into ES-14292-snapshot-queue-time-metrics

544ad23

ywangd added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Mar 12, 2026

elasticsearchmachine merged commit 5152000 into elastic:main Mar 12, 2026
36 checks passed

ywangd deleted the ES-14292-snapshot-queue-time-metrics branch March 12, 2026 03:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metric for shard snapshot queue time#143658

Add metric for shard snapshot queue time#143658
elasticsearchmachine merged 9 commits intoelastic:mainfrom
ywangd:ES-14292-snapshot-queue-time-metrics

ywangd commented Mar 5, 2026 •

edited

Loading

Uh oh!

elasticsearchmachine commented Mar 5, 2026

Uh oh!

ywangd Mar 5, 2026

Uh oh!

nicktindall Mar 11, 2026

Uh oh!

ywangd Mar 5, 2026

Uh oh!

coderabbitai bot commented Mar 6, 2026

Review skipped

Uh oh!

nicktindall left a comment

Uh oh!

nicktindall Mar 11, 2026

Uh oh!

ywangd commented Mar 11, 2026

Uh oh!

nicktindall left a comment

Uh oh!

ywangd commented Mar 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ywangd commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 5, 2026

Uh oh!

ywangd Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

nicktindall Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

ywangd Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Mar 6, 2026

Review skipped

Uh oh!

nicktindall left a comment

Choose a reason for hiding this comment

Uh oh!

nicktindall Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

ywangd commented Mar 11, 2026

Uh oh!

nicktindall left a comment

Choose a reason for hiding this comment

Uh oh!

ywangd commented Mar 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ywangd commented Mar 5, 2026 •

edited

Loading