Add window-function variant (option 4) for list-named-blobs SQL

[zichengl]

Summary

• Adds mysql.named.blob.list.named.blobs.sql.option=4: a window-function variant of LIST_WITH_PREFIX_SQL that replaces option 3's correlated-subquery MAX(version) lookup with MAX(version) OVER (PARTITION BY blob_name) over a single PK range scan.
• Eliminates the per-row inner probe that amplifies transient TiKV coprocessor RPC degradation into multi-second LIST tail latency.
• Default unchanged (DEFAULT_LIST_NAMED_BLOBS_SQL_OPTION = 2). Opt-in per fabric — no behavior change for existing deployments until an operator flips the knob.
• Adds a targeted regression test for the deleted_ts-placement invariant shared by every LIST option.

Why

TiDB's plan for option 3 on named_blobs_v2 is Projection → IndexJoin (outer: StreamAgg over IndexLookup | inner: PointGet by (account_id, container_id, blob_name, MAX(version))). Every outer candidate row drives a per-row PK probe against TiKV to fetch MAX(version) — even when the probe returns zero rows, it inherits the full coprocessor RPC deadline. Under transient single-region degradation in TiKV this fans out: hundreds of zero-result probes stall on the same degraded region and the LIST tail latency rises from sub-second to multi-second (and worse under sustained degradation).

Option 4 runs a single PK range scan with a streaming window operator. No per-row inner probe; no deadline inheritance to amplify.

Empirical evidence

EXPLAIN ANALYZE on a production-replica TiDB against the checkpoints/% prefix on account_id=1437, container_id=4540 with LIMIT 10001:

	Wall clock	Plan time	cop_tasks	Rows pulled into TiDB	TiDB-side memory
Option 3 (current)	~2.0 s	—	—	—	—
Option 4 (this PR)	0.90 s	367 ms	20	215,347	175 MB

The plan tree for option 4 contains no IndexJoin. The operator chain is Projection → Limit → Selection → Window → TableReader → Selection (cop[tikv]) → TableRangeScan (cop[tikv]).

Correctness invariant — preserved and empirically verified

The (deleted_ts IS NULL OR deleted_ts > UTC_TIMESTAMP(6)) predicate lives on the outer WHERE, after the windowed MAX(version). This preserves option 3's exact semantics: when the latest READY version of a blob is expired or soft-deleted, the blob is hidden entirely; an older non-expired version never resurfaces.

During vetting we also tested an alternate form with the deleted_ts predicate pushed inside the windowed scan (0.26 s wall clock — faster, because TiKV gets to filter before shipping rows to TiDB). A set-difference query against the same production prefix showed 4 rows that the alternate form would have surfaced and option 4 correctly hides — older non-expired versions for blobs whose latest version has expired. Option 4 ships the correct-but-slightly-slower form deliberately; the alternate would leak stale blob versions to clients.

Testing Done

Local build & unit tests

• ./gradlew :ambry-api:compileJava :ambry-named-mysql:compileJava :ambry-named-mysql:compileTestJava :ambry-named-mysql:compileIntTestJava — BUILD SUCCESSFUL.
• ./gradlew :ambry-named-mysql:test — 11/11 unit tests pass.

New invariant test

testListHidesBlobWhenLatestVersionIsExpired in MySqlNamedBlobDbListOperationIntegrationTest:

• Constructs a blob with v1 (expiration in the future, not expired) and v2 (latest version, already expired).
• Asserts namedBlobDb.list(...) returns zero entries.
• Runs against options 2, 3, and 4 via the parametrized matrix, with enableHardDelete toggled on and off.
• Will fail loudly for any future "optimization" that moves the deleted_ts predicate inside the windowed/grouped MAX(version) scan.

Integration test matrix extended

@Parameterized.Parameters in MySqlNamedBlobDbListOperationIntegrationTest now runs every existing LIST scenario against options 2, 3, and 4 (× hard-delete on/off). The matrix is pinned to literals {2, 3, 4} rather than {MIN, MAX} so future range bumps cannot silently drop coverage of an option. Existing scenarios covered:

• Single READY version, single-record LIST.
• Multiple records under a shared prefix; ordering and pagination cursor.
• IN_PROGRESS blob while an older READY exists — LIST returns the older READY.
• Soft delete of every version of a blob — blob drops out of LIST.
• TTL-expired records under a prefix — not returned, LIMIT boundary respected.

Integration tests are deferred to CI (require live MySQL).

Production-replica verification

The EXPLAIN ANALYZE numbers above were captured against a production-replica TiDB on the actual checkpoints/% workload. The 4-row leak set-difference was also computed there.

Durability risk analysis

• Write path: not touched.
• Metadata storage: not touched. Same table, same rows, only SELECT shape changes.
• Ordering / atomicity: N/A — single-statement read.
• Callback semantics: same Page<NamedBlobRecord> return shape and content.
• Retries / idempotency: N/A — read.
• Partial failures: identical to today — query fails → exception bubbles → caller retries with same continuation token.
• Schema compatibility: no DDL change. Older code on the same DB rejects option=4 at config-validation time (getIntInRange) and falls back to its own configured option. New code on a DB written by older code is unaffected. No deployment-ordering constraint.
• Correctness regression: explicitly tested via the new invariant test, and empirically verified vs option 3 on a real production prefix (zero row-set delta between option 3 and option 4).

Rollout plan

1. Ship as opt-in. Default stays 2. PR is a no-op for every existing deployment.
2. Stage 1 — single fabric. Flip mysql.named.blob.list.named.blobs.sql.option=4 on prod-lor1 first: smallest blast radius and the fabric where the original incident was observed.
3. Monitor 48 h. TiDB statements_summary digest for option 3 should be replaced by a new digest for option 4; the new digest's p50/p99 latency must drop or stay flat. namedBlobListRate.<dc> flat ±2 %. Zero 10-second coprocessor-deadline timeouts on the new digest. No new error class in frontend logs.
4. Rollback path. Config push back to =3. Single line, no code revert needed.
5. Expand. Fabric-by-fabric on a multi-day cadence. No flips during merge freezes or release cuts.

Out of scope (deliberate)

• LIST_ALL_QUERY (the no-prefix LIST path) is not rewritten in this PR. It uses an INNER-JOIN+MAX(version) shape that may carry similar plan-risk and possibly a related deleted_ts-placement question of its own — tracking as a follow-up so this PR stays surgical and easy to review.
• The default option flip is deliberately not part of this PR.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.21%. Comparing base (52ba813) to head (06b3103).
⚠️ Report is 391 commits behind head on master.

Files with missing lines	Patch %	Lines
.../java/com/github/ambry/named/MySqlNamedBlobDb.java	86.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3260       +/-   ##
=============================================
- Coverage     64.24%   51.21%   -13.03%     
+ Complexity    10398     8675     -1723     
=============================================
  Files           840      931       +91     
  Lines         71755    79540     +7785     
  Branches       8611     9525      +914     
=============================================
- Hits          46099    40739     -5360     
- Misses        23004    35416    +12412     
- Partials       2652     3385      +733     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.