Fix topic lifecycle and bound retained memory

[polRk]

What

Preserve topic delivery and shutdown guarantees while simplifying stream ownership and bounding retained client memory. Partial batches follow the flush timer, pending operations survive recoverable stream failures, and obsolete stream continuations cannot affect replacement connections. Public methods and types remain compatible.

Contracts

• Reader credit returns the server's complete ReadResponse.bytesSize once its final message is delivered or discarded. It is never reconstructed from individual payload sizes.
• Writer admission counts unacknowledged compressed payload bytes. Batching targets 48 MiB of payload; metadata and framing do not reduce this budget.
• flush() waits only for writes accepted before the call. Later writes cannot prolong it; independent boundaries survive reconnect and deduplication. It still rejects if its writes cannot be acknowledged. close() stops admission and drains the whole queue.
• RAW may borrow the input buffer. The same unchanged buffer can be submitted repeatedly, including before earlier ACKs. Its bytes must remain unchanged and its backing buffer must remain attached until every write using it is acknowledged. A flush only covers writes submitted before that call.
• Rediscovery changes selection for new unbound RPCs. Existing calls remain usable until they finish or their connection fails.

Changes

• Keep one writer message queue and sent-prefix boundary, cache unsent payload bytes, and capture each flush boundary by the last accepted message, without a second sequence-number counter. Calls at the same boundary share one completion. Send full batches immediately and partial batches on the flush timer. Retained messages become eligible after reconnect without restarting the batching delay.
• Use explicit stream ownership and ordered outgoing queues. Ignore stale readiness/token completions and centralize terminal cleanup. Reader token renewal continues when YDB omits an unchanged-token ACK; the writer retains its ACK gate.
• Use one topic retry policy for reader and writer. Recover server deadlines and conditionally retryable YDB failures; stop on deterministic gRPC frame-size failures while allowing temporary resource exhaustion to recover.
• Represent an initialized reader with a full retained window as waiting-credit, removing the redundant stored session-ID flag. Partition control, commits and token renewal remain active in this state.
• Preserve exact commit ranges across reconnect, reject stale partition callbacks, and retain delivered transactional offsets until transaction completion. Snapshot reader source filters so later caller mutation cannot change reconnect behavior.
• Retain encoded responses and decode the next requested batch. Release consumed queue references, obsolete partition entries, cancelled readiness waiters and terminal callbacks/codecs. Preserve unread data after clean reader close; discard it on destruction or failure.
• Snapshot writer metadata and timestamps, validate before admission, and handle rejected async acknowledgment observers without delaying ACK/commit completion.
• Preserve independent discovered/pinned channel caches, explicit TLS server names and active calls during retirement. Includes the previously merged core pin lifecycle fix.

Behavior change: callers that relied on later writes extending an existing flush() should use close() for a final whole-writer drain.

Payload budgets are not exact RSS limits. Decoded payload expansion, metadata and queued object counts remain separate concerns; this PR does not add estimated byte charges or reinterpret maxInflightCount.

Validation

Current audit:

• Full workspace unit run: 1,242 passed, including six independent flush-boundary scenarios and the updated reference model.
• After the flush contract change, all 59 topic integration/e2e tests pass on real YDB. The earlier core/query/topic run covered 95 tests: 94 passed initially, and the remaining stale-import failure passed after correcting the test import. No subsequent production core changes were made.
• Workspace build: 34 tasks passed. Lint passes with existing repository warnings.
• A real TCP proxy withholds ACK/commit responses and cuts sockets; the test verifies actual reconnects and all 30 committed payload identities. Rediscovery is tested separately over real gRPC, including an active call after GOAWAY.
• Negative controls reproduce permanent receive-size retry, partial-batch starvation, async observer rejection and mutable reader filters. Mutations of deadline classification, transaction-close success and discovery routing each fail their corresponding tests.
• Compared topic behavior and tests with Go 229af32f876ca3e96ab962436cd670cb2e174872 , Rust 0b83f5a1f63d758a73006e7d8389246ac20b683c and Java d854009713a2eb06e0145b14d20cdea1e288092b, including public contracts, ownership, retry, commits, partition lifecycle and issue-reported failure modes. Three Go topic packages pass with -race; Rust passes 116 topic unit and 18 controlled-gRPC protocol tests; Java passes 145 topic unit tests. Java comparisons additionally cover separate decoding budgets, control-event serialization, deferred commits and direct partition routing. These peer runs are not real-YDB qualification. JS regressions cover repeated RAW/GZIP input across reconnect and continued child delivery after a forced parent stop.

Earlier validation, retained as historical evidence rather than reruns of this head:

• Real-YDB 48 MiB payload boundaries, transaction commit/rollback, RAW/GZIP/ZSTD, metadata, flow control and memory reachability controls.
• A 390-second TLS/login run reconciled 2,859,015 accepted/ACK/read/committed messages and observed token renewal on the reader and all 24 writers after original-token expiry.
• Linux TLS memory profiles passed on Node 22/24/26 and Bun 1.4.3-canary.1+620b50f6a. Bun 1.4.2 failed the native-memory check; a separate HTTP/2 reproducer matched oven-sh/bun#42265. The canary result does not qualify Bun 1.4.2.

A fresh multi-hour runtime matrix and live split/merge under sustained high load remain outstanding. Current integration tests use local-ydb:25.3; inspected server source is a separate revision.

Checklist

• Changesets included
• Existing README and architecture documentation updated
• Public methods and type exports preserved

[github-actions]

🌋 SLO Test Results

🔴 4 workload(s) tested — 1 workload(s) exceeded failure thresholds

Commit: 558ddc9 · View run

Workload	Thresholds	Duration	Report
bun-kv-2dc	🟢 OK	15m 4s	📄 Report
node-kv-2dc	🟡 Warning	15m 4s	📄 Report
node-kv	🔴 Failure	15m 2s	📄 Report
bun-kv	🟢 OK	15m 2s	📄 Report

Threshold violations:

node-kv-2dc:

• read_retry_attempts: ▲ 47.9% (≥ 20% warn)

node-kv:

• read_retry_attempts: ▲ 60.7% (≥ 50% fail)

---

Generated by ydb-slo-action