[NSTA1]

What this fixes

Nothing in the repocontext host ever removed a row from OrleansStorage.

• The deployed OrleansQuery catalogue contains no grain-state delete at all. ClearStorageKey is an UPDATE that nulls PayloadBinary, stamps ModifiedOn and increments Version, keeping the row. The only DELETE FROM OrleansStorage... anywhere in the schema targets the temp OrleansStorageWriteState bookkeeping table inside WriteToStorageKey.
• AdoNetGrainStorageOptions.DeleteStateOnClear sat at the Orleans default of false and was referenced nowhere in the repository.

Measured on the deployed container, 29.8% of OrleansStorage rows (18,810 of 63,092) were dead, accruing at roughly 3,000 a day.

The mechanism that makes this unbounded is key shape, not grain type. A grain keyed stably rewrites one row in place, so clear-without-delete costs it at most one dead row - leaf-snapshot sits at Version 30-66 for exactly that reason. A grain keyed generationally never revisits a key, so every generation it has ever taken leaves a permanent row: leaf-snapshot-segment keys on a per-leaf generation counter and all 19 generations coexist in the live database, with generations 10, 11 and 15-19 holding zero live bytes.

What this does NOT fix, stated so nothing here is read as claiming it

This does not reclaim disk and the database will not shrink. The file is 97.7% live payload (page_size 4096 x page_count 1,283,605 = 5,257,646,080 bytes against SUM(LENGTH(PayloadBinary)) = 5,136,292,251), and freelist_count is 1,990 pages, i.e. 8.1 MB. A cleared row holds zero payload bytes by construction, so the tombstones cost fixed columns plus an index entry - order 250 MB/year. The gigabytes are live leaf-snapshot data and are unaffected by this change.

What it does bound is row count (about 18x table inflation a year), IX_OrleansStorage depth and scan cost, and the legibility of the store - at present you cannot tell a live grain from a dead one.

It also does not remove the rows already tombstoned. A stable-key row is reclaimed on its next rewrite; a generational-key row never is. A one-time scrub is a bulk delete against a live 4.9 GB database with a different risk profile and belongs in its own issue.

The change

Two coupled edits that must land together.

1. apps/repocontext/Persistence/Sqlite/SQLite-Persistence.sql gains a DeleteStorageKey query.
2. apps/repocontext/Hosting/DurabilitySelector.cs enables DeleteStateOnClear on the SQLite and Azure Table branches.

They are coupled because Orleans resolves the query key literal DeleteStorageKey during AdoNetGrainStorage.Init - it is one of the four keys selected by the public AdoNetGrainStorage.DefaultInitializationQuery constant - and throws when the option is set and the key is absent. That fails silo startup, not the first clear, so enabling the option alone would produce a container that does not boot. ClearStorageKey is retained because Orleans resolves it unconditionally either way.

No migration step is needed. SqliteSchemaInitializer reapplies the embedded script on every start and every query is INSERT OR REPLACE, so an existing database picks the new key up on the next deploy.

Why not upstream's query text

Orleans' PostgreSQL form is DELETE ... RETURNING Version + 1, which emits one row per deleted row. IX_OrleansStorage is deliberately non-unique, so against a grain identity that had acquired a duplicate row that form hands Orleans two rows and its SingleOrDefault() throws InvalidOperationException("Sequence contains more than one element") - reintroducing precisely the wedge #3116 removed from this file. The new query computes the version as a scalar, matching the local shape already used by WriteToStorageKey and ClearStorageKey, and returns exactly one row however many were deleted.

Decision on the three storage branches

Branch	DeleteStateOnClear	Why
SQLite	true	This repository owns the schema and reapplies it on every start, so the query cannot be missing. This is the deployed branch and where the evidence was taken.
Azure Table	true	The provider honours the option directly against the table API. There is no query catalogue and so no coupling to satisfy, and the growth is a property of the grain keys rather than the store, so the same defect would apply to an Azure deployment.
Postgres	false, deliberately	This host ships and applies no Postgres schema; the catalogue is operator-provisioned and this host never reads or repairs it. Enabling the option would convert an absent query in a store outside this repository's control into a silo that fails to start. Orleans only aligned its own PostgreSQL-Persistence.sql with its 3.6.0 migration in dotnet/orleans#11247, so presence is both version- and provisioning-path-dependent. The reason and the opt-in recipe are recorded in a comment beside the arm, and pinned by a test.

The two leads from the issue

Both are explained, and both turn out to be the same mechanism as the headline defect rather than separate defects.

snapshot-baseline, 100% dead (3,776/3,776), accruing ~1,100/day. Not a second defect. SnapshotLeafGrain.BuildBaselineKey is {treeId}/{shardIndex}/{baselineToken:N} - a per-cursor GUID, so the key is generational in the same sense as leaf-snapshot-segment. A baseline is written lazily only when a snapshot scan must survive past its first page, and is cleared on cursor close (LatticeCursorGrain.Snapshot.cs) or by the sliding-TTL leak guard. Every baseline is therefore ephemeral by design and every key is used once. 100% dead is the expected census signature for such a population under clear-without-delete, not evidence of a leak: a live row would only appear if a scan happened to be mid-page at the sampling instant. The name is misleading - it is a per-scan baseline, not a durable one. Fixed entirely by this change.

lattice-cursor, 100% dead (80/80) at Version 3. The same mechanism. LatticeCursorGrain is per-cursor, keyed repo-context-memory/<guid>, and calls ClearStateAsync() on close. Version 3 is consistent with an open/advance/close lifecycle. No connection to #3300 was established and none is asserted here. The 100% dead census is fully accounted for by ephemeral keys plus clear-without-delete, so it is not evidence either way about whether that tree checkpoints.

Regression review

The one real interaction is WAL GC blocking-pin classification. A cleared leaf currently reads back as RecordExists == true with a default state and an empty TreeId, classifying as Orphaned; with the row gone it reads as no record and classifies as NoDurableState. The orphan-retirement sweep in LatticeWalGcScheduler handles both in the same case block, so the repair added by #3297 is unaffected and only the metric label moves.

IndexedPersistentState enrolment is gated on RecordExists and becomes strictly more correct, since a dead grain stops projecting a default-state index entry. Clear-then-write within an activation stays correct: the ETag is now null, so the write takes the insert path rather than the update path, which is what the round-trip test pins.

Tests

test/lattice.api.mcp.repocontext/Host/SqliteGrainStorageQueryTests.cs drives the real embedded query text against a real SQLite file:

• A_delete_removes_the_row_entirely asserts COUNT(*) == 0. This is deliberately the row-count assertion rather than "a read returns empty" - the latter is the weaker claim that current clear-without-delete behaviour already satisfies, so it could not distinguish the fix from the bug.
• A_clear_leaves_a_tombstone_row_behind_which_is_the_behaviour_delete_replaces pins the contrast that gives the test above its meaning, so the two queries cannot silently converge.
• Version reporting on success, the stale-version case (row must survive), the duplicate-row case (exactly one reported row, both storage rows deleted), and a write-after-delete round trip.

SqliteDeleteOnClearContractTests.cs is the coupling guard: it executes Orleans' own AdoNetGrainStorage.DefaultInitializationQuery verbatim against an initialised database and checks the resulting key set against the option value the real wiring produces. It reddens if the query is renamed or dropped, if the option is enabled on a branch whose schema lacks the key, or if Orleans changes which keys it demands. It carries a negative control so the assertion cannot go vacuous.

DurabilitySelectorWiringTests.cs pins DeleteStateOnClear as true on the SQLite arm and false on the Postgres arm, with the reason in the assertion message.

Follow-up worth its own issue

Purging the ~18,810 rows already tombstoned. Not done here: it is a bulk delete against a live 4.9 GB database, it is not in this issue's acceptance criteria, and it would still reclaim no disk.

Refs #3307