fix(audit): comprehensive HashCode.Combine sweep + SplitMix64 refactor (Otto-281 follow-up)

docs/BACKLOG.md

@@ -12758,6 +12758,60 @@ Aarav.
   msbuild-expert. **Source:** Copilot thread
   `PRRT_kwDOSF9kNM59geKB` on PR #147.
 
+## P2 — DST-compatible HashDoS-resistant Shard.OfKey research (Otto-281 follow-up)
+
+- [ ] **Investigate whether `Shard.OfKey` can be redesigned to be
+  DST-compatible AND retain HashDoS defence simultaneously.**
+  Current state (post Otto-281 audit): `OfKey` uses
+  `HashCode.Combine(key, Shard.Salt)` which is process-randomized
+  for security (anti-hash-flooding) and therefore NOT
+  cross-process deterministic; tests use `OfFixed` /
+  `OfFixedBytes` (deterministic XxHash3) for DST-compatibility
+  and accept the separation. Aaron 2026-04-25 directive:
+  *"backlog if we can keep all performance and tradeoffs we
+  wanting and design it in a different way that is DST
+  compatable and safe. We may already be there, 1 is a good
+  choice, but we should just backlog that research."*
+
+  **Research questions:**
+
+  - Can a **per-test-suite-pinned salt** combined with a
+    deterministic mixer (XxHash3 instead of `HashCode.Combine`)
+    deliver both HashDoS defence in production and DST
+    determinism in tests, via a single API?
+  - Does the production HashDoS posture actually require
+    *runtime-randomized* salt, or is a **per-deployment static
+    salt** (set at process start from secret config) sufficient
+    against the attacker model? If the latter, tests can run
+    with a fixed salt without weakening production.
+  - Is `HashCode.Combine` strictly required for HashDoS defence,
+    or is the *salt-mixed-into-deterministic-XxHash* sufficient?
+    The attacker doesn't know the salt; XxHash on
+    `(salt-bytes ⊕ key-bytes)` is just as hard to attack as
+    HashCode.Combine without leaking the salt.
+  - What's the perf cost of `XxHash3.HashToUInt64` vs
+    `HashCode.Combine` on the hot path? Earlier perf concern
+    on `Sketch.fs::Add` was Gen-0 allocations per Add (`new
+    byte[]`); a stack-allocated `Span<byte>` for 4-byte
+    primitive keys avoids that entirely.
+
+  **If the research lands "yes":** unify `OfKey` and `OfFixed`
+  into a single `OfKey(key, shards, ?salt)` that is both
+  cross-process-deterministic-given-salt AND HashDoS-resistant.
+  Tests pass `salt: 0u`; production reads salt from process
+  config.
+
+  **If the research lands "no":** keep the current two-API
+  design (option 1 from the Otto-281 PR triage). Document the
+  trade-off as final; this BACKLOG row resolves to a decision
+  ADR.
+
+  **Effort:** S (research-only; design doc + benchmark
+  comparison; no shipped code unless conclusion is "yes").
+  **Owner:** performance-engineer + security-researcher
+  (cross-cut). **Composes with:** Otto-281 audit, Otto-272
+  DST-everywhere, GOVERNANCE §threat-model section on HashDoS.
+
 ---
 
 ## Source of this backlog

memory/feedback_dst_exempt_is_deferred_bug_not_containment_otto_281_2026_04_25.md

@@ -0,0 +1,148 @@
+---
+name: DST-EXEMPT IS DEFERRED-BUG — never a safe state, no comment that says "DST-exempt" should ship without a deadline; the SharderInfoTheoreticTests case: a previous tick marked the test "DST-exempt: uses HashCode.Combine which is process-randomized" + filed a BACKLOG row, treated as containment; in reality the exemption masked the determinism violation which compounded into 3 unrelated flakes this session (#454 / #458 / #473) before Aaron 2026-04-25 surfaced the rule "see how that one DST exception caused the flake, this is why DST is so important, when we violate, we introduce random failures"; ship the FIX (Otto-281: detHash via XxHash3), not the EXEMPTION; counterweight to Otto-272 DST-everywhere
+description: Otto-281 counterweight memory. DST exemptions are not containment; they are deferred bugs. The SharderInfoTheoreticTests "DST-exempt" comment masked a flake that fired 3 times on unrelated PRs before getting fixed. Pattern: when tempted to write "DST-exempt", instead either fix the determinism or delete the test. Never ship a long-lived DST-exempt tag.
+type: feedback
+---
+## The rule
+
+**"DST-exempt" is not a safe state. It is a deferred bug.**
+
+When a test or code path is marked "DST-exempt" with a comment
+that says "uses X which is process-randomized" or "depends on
+environment Y", that is NOT containment. It is *masking* the
+determinism violation behind a label that sounds like an escape
+hatch.
+
+Aaron's verbatim framing 2026-04-25:
+
+> *"see how that one DST exception caused the fake [flake],
+> this is why DST is so important, when we violate, we
+> introduce random failures."*
+
+## The case that triggered the rule
+
+`tests/Tests.FSharp/Formal/Sharder.InfoTheoretic.Tests.fs`
+contained three uses of `HashCode.Combine k` for hashing
+integer keys:
+
+```fsharp
+let h = uint64 (HashCode.Combine k)
+let s = JumpConsistentHash.Pick(h, shards)
+```
+
+`System.HashCode.Combine` is **process-randomized by .NET
+design** to deter hash-flooding attacks on dictionaries. So
+the same int produces different hashes in different processes.
+Jump-consistent-hash output depends on the input hash, so
+shard assignments varied across CI runs. The
+`< 1.2 max/avg ratio` assertion sometimes held, sometimes
+didn't.
+
+A previous tick (Aaron 2026-04-24 directive territory) added
+this comment above one of the tests:
+
+```fsharp
+// DST-exempt: uses `HashCode.Combine` which is process-randomized
+// per-run. Flake analysis + fix pipeline tracked in docs/BACKLOG.md
+// "SharderInfoTheoreticTests 'Uniform traffic' flake" row. DST
+// marker convention + lint rule: docs/BACKLOG.md "DST-marker
+// convention + lint rule" row (Aaron 2026-04-24 directive).
+```
+
+**That comment treated the issue as contained.** It identified
+the cause (`HashCode.Combine` process-randomization), filed a
+BACKLOG row, and added a "DST-marker convention" idea — all of
+which are good housekeeping. But it left the determinism
+violation in the test code.
+
+## What the masking cost
+
+The "Uniform traffic: consistent-hash is already near-optimal"
+test flaked **three times in this session** on unrelated PRs:
+
+- **#454** (FSharp.Core 10.1.202 → 10.1.203) — pure dep bump,
+  failed on a probabilistic sharder test.
+- **#458** (System.Numerics.Tensors 10.0.6 → 10.0.7) — same.
+- **#473** (Dependabot grouping config) — yaml-only change,
+  same flake.
+
+Three unrelated PRs each had to be diagnosed, the flake ruled
+"unrelated to this change", and the job rerun. Three rerun
+cycles of compute. Three opportunities for an autonomous-loop
+agent to misidentify a real bug as "the same flake — rerun"
+and ship a regression. Three eyeball-time costs for the
+maintainer.
+
+**That's the compounding cost of a DST exemption left to live.**
+The exemption didn't contain the cost — it spread the cost
+across N PRs and N reruns instead of concentrating it on one
+fix.
+
+## The correct response patterns
+
+**When you encounter a flake that uses non-deterministic
+primitives:**
+
+1. **Fix the determinism.** Replace the non-deterministic
+   primitive with a deterministic one of the same kind.
+   For `HashCode.Combine` → `XxHash3.HashToUInt64` (Otto-281).
+   For `Random` (no seed) → `Random seed`. For `DateTime.UtcNow`
+   inside a property check → fixed `DateTimeOffset` constant.
+
+2. **If the determinism cannot be fixed, delete the test.** A
+   test that is *probabilistic in CI* is not a test — it's a
+   coin flip that gets logged. Either commit to fixing the
+   determinism or delete the test entirely. Don't ship a
+   dual-state "sometimes-pass-sometimes-fail" thing under a
+   "DST-exempt" label.
+
+3. **If the determinism cannot be fixed AND the test cannot be
+   deleted (e.g., it tests an inherently-stochastic property
+   like a Monte Carlo bound), wrap the entire test body in a
+   loop with a fixed seed and assert the *aggregate* property
+   over N runs.** That converts the stochastic property into a
+   deterministic-meta-property over fixed seeds.
+
+**Never** leave the test in CI with a "DST-exempt" comment. The
+comment doesn't make the determinism violation safe — it just
+defers the cost.
+
+## The fix shape (Otto-281)
+
+```fsharp
+open System.IO.Hashing
+
+let private detHash (k: int) : uint64 =
+    XxHash3.HashToUInt64 (ReadOnlySpan (BitConverter.GetBytes k))
+
+// All three call sites changed:
+//   let h = uint64 (HashCode.Combine k)
+// becomes:
+//   let h = detHash k
+```
+
+Three iterations in separate processes — all pass with
+identical output. Determinism restored. PR #478.
+
+## Composes with
+
+- **Otto-272** *DST-ify the stabilization process* — counterweight
+  discipline must be deterministic. Same energy: don't carve
+  out exceptions; fix the root.
+- **Otto-248** *never ignore flakes* — flakes ARE the determinism
+  violation, not "transient infra noise". Same shape applied
+  to test-side code.
+- **Otto-264** *rule of balance* — every found mistake triggers
+  a counterweight. This memory IS the counterweight to the
+  "DST-exempt" mistake.
+- **GOVERNANCE.md §section on DST** — DST-everywhere as the
+  default mode, not the special mode.
+
+## Pre-commit-lint candidate
+
+A grep for `DST-exempt` / `DST exempt` / `dst-exempt` in
+comments inside `tests/**` should fire as a warning at
+pre-commit time. Each occurrence is a deferred bug that
+needs a deadline. The lint comment can include the DST
+discipline reminder: "DST-exempt is not containment — fix or
+delete; don't ship dual-state tests."

src/Core/ConsistentHash.fs

@@ -73,10 +73,8 @@ type RendezvousHash(bucketSeeds: uint64 array) =
     [<MethodImpl(MethodImplOptions.AggressiveInlining)>]
     static member private Mix(a: uint64, b: uint64) : uint64 =
         // SplitMix64 of (a xor b) — good enough for HRW scoring.
-        let mutable z = (a ^^^ b) * 0x9E3779B97F4A7C15UL
-        z <- (z ^^^ (z >>> 30)) * 0xBF58476D1CE4E5B9UL
-        z <- (z ^^^ (z >>> 27)) * 0x94D049BB133111EBUL
-        z ^^^ (z >>> 31)
+        // See `src/Core/SplitMix64.fs` for the constant rationale.
+        SplitMix64.mix (a ^^^ b)
 
     /// Pick a bucket by maximum-score-wins. O(bucketCount).
     member _.Pick(key: uint64) : int =

src/Core/Core.fsproj

@@ -27,6 +27,7 @@
     <Compile Include="FeatureFlags.fs" />
     <Compile Include="Simd.fs" />
     <Compile Include="HardwareCrc.fs" />
+    <Compile Include="SplitMix64.fs" />
     <Compile Include="Sketch.fs" />
     <Compile Include="CountMin.fs" />
     <Compile Include="BloomFilter.fs" />

src/Core/FastCdc.fs

@@ -53,11 +53,9 @@ open System.Runtime.CompilerServices
 module private Gear =
     // Deterministic table — generated once via SplitMix64 seeded
     // with the paper's seed so cross-process chunking is repeatable.
+    // See `src/Core/SplitMix64.fs` for the constant rationale.
     let private seedAt (i: int) : uint64 =
-        let mutable z = uint64 i * 0x9E3779B97F4A7C15UL
-        z <- (z ^^^ (z >>> 30)) * 0xBF58476D1CE4E5B9UL
-        z <- (z ^^^ (z >>> 27)) * 0x94D049BB133111EBUL
-        z ^^^ (z >>> 31)
+        SplitMix64.mix (uint64 i)
 
     let Table : uint64 array = Array.init 256 seedAt
 

src/Core/NovelMathExt.fs

@@ -119,6 +119,21 @@ type InfoTheoreticSharder(shardCount: int, epsilon: double, delta: double, seed:
     /// index computed from the key's hash — so on a cold start
     /// (all shards at zero load) load is distributed by the hash
     /// rather than always falling on shard 0.
+    ///
+    /// **Process-randomization caveat (Otto-281 audit):** the
+    /// `hashTieBreak` uses `HashCode.Combine` which re-seeds
+    /// per-process. On a *cold start*, two processes will pick
+    /// different tie-break shards for the same key — the
+    /// load-distribution is process-dependent at the cold-start
+    /// boundary. Once observed loads diverge (after a few
+    /// `Observe` + `Pick` cycles), the load-based picker
+    /// dominates and the assignment becomes deterministic given
+    /// the CMS seed. Tests asserting cross-process determinism
+    /// of cold-start `Pick`s would flake; tests asserting
+    /// post-warmup load-based picks are robust. The trade-off
+    /// is intentional: cold-start tie-breaking by hash is a
+    /// load-distribution flexibility feature, not a correctness
+    /// invariant.
     member _.Pick(key: 'K) : int =
         let predicted = cms.Estimate key
         let hash32 = uint32 (HashCode.Combine key)

src/Core/Shard.fs

@@ -3,6 +3,7 @@ namespace Zeta.Core
 
 open System
 open System.Collections.Generic
+open System.IO.Hashing
 open System.Runtime.CompilerServices
 open System.Runtime.InteropServices
 open System.Threading
@@ -51,6 +52,16 @@ type Shard =
     /// Shard a key using `HashCode.Combine` mixed with the per-process
     /// salt. Good default for any ingest path that touches user-controlled
     /// keys — rules out the HashDoS attack.
+    ///
+    /// **Intentionally non-deterministic across processes.** `HashCode.Combine`
+    /// is process-randomized by .NET design; combining with the per-process
+    /// `Shard.Salt` doubles down on that. Two processes will assign the same
+    /// key to different shards. That is the *point* — it's HashDoS defence:
+    /// an attacker cannot pre-compute a worst-case input set for our shard
+    /// distribution because they don't know our seed.
+    ///
+    /// If you need cross-process / cross-restart determinism, use
+    /// `OfFixed`, which uses `XxHash3` instead.
     [<MethodImpl(MethodImplOptions.AggressiveInlining)>]
     static member OfKey(key: 'K, shards: int) : int =
         let h = uint32 (HashCode.Combine(key, Shard.Salt))
@@ -59,9 +70,51 @@ type Shard =
     /// Shard without the per-process salt — stable across restarts.
     /// Prefer `OfKey` unless you *specifically* need cross-process
     /// determinism (e.g. for Kafka key-to-partition consistency).
+    ///
+    /// **Determinism contract** (Otto-281 honesty audit):
+    ///
+    /// - For **value-type keys** (`int`, `int64`, `uint32`, `byte`, etc.):
+    ///   fully deterministic across processes. `int.GetHashCode()` returns
+    ///   `this` and the rest is deterministic mixing.
+    /// - For **string keys**: NOT deterministic across processes.
+    ///   `string.GetHashCode()` is per-process-randomized in .NET Core+
+    ///   (anti-hash-flooding), and we cannot recover from that within a
+    ///   generic `'K` API. String-key callers who need cross-process
+    ///   consistency MUST hash their UTF-8 bytes themselves and call
+    ///   `Shard.Of(uint32 hash, shards)` directly — for example with
+    ///   `XxHash3.HashToUInt64(Encoding.UTF8.GetBytes(s))`.
+    /// - For **other reference types**: deterministic only if the type's
+    ///   `GetHashCode()` is deterministic (most user-defined records are;
+    ///   classes that depend on instance identity are not).
+    ///
+    /// Otto-281 fix replaced an earlier implementation that used
+    /// `HashCode.Combine key`, which is *also* process-randomized for
+    /// value types (because `HashCode.Combine`'s mixer is randomized
+    /// regardless of input). The new implementation is strictly better
+    /// for value types and no-worse for strings.
+    ///
+    /// String-key cross-process consistency is tracked as a separate
+    /// follow-up: typed overloads `OfFixedString(s: string, shards)` and
+    /// `OfFixedBytes(bytes: ReadOnlySpan&lt;byte&gt;, shards)`.
     [<MethodImpl(MethodImplOptions.AggressiveInlining)>]
     static member OfFixed(key: 'K, shards: int) : int =
-        Shard.Of(uint32 (HashCode.Combine key), shards)
+        let intHash = key.GetHashCode()
+        let bytes = BitConverter.GetBytes intHash
+        let h64 = XxHash3.HashToUInt64 (ReadOnlySpan bytes)
+        Shard.Of(uint32 h64, shards)
+
+    /// Shard a UTF-8 byte sequence without per-process salt — fully
+    /// deterministic across processes and machines. Use this for any
+    /// cross-process / cross-restart shard assignment where the key
+    /// has a canonical byte representation.
+    ///
+    /// String callers: `Shard.OfFixedBytes(Encoding.UTF8.GetBytes s, shards)`
+    /// is the cross-process-consistent shard for `s`. The plain
+    /// `Shard.OfFixed("...", shards)` is NOT (see `OfFixed` doc).
+    [<MethodImpl(MethodImplOptions.AggressiveInlining)>]
+    static member OfFixedBytes(bytes: ReadOnlySpan<byte>, shards: int) : int =
+        let h64 = XxHash3.HashToUInt64 bytes
+        Shard.Of(uint32 h64, shards)
 
 
 /// Exchange operator — partitions a Z-set across `shards` sub-streams by

src/Core/Sketch.fs

@@ -54,14 +54,26 @@ type HyperLogLog(logBuckets: int) =
     /// ≥ ~65 k the 32-bit floor begins to dominate; use `AddHash`
     /// directly with a proper `XxHash3`/`XxHash64` on bytes if you need
     /// billions-scale accuracy.
+    ///
+    /// **Process-randomization caveat (Otto-281 audit):**
+    /// `HashCode.Combine` re-seeds per-process by .NET design (anti-
+    /// hash-flooding). Two processes will produce different cardinality
+    /// estimates for the same input stream — the *bound* is correct
+    /// (within ~2% relative error per HLL's `alpha`), but the exact
+    /// estimate jitters. For tests requiring deterministic estimates
+    /// across runs, use `AddBytes` with a canonical byte representation;
+    /// see `tests/Tests.FSharp/Sketches/HyperLogLog.Tests.fs` for the
+    /// XxHash3 path. The jittery estimate is a *deliberate* trade-off
+    /// for hot-path performance: an earlier revision called
+    /// `XxHash3.HashToUInt64` on a 4-byte heap array per Add; for a 1 M
+    /// element stream that's 1 M Gen-0 allocations purely for HLL.
     [<MethodImpl(MethodImplOptions.AggressiveInlining)>]
     member this.Add(value: 'T) =
         let h32 = HashCode.Combine value |> uint64
-        // SplitMix64 finaliser — public-domain constants, 5 ops, no alloc.
-        let mutable z = h32 * 0x9E3779B97F4A7C15UL
-        z <- (z ^^^ (z >>> 30)) * 0xBF58476D1CE4E5B9UL
-        z <- (z ^^^ (z >>> 27)) * 0x94D049BB133111EBUL
-        this.AddHash (z ^^^ (z >>> 31))
+        // SplitMix64 finaliser — see `src/Core/SplitMix64.fs` for the
+        // constant rationale (golden-ratio + Vigna's BigCrush-validated
+        // multipliers). 5 ops, no alloc, hot-path safe.
+        this.AddHash (SplitMix64.mix h32)
 
     /// Add a byte span directly — lets callers with a canonical byte
     /// representation (serialised key, UTF-8 string) bypass the 32-bit