memory: Otto-285 — DST and determinism are not edge-case avoidance#483
Merged
memory: Otto-285 — DST and determinism are not edge-case avoidance#483
Conversation
Aaron 2026-04-25 standing meta-rule above Otto-281: > "we never want to use random seed pins to cheat by not > fully testing if you understand what I mean." > "the general rule is dont use DST and determinism to > avoid edge cases handling." > "tests are all deterministic but the real world isn't, > our tests are trying to test all the edge cases of the > real world but in a deterministic way not reduce scope > by eliminating edge cases of the real world in our tests > with determinism. that will lead to more robust tests." The principle: determinism is the WAY to test chaos reproducibly (so bugs replay), NOT the REASON to skip chaos. The real world is non-deterministic — tests should deterministically exercise every flavor of chaos the algorithm will encounter in production. The discriminator question for any "make it deterministic" fix: does it INVOKE the algorithm's actual contract (legitimate — algorithm has documented input invariants you're satisfying) or SHRINK the test's coverage of what the algorithm is supposed to handle (cheat)? Same input space via deterministic primitive = fine (invoke contract). Narrower input space because broader revealed problems = cheat. Empirical verification this session: PR #482 HLL fuzz fix is LEGITIMATE per discriminator. HLL theorem requires uniform 64-bit hashes; HashCode.Combine violated contract; XxHash3 satisfies contract; same n values FsCheck generates are still tested. 500-trial × 5-offset sweep shows max error 1.96% vs 4% bound — contract-satisfied means bound holds. Composes with Otto-281 (fix the determinism — Otto-285 is the meta-rule above), Otto-272 (DST as substrate), Otto-264 (rule of balance — paired coverage verification), Otto-282 (comment WHY discriminator), Otto-248 (never ignore flakes — flakes ARE the contract-violation signal). CLAUDE.md candidate; deferred to maintainer discretion per Otto-283. Memory-reference lint verified: 497/497 refs resolve.
There was a problem hiding this comment.
Pull request overview
Adds a new in-repo memory entry (Otto-285) clarifying the intended discipline around determinism/DST: determinism should be used to reproduce real-world chaos reliably in tests, not to “cheat” by shrinking away legitimate edge cases.
Changes:
- Add new feedback memory file documenting Otto-285’s discriminator (“invoke algorithm contract” vs “shrink test coverage”).
- Update
memory/MEMORY.mdto index/link the new Otto-285 entry near the top.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
memory/feedback_dst_not_edge_case_avoidance_otto_285_2026_04_25.md |
New Otto-285 memory: determinism-as-reproducibility, plus the “invoke contract vs shrink coverage” discriminator and examples. |
memory/MEMORY.md |
Adds an index entry linking to the new Otto-285 memory. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Standing meta-rule above Otto-281 captured from Aaron's framing 2026-04-25 across PR #482 (HLL fuzz test fix). The principle: determinism is the WAY to test chaos reproducibly, not the REASON to skip chaos. The real world is non-deterministic; tests should deterministically exercise every flavor of chaos the algorithm encounters in production.
The discriminator for any "make it deterministic" fix:
Same input space via deterministic primitive = fine. Narrower input space because broader revealed problems = cheat.
Empirical evidence
PR #482 HLL fuzz fix verified LEGITIMATE per discriminator: HLL's correctness theorem requires uniform 64-bit hashes;
HashCode.Combineviolated that contract;XxHash3satisfies it; samenvalues FsCheck generates are still tested. 500-trial sweep × 5 offset baselines: max error 1.96%, 0 exceeded 4% bound. With contract satisfied, bound holds — confirming the 4% tolerance was correctly chosen.Test plan
🤖 Generated with Claude Code