-
Notifications
You must be signed in to change notification settings - Fork 1
test: toy cartel detector — Amara Otto-122 validation bar CLEARED #323
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| module Zeta.Tests.Simulation.CartelToyTests | ||
|
|
||
| open FsUnit.Xunit | ||
| open global.Xunit | ||
| open Zeta.Core | ||
| open Zeta.Tests.Support.CartelInjector | ||
|
|
||
|
|
||
| /// **Toy cartel detector — Amara Otto-122 validation bar.** | ||
| /// | ||
| /// The test that closes the theory-cathedral-prevention loop: | ||
| /// if the Graph substrate design is RIGHT, a dumb detector | ||
| /// using only `largestEigenvalue` should catch a dumb 5-node | ||
| /// cartel among 50 baseline validators at high detection rate | ||
| /// across many seeds. If it fails, the ADR is wrong. | ||
| /// | ||
| /// Detection rule (MVP; single-signal): | ||
| /// detected = lambda_attacked >= detectionMultiplier * lambda_baseline | ||
| /// | ||
| /// Real-world detection uses composite scores + null-baseline | ||
| /// calibration; this MVP establishes that the PRIMITIVE WORKS. | ||
|
|
||
|
|
||
| /// Parameters matching Amara's 15th/16th ferry prescription: | ||
| /// 50 validators + 5-node cartel + ~3 average degree per node. | ||
| let private nodeCount = 50 | ||
| let private cartelSize = 5 | ||
| let private avgDegree = 3 | ||
| let private cartelWeight = 10L | ||
| let private detectionMultiplier = 2.0 | ||
| let private eigenIter = 500 | ||
| let private eigenTol = 1e-9 | ||
|
|
||
|
|
||
| /// Single trial: generate baseline + attacked variants with a | ||
| /// fresh RNG seeded by `seed`; compute eigenvalues; return true | ||
| /// iff attacked-lambda crossed the `detectionMultiplier * baseline- | ||
| /// lambda` threshold. | ||
| let private runTrial (seed: int) : bool = | ||
| let rng = System.Random(seed) | ||
| let baseline = buildBaseline rng nodeCount avgDegree | ||
| let attacked, _cartelNodes = | ||
| injectCartel rng baseline cartelSize cartelWeight nodeCount | ||
| let baselineLambda = | ||
| Graph.largestEigenvalue eigenTol eigenIter baseline | ||
| |> Option.defaultValue 0.0 | ||
| let attackedLambda = | ||
| Graph.largestEigenvalue eigenTol eigenIter attacked | ||
| |> Option.defaultValue 0.0 | ||
| attackedLambda >= detectionMultiplier * baselineLambda | ||
|
|
||
|
|
||
| [<Fact>] | ||
| let ``toy cartel detector — 100 seeds, detection rate >= 90%`` () = | ||
| // 100-seed MVP. Target 90% detection rate per Amara Otto-122 | ||
| // validation bar (1000-seed scaled-up run is a follow-up | ||
| // bench-project, not a unit-test obligation). | ||
| let trials = 100 | ||
| let hits = | ||
| [| 0 .. trials - 1 |] | ||
| |> Array.map runTrial | ||
| |> Array.filter id | ||
| |> Array.length | ||
| let rate = double hits / double trials | ||
| rate |> should (be greaterThanOrEqualTo) 0.9 | ||
|
|
||
|
|
||
| [<Fact>] | ||
| let ``toy cartel detector — clean baseline rarely triggers (false-positive rate <= 20%)`` () = | ||
| // Sanity check: run detection on baseline vs baseline (no | ||
| // cartel injection). The rule compares `baseline-lambda vs | ||
| // 2*baseline-lambda-of-another-seed`. Because baseline is | ||
| // synthetic-random, two independent baselines can have | ||
| // mildly different lambdas; we allow up to 20% false- | ||
| // positive rate as a generous upper bound. Real null- | ||
| // baseline calibration (per Amara 14th ferry) would set | ||
| // thresholds from percentile-of-baseline-distribution; | ||
| // this MVP verifies the order-of-magnitude is right. | ||
| let trials = 100 | ||
| let falsePositives = | ||
| [| 0 .. trials - 1 |] | ||
| |> Array.map (fun seed -> | ||
| let rng1 = System.Random(seed * 2) | ||
| let rng2 = System.Random(seed * 2 + 1) | ||
| let baseline1 = buildBaseline rng1 nodeCount avgDegree | ||
| let baseline2 = buildBaseline rng2 nodeCount avgDegree | ||
| let lambda1 = | ||
| Graph.largestEigenvalue eigenTol eigenIter baseline1 | ||
| |> Option.defaultValue 0.0 | ||
| let lambda2 = | ||
| Graph.largestEigenvalue eigenTol eigenIter baseline2 | ||
| |> Option.defaultValue 0.0 | ||
| lambda2 >= detectionMultiplier * lambda1) | ||
| |> Array.filter id | ||
| |> Array.length | ||
| let rate = double falsePositives / double trials | ||
| rate |> should (be lessThanOrEqualTo) 0.2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| module Zeta.Tests.Support.CartelInjector | ||
|
|
||
| open Zeta.Core | ||
|
|
||
|
|
||
| /// **CartelInjector** — test-only red-team synthetic cartel | ||
| /// generator. Lives in `tests/Tests.FSharp/_Support/` per Otto-118 | ||
| /// discipline: adversarial tooling is NOT shipped as Zeta public | ||
| /// API. Purpose: validate detectors, not attack production systems. | ||
| /// | ||
| /// Provenance: 13th ferry §3 (Synthetic Cartel Injector) + 14th | ||
| /// ferry §Adversarial Simulation Loop + Amara Otto-122 validation | ||
| /// bar (toy cartel at 50 validators + 5-node cartel). | ||
|
|
||
|
|
||
| /// Build a baseline synthetic validator network with random-ish | ||
| /// sparse edges. The call sweeps source indices `0 .. nodeCount - 1` | ||
| /// and for each source emits up to `avgDegree` outbound edges to | ||
| /// random targets drawn uniformly from the same index range; weight | ||
| /// is `1`. The resulting graph's node set is derived from the edges | ||
| /// actually emitted (self-edges are skipped, and isolated indices | ||
| /// never appear as endpoints), so `Graph.nodes baseline` may be a | ||
| /// **strict subset** of `0 .. nodeCount - 1`. The baseline has no | ||
| /// deliberate community structure — a "null" input the detector | ||
| /// should NOT flag. | ||
| let buildBaseline (rng: System.Random) (nodeCount: int) (avgDegree: int) : Graph<int> = | ||
| let edges = | ||
| [ for s in 0 .. nodeCount - 1 do | ||
| for _ in 1 .. avgDegree do | ||
| let t = rng.Next(nodeCount) | ||
| if t <> s then yield (s, t, 1L) ] | ||
| Graph.fromEdgeSeq edges | ||
|
|
||
|
|
||
| /// Inject a dense cartel clique of `cartelSize` nodes picked | ||
| /// uniformly at random from the **baseline's actual node set** | ||
| /// (`Graph.nodes baseline`), not from a precomputed index range — | ||
| /// otherwise the cartel could land on indices that never appeared | ||
| /// as endpoints in `baseline`. Every ordered pair within the cartel | ||
| /// gets an edge of weight `cartelWeight`. Lower `cartelWeight` = | ||
| /// stealthier cartel; higher = louder. | ||
| /// | ||
| /// Returns the attacked graph + the set of cartel node-IDs so | ||
| /// the test can verify detection correctness. | ||
| let injectCartel | ||
| (rng: System.Random) | ||
| (baseline: Graph<int>) | ||
| (cartelSize: int) | ||
| (cartelWeight: int64) | ||
| (_nodeCount: int) | ||
| : Graph<int> * Set<int> = | ||
| let cartelNodes = | ||
| let shuffled = Graph.nodes baseline |> Set.toArray | ||
| // Fisher-Yates shuffle | ||
| for i in shuffled.Length - 1 .. -1 .. 1 do | ||
| let j = rng.Next(i + 1) | ||
| let tmp = shuffled.[i] | ||
| shuffled.[i] <- shuffled.[j] | ||
| shuffled.[j] <- tmp | ||
| shuffled |> Array.take (min cartelSize shuffled.Length) |> Set.ofArray | ||
| let cartelEdges = | ||
|
AceHack marked this conversation as resolved.
|
||
| [ for s in cartelNodes do | ||
| for t in cartelNodes do | ||
| if s <> t then yield (s, t, cartelWeight) ] | ||
| let attacked = | ||
| let combined = | ||
| List.append | ||
| (baseline.Edges.AsSpan().ToArray() | ||
| |> Array.map (fun entry -> | ||
| let (s, t) = entry.Key | ||
| (s, t, entry.Weight)) | ||
| |> Array.toList) | ||
| cartelEdges | ||
| Graph.fromEdgeSeq combined | ||
| attacked, cartelNodes | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.