feat(wonder): #228 bake-off harness — strategies + corpus + evaluator + runner - #397
Merged
Merged
Conversation
RW (random walk), TC (triangle closure), STS (span-topic sampling) implementing the candidates from docs/v2_wonder_consolidation.md "The three candidate strategies". Each is a pure read over a MemoryStore returning a list of Phantom dataclasses; output sorted by composition for the bake-off determinism story. No write-path activation here — this is the research surface only; ship-decision PR per spec § "Adoption criteria" wires the chosen strategy into production retrieval.
simulator.py: deterministic corpus generator (n_topics × n_atoms_per_topic) + feedback simulator (single-topic agreement = confirm) + populate_store helper. Synthetic from seed; no lab-fixture dependency. evaluator.py: four metrics from spec § Adoption criteria — confirmation_rate, retrieval_freq_per_cost, pairwise_jaccard, junk_rate — plus the verdict function applying the four ship rules (single / ensemble / defer / drop). runner.py: orchestrates a multi-seed sweep (default N=10 per planning memo Decision D), aggregates mean metrics across seeds, emits result JSON. CLI entry point at `python -m aelfrice.wonder.runner`. wonder_consolidation.py: thin shim keeping the existing bench-gate stub honest (Decision A). Token-overlap relatedness in [0,1]; not the bake-off. Smoke run on 2-seed × 20-walk preview produces sane verdict distribution (defer at default densities — corpus tuning is the running session's job for R<final>, not part of this harness PR).
45 tests across four files: - test_wonder_strategies.py (15) — RW/TC/STS unit coverage incl. empty-store, no-edges, determinism-from-seed, edge-type filtering, unordered-pair dedup, session-diversity floor. - test_wonder_simulator.py (14) — corpus determinism, 200-atom default, topic partition invariant, populate_store stamping, feedback verdict shape, promotion gate at α≥12. - test_wonder_evaluator.py (11) — metric math, four verdict rules (single/ensemble/defer/drop) on synthetic metric shapes. - test_wonder_runner.py (5) — runner smoke + determinism + JSON output to file. Bench-gate stub (tests/bench_gate/test_wonder_consolidation.py) continues to skip without AELFRICE_CORPUS_ROOT — the new `aelfrice.wonder_consolidation` shim is what unblocks the ModuleNotFoundError skip path.
Reviewer's GuideImplements the full wonder-consolidation bake-off harness: a synthetic corpus + simulator, three phantom-generation strategies, an evaluator with spec-aligned metrics and adoption rules, a multi-seed runner/CLI producing JSON results, and a bench-gate shim plus tests wiring it all together. Sequence diagram for run_bakeoff orchestration per seedsequenceDiagram
actor Operator
participant Runner as run_bakeoff
participant SeedLoop as _run_one_seed
participant RNG as random.Random
participant Simulator as simulator
participant Store as MemoryStore
participant RW as random_walk
participant TC as triangle_closure
participant STS as span_topic_sampling
participant Evaluator as evaluate_strategy
Operator->>Runner: run_bakeoff(config)
Runner->>SeedLoop: _run_one_seed(seed, n_topics, n_atoms_per_topic, n_walks, n_sts_samples, feedback_budget)
SeedLoop->>RNG: create Random(seed)
SeedLoop->>Simulator: build_corpus(rng, n_topics, n_atoms_per_topic)
Simulator-->>SeedLoop: SyntheticCorpus
SeedLoop->>Store: MemoryStore(":memory:")
SeedLoop->>Simulator: populate_store(Store, SyntheticCorpus, rng)
SeedLoop->>RW: random_walk(Store, rng=Random(seed+1), n_walks)
RW-->>SeedLoop: list Phantom (RW)
SeedLoop->>TC: triangle_closure(Store)
TC-->>SeedLoop: list Phantom (TC)
SeedLoop->>STS: span_topic_sampling(Store, rng=Random(seed+2), n_samples)
STS-->>SeedLoop: list Phantom (STS)
SeedLoop->>Evaluator: evaluate_strategy(RW_phantoms, SyntheticCorpus, feedback_budget)
Evaluator-->>SeedLoop: StrategyMetrics RW
SeedLoop->>Evaluator: evaluate_strategy(TC_phantoms, SyntheticCorpus, feedback_budget)
Evaluator-->>SeedLoop: StrategyMetrics TC
SeedLoop->>Evaluator: evaluate_strategy(STS_phantoms, SyntheticCorpus, feedback_budget)
Evaluator-->>SeedLoop: StrategyMetrics STS
SeedLoop->>Evaluator: pairwise_jaccard(RW, TC / STS)
Evaluator-->>SeedLoop: dict pairwise_jaccard
SeedLoop->>Evaluator: adoption_verdict(metrics_tuple, jaccard)
Evaluator-->>SeedLoop: verdict
SeedLoop-->>Runner: per_seed_result{seed, strategy_metrics, jaccard, verdict}
loop for each seed
Runner->>SeedLoop: _run_one_seed(...)
end
Runner->>Runner: _aggregate(per_seed_results)
Runner-->>Operator: result JSON structure{config, per_seed, aggregate}
Operator-->>Operator: inspect JSON, cite in ship-decision PR
Class diagram for wonder bake_off harness core modelsclassDiagram
class Phantom {
+tuple~str~ composition
+str strategy
+float construction_cost
+str seed_id
}
class CorpusAtom {
+str belief_id
+int topic
+str session_id
}
class SyntheticCorpus {
+tuple~CorpusAtom~ atoms
+tuple~Edge~ edges
+int topic_of(belief_id)
}
class StrategyMetrics {
+str strategy
+int n_phantoms
+float confirmation_rate
+float retrieval_freq_per_cost
+float junk_rate
+float mean_construction_cost
}
class BakeoffResult {
+tuple~StrategyMetrics~ metrics
+dict~tuple~str,str~~ pairwise_jaccard
+str verdict
+float h0_floor
+tuple~str~ notes
}
class MemoryStore {
+list~str~ list_belief_ids()
+Belief get_belief(belief_id)
+list~Edge~ edges_from(belief_id)
+iter_all_edges()
+insert_belief(Belief)
+insert_edge(Edge)
}
class Belief {
+str id
+float alpha
+float beta
+str type
+str session_id
}
class Edge {
+str src
+str dst
+str type
+float weight
}
%% Strategy identifiers
class StrategyIds {
+str STRATEGY_RW
+str STRATEGY_TC
+str STRATEGY_STS
+frozenset~str~ STRATEGIES
}
%% Relationships
SyntheticCorpus "*" --> "*" CorpusAtom : contains
SyntheticCorpus "*" --> "*" Edge : contains
Phantom "*" --> "1" StrategyIds : uses_strategy_id
StrategyMetrics "*" --> "*" Phantom : derived_from
BakeoffResult "*" --> "*" StrategyMetrics : aggregates
SyntheticCorpus --> MemoryStore : populate_store
MemoryStore --> Belief : stores
MemoryStore --> Edge : stores
Phantom ..> Belief : composition_ids_ref
note for Phantom "Represents a speculative composition of beliefs"
note for SyntheticCorpus "Deterministic synthetic graph used by simulator and strategies"
note for StrategyMetrics "Per-strategy performance metrics used by adoption_verdict"
Flow diagram for bake_off data pipelineflowchart LR
subgraph SyntheticCorpusBuilder
Bld["build_corpus\nSyntheticCorpus"]
end
subgraph StoreLayer
Mem["MemoryStore"]
Pop["populate_store"]
end
subgraph Strategies
RW["random_walk\nstrategy RW"]
TC["triangle_closure\nstrategy TC"]
STS["span_topic_sampling\nstrategy STS"]
end
subgraph Evaluator
Eval["evaluate_strategy"]
Jacc["pairwise_jaccard"]
Verdict["adoption_verdict"]
end
subgraph Runner
Run["run_bakeoff"]
SeedLoop["_run_one_seed (multi-seed loop)"]
Agg["_aggregate"]
end
Operator((Operator))
Output["Result JSON\nconfig + per_seed + aggregate"]
Operator --> Run
Run --> SeedLoop
SeedLoop --> Bld
Bld -->|SyntheticCorpus| Pop
Pop --> Mem
Mem --> RW
Mem --> TC
Mem --> STS
RW -->|Phantoms| Eval
TC -->|Phantoms| Eval
STS -->|Phantoms| Eval
Eval -->|StrategyMetrics| SeedLoop
RW -->|Compositions| Jacc
TC -->|Compositions| Jacc
STS -->|Compositions| Jacc
Jacc --> Verdict
Verdict --> SeedLoop
SeedLoop --> Agg
Agg --> Output
Output --> Operator
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
📝 WalkthroughWalkthroughAdds a new "wonder" research subpackage implementing a phantom generation strategy bake-off system. Creates three strategy implementations (random walk, triangle closure, span-topic sampling), a synthetic corpus simulator, an evaluator computing per-strategy metrics, and a campaign runner orchestrating multi-seed bake-off execution. Also adds a consolidation-scoring shim and comprehensive test coverage. ChangesWonder Bake-off System
Sequence DiagramsequenceDiagram
actor User
participant Runner as run_bakeoff()
participant Corpus as Simulator:<br/>build_corpus()
participant Store as MemoryStore
participant Strategies as RW, TC, STS
participant Evaluator as evaluate_strategy()
participant Verdict as adoption_verdict()
participant Output as JSON Output
User->>Runner: run_bakeoff(seeds=10)
loop per seed
Runner->>Corpus: build_corpus(rng, n_topics, n_atoms)
Corpus-->>Store: populate_store(atoms, edges)
par Parallel Strategies
Runner->>Strategies: random_walk(store, rng)
Strategies-->>Runner: RW phantoms
and
Runner->>Strategies: triangle_closure(store)
Strategies-->>Runner: TC phantoms
and
Runner->>Strategies: span_topic_sampling(store, rng)
Strategies-->>Runner: STS phantoms
end
par Evaluate Each Strategy
Runner->>Evaluator: evaluate_strategy(RW_phantoms, corpus, budget=16)
Evaluator-->>Runner: RW_metrics
and
Runner->>Evaluator: evaluate_strategy(TC_phantoms, corpus, budget=16)
Evaluator-->>Runner: TC_metrics
and
Runner->>Evaluator: evaluate_strategy(STS_phantoms, corpus, budget=16)
Evaluator-->>Runner: STS_metrics
end
Runner->>Evaluator: pairwise_jaccard(all strategies)
Evaluator-->>Runner: jaccard_matrix
Runner->>Verdict: adoption_verdict(metrics, jaccard)
Verdict-->>Runner: per_seed_verdict
end
Runner->>Output: aggregate(per_seed results)
Output-->>User: JSON(per_seed, aggregate, verdict_distribution)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| from dataclasses import dataclass, field | ||
| from typing import Literal | ||
|
|
||
| from .models import STRATEGY_RW, STRATEGY_STS, STRATEGY_TC, Phantom |
| """ | ||
| if h0_floor is None: | ||
| h0_floor = H0_NULL_RATE + H0_PLUS_PP_FLOOR | ||
| by_strategy = {m.strategy: m for m in metrics} |
Comment on lines
+16
to
+21
| from aelfrice.wonder.models import ( | ||
| STRATEGY_RW, | ||
| STRATEGY_STS, | ||
| STRATEGY_TC, | ||
| Phantom, | ||
| ) |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
build_corpusthehigh_uncertainty_fractionparameter is currently unused, which is confusing given the docstring; either remove it from the signature or apply it during corpus generation (or clearly delegate that responsibility topopulate_store). evaluate_strategyreturnsstrategy="<empty>"for empty phantom lists, which then flows into aggregation inrun_bakeoff(and tests expect strategy keys likeRW/TC/STS); consider threading the strategy identifier explicitly intoevaluate_strategyor handling empty cases in the caller to preserve consistent strategy labels.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `build_corpus` the `high_uncertainty_fraction` parameter is currently unused, which is confusing given the docstring; either remove it from the signature or apply it during corpus generation (or clearly delegate that responsibility to `populate_store`).
- `evaluate_strategy` returns `strategy="<empty>"` for empty phantom lists, which then flows into aggregation in `run_bakeoff` (and tests expect strategy keys like `RW`/`TC`/`STS`); consider threading the strategy identifier explicitly into `evaluate_strategy` or handling empty cases in the caller to preserve consistent strategy labels.
## Individual Comments
### Comment 1
<location path="src/aelfrice/wonder/evaluator.py" line_range="100-101" />
<code_context>
+ share the same verdict — but the rate definition is preserved
+ in case the simulator gains randomness later).
+ """
+ if not phantoms:
+ return StrategyMetrics(
+ strategy="<empty>",
+ n_phantoms=0,
</code_context>
<issue_to_address>
**issue (bug_risk):** Empty-phantom handling sets strategy to "<empty>", which can break aggregation and adoption logic.
When `phantoms` is empty, `evaluate_strategy` returns `StrategyMetrics(strategy="<empty>")`. Since `_run_one_seed` always calls `evaluate_strategy` for RW, TC, and STS, all three can end up labeled `"<empty>"`.
This breaks downstream logic:
* `_aggregate` uses the strategy names as dict keys, so duplicate `"<empty>"` keys collapse per-strategy data.
* `adoption_verdict` also builds a dict keyed by `m.strategy`, so different strategies with no phantoms overwrite each other.
Instead, keep the real strategy name even when `phantoms` is empty (e.g., pass a `strategy_name` into `evaluate_strategy` or derive it from `_run_one_seed` and use that for `StrategyMetrics.strategy`).
</issue_to_address>
### Comment 2
<location path="src/aelfrice/wonder/simulator.py" line_range="84-92" />
<code_context>
+ return f"t{topic:02d}_a{idx:03d}"
+
+
+def build_corpus(
+ *,
+ rng: random.Random,
+ n_topics: int = 8,
+ n_atoms_per_topic: int = 25,
+ n_sessions: int = 8,
+ intra_edge_density: float = 0.25,
+ cross_edge_density: float = 0.02,
+ high_uncertainty_fraction: float = 0.15,
+) -> SyntheticCorpus:
+ """Generate a deterministic synthetic corpus.
</code_context>
<issue_to_address>
**suggestion:** The `high_uncertainty_fraction` parameter on `build_corpus` is unused.
Since this argument is never used, it makes the API misleading, especially given `populate_store` uses a parameter with the same name. Either remove `high_uncertainty_fraction` from `build_corpus` if corpus generation is intended to be independent of this prior, or thread it through to the relevant logic so the parameter has a well-defined effect.
Suggested implementation:
```python
def build_corpus(
*,
rng: random.Random,
n_topics: int = 8,
n_atoms_per_topic: int = 25,
n_sessions: int = 8,
intra_edge_density: float = 0.25,
cross_edge_density: float = 0.02,
) -> SyntheticCorpus:
```
1. Search the codebase for all call sites of `build_corpus` and remove any `high_uncertainty_fraction=...` keyword argument (or positional argument in that position) to match the updated function signature.
2. If the documentation or docstrings elsewhere mention `high_uncertainty_fraction` in relation to `build_corpus`, update or remove those references to avoid confusion.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+100
to
+101
| if not phantoms: | ||
| return StrategyMetrics( |
There was a problem hiding this comment.
issue (bug_risk): Empty-phantom handling sets strategy to "", which can break aggregation and adoption logic.
When phantoms is empty, evaluate_strategy returns StrategyMetrics(strategy="<empty>"). Since _run_one_seed always calls evaluate_strategy for RW, TC, and STS, all three can end up labeled "<empty>".
This breaks downstream logic:
_aggregateuses the strategy names as dict keys, so duplicate"<empty>"keys collapse per-strategy data.adoption_verdictalso builds a dict keyed bym.strategy, so different strategies with no phantoms overwrite each other.
Instead, keep the real strategy name even when phantoms is empty (e.g., pass a strategy_name into evaluate_strategy or derive it from _run_one_seed and use that for StrategyMetrics.strategy).
Comment on lines
+84
to
+92
| def build_corpus( | ||
| *, | ||
| rng: random.Random, | ||
| n_topics: int = 8, | ||
| n_atoms_per_topic: int = 25, | ||
| n_sessions: int = 8, | ||
| intra_edge_density: float = 0.25, | ||
| cross_edge_density: float = 0.02, | ||
| high_uncertainty_fraction: float = 0.15, |
There was a problem hiding this comment.
suggestion: The high_uncertainty_fraction parameter on build_corpus is unused.
Since this argument is never used, it makes the API misleading, especially given populate_store uses a parameter with the same name. Either remove high_uncertainty_fraction from build_corpus if corpus generation is intended to be independent of this prior, or thread it through to the relevant logic so the parameter has a well-defined effect.
Suggested implementation:
def build_corpus(
*,
rng: random.Random,
n_topics: int = 8,
n_atoms_per_topic: int = 25,
n_sessions: int = 8,
intra_edge_density: float = 0.25,
cross_edge_density: float = 0.02,
) -> SyntheticCorpus:- Search the codebase for all call sites of
build_corpusand remove anyhigh_uncertainty_fraction=...keyword argument (or positional argument in that position) to match the updated function signature. - If the documentation or docstrings elsewhere mention
high_uncertainty_fractionin relation tobuild_corpus, update or remove those references to avoid confusion.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
src/aelfrice/wonder/evaluator.py (2)
115-118: 💤 Low valueRedundant repeated calls for deterministic verdict.
The
feedback_verdictcall is deterministic given(phantom.composition, corpus), so calling itfeedback_budget_per_phantomtimes always returns identical results. The code documents this as future-proofing for potential simulator randomness.For the research harness this is fine, but if performance matters, caching or single-call would suffice.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/aelfrice/wonder/evaluator.py` around lines 115 - 118, The loop building verdicts repeatedly calls the deterministic function feedback_verdict(phantom.composition, corpus) feedback_budget_per_phantom times; replace this with a single call to feedback_verdict and replicate or cache that single result to produce the list (or memoize feedback_verdict for the (phantom.composition, corpus) key) so you avoid redundant work while keeping the same verdicts list semantics used by verdicts.
192-207: 💤 Low valueClarify the adoption_verdict fallthrough logic.
When 2+ strategies clear the floor:
- Top-two complementary (Jaccard < 0.3) →
"ensemble"- Otherwise →
"single"(fallback)The intermediate
cost_windowcheck (lines 202-206) appears to be defensive guarding but always passes whenclearingis non-empty with positive retrieval. Consider simplifying or adding a comment explaining when this guard would actually trigger a different path.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/aelfrice/wonder/evaluator.py` around lines 192 - 207, The fallthrough logic in the adoption decision can be simplified: after computing sorted_clearing, top_two, pair and checking if their Jaccard < JACCARD_COMPLEMENT to return "ensemble", the subsequent any(j >= JACCARD_REDUNDANCY ...) + cost_window check is redundant because the function ultimately returns "single" in all other cases; remove that intermediate block (the any(...) check and the cost_window conditional) and simply return "single" as the fallback, or if you want to preserve it for documentation/defensive reasons, replace it with a clear comment referencing sorted_clearing/top_two/pair and why cost_window would ever alter the outcome (using symbols JACCARD_REDUNDANCY and cost_window) so the intent is explicit.src/aelfrice/wonder/runner.py (1)
136-142: 💤 Low valueTie-breaking behavior for majority verdict is arbitrary.
With N=10 seeds, verdict ties are possible.
Counter.most_common(1)[0][0]returns an arbitrary winner among tied values (depends on insertion order in Python 3.7+). Consider documenting this or adding deterministic tie-breaking if reproducibility matters for audit.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/aelfrice/wonder/runner.py` around lines 136 - 142, The current majority selection uses verdict_counter.most_common(1)[0][0], which can return an arbitrary value when counts tie; make this deterministic by computing the highest count from verdict_counter (variable verdict_counter built from per_seed) and then selecting the tie by a deterministic rule (e.g., lexicographic order, fixed priority list, or sort by (-count, verdict) to break ties consistently) before assigning majority_verdict, and update the returned "majority_verdict" accordingly so repeated runs yield reproducible results.src/aelfrice/wonder/simulator.py (1)
137-140: 💤 Low valueFragile topic detection via string slicing.
The cross-edge logic extracts topic identity using
src[:3]which assumes the atom ID format"tNN_..."from_atom_id. If the ID format changes, this check silently fails to detect same-topic pairs correctly.Consider reusing the topic lookup from
by_topicor storing topic info alongside IDs for robustness.♻️ Suggested fix using existing topic mapping
- all_ids = [a.belief_id for a in atoms] - for src in all_ids: - for dst in all_ids: - if src == dst: - continue - src_topic = src[:3] - dst_topic = dst[:3] - if src_topic == dst_topic: - continue + id_to_topic = {a.belief_id: a.topic for a in atoms} + all_ids = list(id_to_topic.keys()) + for src in all_ids: + for dst in all_ids: + if src == dst: + continue + if id_to_topic[src] == id_to_topic[dst]: + continue🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/aelfrice/wonder/simulator.py` around lines 137 - 140, The current same-topic check uses brittle string slicing (src_topic = src[:3]) which breaks if atom ID format changes; replace this with a robust lookup using the existing by_topic mapping or an explicit id->topic dict: determine each atom's topic by searching by_topic (or a new mapping built once from by_topic) and compare those topic values instead of src[:3]/dst[:3]; update the code paths that reference src_topic/dst_topic (the block containing src, dst, src_topic, dst_topic) to use the topic lookup function or mapping so same-topic pairs are detected reliably.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@src/aelfrice/wonder/evaluator.py`:
- Around line 115-118: The loop building verdicts repeatedly calls the
deterministic function feedback_verdict(phantom.composition, corpus)
feedback_budget_per_phantom times; replace this with a single call to
feedback_verdict and replicate or cache that single result to produce the list
(or memoize feedback_verdict for the (phantom.composition, corpus) key) so you
avoid redundant work while keeping the same verdicts list semantics used by
verdicts.
- Around line 192-207: The fallthrough logic in the adoption decision can be
simplified: after computing sorted_clearing, top_two, pair and checking if their
Jaccard < JACCARD_COMPLEMENT to return "ensemble", the subsequent any(j >=
JACCARD_REDUNDANCY ...) + cost_window check is redundant because the function
ultimately returns "single" in all other cases; remove that intermediate block
(the any(...) check and the cost_window conditional) and simply return "single"
as the fallback, or if you want to preserve it for documentation/defensive
reasons, replace it with a clear comment referencing
sorted_clearing/top_two/pair and why cost_window would ever alter the outcome
(using symbols JACCARD_REDUNDANCY and cost_window) so the intent is explicit.
In `@src/aelfrice/wonder/runner.py`:
- Around line 136-142: The current majority selection uses
verdict_counter.most_common(1)[0][0], which can return an arbitrary value when
counts tie; make this deterministic by computing the highest count from
verdict_counter (variable verdict_counter built from per_seed) and then
selecting the tie by a deterministic rule (e.g., lexicographic order, fixed
priority list, or sort by (-count, verdict) to break ties consistently) before
assigning majority_verdict, and update the returned "majority_verdict"
accordingly so repeated runs yield reproducible results.
In `@src/aelfrice/wonder/simulator.py`:
- Around line 137-140: The current same-topic check uses brittle string slicing
(src_topic = src[:3]) which breaks if atom ID format changes; replace this with
a robust lookup using the existing by_topic mapping or an explicit id->topic
dict: determine each atom's topic by searching by_topic (or a new mapping built
once from by_topic) and compare those topic values instead of src[:3]/dst[:3];
update the code paths that reference src_topic/dst_topic (the block containing
src, dst, src_topic, dst_topic) to use the topic lookup function or mapping so
same-topic pairs are detected reliably.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ddc0e7da-6f23-4037-ac6f-e8dbc4244a4e
📒 Files selected for processing (11)
src/aelfrice/wonder/__init__.pysrc/aelfrice/wonder/evaluator.pysrc/aelfrice/wonder/models.pysrc/aelfrice/wonder/runner.pysrc/aelfrice/wonder/simulator.pysrc/aelfrice/wonder/strategies.pysrc/aelfrice/wonder_consolidation.pytests/test_wonder_evaluator.pytests/test_wonder_runner.pytests/test_wonder_simulator.pytests/test_wonder_strategies.py
Collaborator
Author
|
[claim:review:Kulili:2026-05-04T07:58:50Z] |
Collaborator
Author
|
[claim:review:Toug:2026-05-04T07:59:25Z] |
Collaborator
Author
|
[release:review:Toug:2026-05-04T07:59:30Z] |
Collaborator
Author
|
Reviewed and FF-merged to main as 61ab575. Diff (1582+ across 11 files): self-contained Spec adherence checked against Minor smells (non-blocking, follow-up if useful when R output gets cited):
|
Collaborator
Author
|
[release:review:Kulili:2026-05-04T08:00:10Z] |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the wonder-consolidation bake-off harness from
docs/v2_wonder_consolidation.md(#228). Three generation strategies (RW / TC / STS), a deterministic synthetic 200-atom corpus, a feedback simulator, an evaluator producing the four spec-defined metrics, and a multi-seed runner that emits a result JSON the v2.0 ship-decision can cite.The actual ship-decision (single-strategy / ensemble / defer / drop) is the output of running
python -m aelfrice.wonder.runnerto convergence — not part of this PR. This PR ships only the harness so that R can be run separately and its output cited in the ship-decision PR per the spec § "Adoption criteria for v2.0 ship".What's in tree
src/aelfrice/wonder/__init__.py— package surface.src/aelfrice/wonder/models.py—Phantomdataclass (kept out ofaelfrice.modelsper planning memo Decision B;Phantomis research-only).src/aelfrice/wonder/strategies.py—random_walk,triangle_closure,span_topic_sampling. Pure reads over aMemoryStore; deterministic given seeded RNGs.src/aelfrice/wonder/simulator.py—build_corpus+populate_store+feedback_verdict+simulate_promotion. Single-topic agreement is the corpus's ground-truth predicate for "real relationship".src/aelfrice/wonder/evaluator.py—evaluate_strategy,pairwise_jaccard,adoption_verdict(the four-rule decision tree from spec § "Adoption criteria").src/aelfrice/wonder/runner.py— orchestrator + CLI entry point. Default sweep is N=10 seeds per planning memo Decision D.src/aelfrice/wonder_consolidation.py— thin shim keeping the bench-gate stub honest (Decision A). Returns a token-overlap relatedness in [0,1]; not the bake-off itself.Decisions locked in
Calling out the planning-memo decision points so reviewers can contest before merge:
wonder_consolidation.score()shimPhantomlivesaelfrice.wonder.models(notaelfrice.models)construction_costunitsTests
45 tests across four files (
test_wonder_strategies.py,test_wonder_simulator.py,test_wonder_evaluator.py,test_wonder_runner.py). Cover: each strategy's empty/no-edge/normal/determinism shape; corpus generator determinism + size invariants; feedback verdict edge cases; α≥12 promotion gate; all fouradoption_verdictpaths; runner JSON-to-file round-trip.The existing
tests/bench_gate/test_wonder_consolidation.pycontinues to skip withoutAELFRICE_CORPUS_ROOT— the shim unblocks theModuleNotFoundErrorskip path so once awonder_consolidation/corpus lands lab-side, the gate runs.What this is not
Spec checks
docs/v2_wonder_consolidation.md§ "The three candidate strategies" — RW / TC / STS implemented as described.evaluator.py.aelf reasontouch. No lab-fixture imports.Closes #228 once R is run and the chosen strategy lands in production retrieval (separate PR).
Summary by Sourcery
Introduce a self-contained wonder-consolidation bake-off harness with three phantom generation strategies, a synthetic corpus and feedback simulator, an evaluator implementing spec-defined metrics and adoption rules, and a multi-seed runner producing JSON results for the v2.0 ship decision.
New Features:
Tests:
Summary by CodeRabbit
Release Notes
New Features
Tests