docs: R_final — wonder-consolidation bake-off ship decision (#228) - #398
Merged
Conversation
Verdict: defer. STS junk_rate (88.9%) trips spec rule 2 decisively; RW (62.6%) and TC (70.6%) also exceed the 60% defer threshold. RW and TC do clear the H0+10pp adoption floor on confirmation rate, so the offline-generation premise is not falsified — ship v2.0 wonder surface as on-line prompted only and leave the runner/strategies/evaluator in tree for a later corpus revisit. Three result JSONs committed alongside the memo: - R0_default.json (n_atoms=25, budget=16): defer x10 seeds. - R_a50_b8.json (n_atoms=50, budget=8): drop x10 (no phantom reaches alpha>=12 inside the budget). - R_a50_b32.json (n_atoms=50, budget=32): defer x10. Closes #228.
|
Important Review skippedIgnore keyword(s) in the title. ⛔ Ignored keywords (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Reviewer's GuideAdds the R_final wonder-consolidation ship decision memo and its associated bake-off result JSONs, documenting how the bake-off harness was run, the quantitative outcomes across configurations, and the resulting decision to defer offline wonder generation for v2.0 while keeping the runner and strategies in-tree for future reruns. Flow diagram for wonder offline-generation adoption decisionflowchart TD
Start[Start evaluation for offline_generation] --> Rule1{Drop?
all_strategies_confirm_rate <= H0_null_floor}
Rule1 -- Yes --> VerdictDrop[Verdict: drop offline_generation line item]
Rule1 -- No --> Rule2{Defer?
any_strategy_junk_rate > 0.60}
Rule2 -- Yes --> VerdictDefer[Verdict: defer offline_generation from v2_0]
Rule2 -- No --> Rule3{Single_strategy_ship?
any_strategy_meets_adoption_floor}
Rule3 -- Yes --> VerdictSingle[Verdict: ship best_single_strategy offline_generation]
Rule3 -- No --> Rule4{Ensemble?
complementarity_passes
pairwise_Jaccard_low}
Rule4 -- Yes --> VerdictEnsemble[Verdict: ship ensemble offline_generation]
Rule4 -- No --> VerdictDrop2[Verdict: drop offline_generation line item]
VerdictDrop --> End[End]
VerdictDefer --> End
VerdictSingle --> End
VerdictEnsemble --> End
VerdictDrop2 --> End
subgraph Inputs
H0_null_floor[H0_null_floor = 0.065]
Adoption_floor[H0_plus_10pp_adoption_floor = 0.165]
Junk_threshold[Junk_rate_threshold = 0.60]
end
Inputs --> Rule1
Inputs --> Rule2
Inputs --> Rule3
Inputs --> Rule4
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- Consider briefly expanding the first mention of RW/TC/STS with a one-line description or link so readers unfamiliar with the strategies can interpret the table without cross-referencing code.
- The references to planning memo decisions (e.g., “Decision D/E/F”) could be ambiguous to future readers; adding a short parenthetical or link to the relevant section of the planning memo would make the provenance clearer.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consider briefly expanding the first mention of RW/TC/STS with a one-line description or link so readers unfamiliar with the strategies can interpret the table without cross-referencing code.
- The references to planning memo decisions (e.g., “Decision D/E/F”) could be ambiguous to future readers; adding a short parenthetical or link to the relevant section of the planning memo would make the provenance clearer.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Collaborator
Author
|
[claim:review:Setr:2026-05-04T08:11:35Z] |
Collaborator
Author
|
[release:review:Setr:2026-05-04T08:12:58Z] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
R_final ship-decision PR for #228. Runs the bake-off harness (PR #397, commit
61ab575) end-to-end against the synthetic corpus across three configurations, applies spec § "Adoption criteria for v2.0 ship", and lands the verdict + result JSONs.Verdict
Defer the wonder offline-generation line item from v2.0. The v2.0
wondersurface ships as on-line wonder-prompted generation only.Trigger: spec rule 3 ("junk rate > 60%"). All three strategies exceed the 60% threshold; STS at 88.9% is the decisive failure.
Verdict is robust across
feedback_budget∈ {8, 16, 32} andn_atoms_per_topic∈ {25, 50}: budget=8 trips rule 4 (drop) because no phantom can hit α≥12; budget∈{16,32} returns defer.The offline-generation premise is not falsified — RW and TC clear the H0+10pp adoption floor (0.165) on confirmation rate. The runner/strategies/evaluator stay in tree (
src/aelfrice/wonder/) so a future corpus or threshold revisit can re-run without re-implementation.What's in tree
docs/v2_wonder_consolidation_R_final.md— the result memo, 71 lines. Cites the spec, applies the four-rule decision tree, calls out honest limitations (synthetic-only corpus; spec rule 2 fires on a single-strategy threshold breach which is harsher than an alternative all-strategies reading).docs/bake_off_results/R0_default.json— default sweep, 10 seeds.docs/bake_off_results/R_a50_b8.json— low-budget sweep.docs/bake_off_results/R_a50_b32.json— high-budget sweep.No code changes; the harness shipped in #397.
What this is not
src/aelfrice/wonder/stays.wonder_ingest/wonder_gc), or retrieval-side surfacing of phantoms — those remain separately tracked and unaffected.Closes #228.
Summary by Sourcery
Document the bake-off results and ship decision for the v2.0 wonder offline-generation line item and check in the associated result artifacts.
Documentation:
Chores: