feat: bench-gate harness — corpus-root fixture, six module stubs, lab runner (#319) - #320
Merged
Conversation
Public-side scaffold for the v2.0 bench-gate harness. Resolves AELFRICE_CORPUS_ROOT to a directory and exposes load_corpus_module() to bench-gate tests. Public CI passes without corpus access; lab runners export the env var to point at ~/projects/aelfrice-lab/tests/corpus/v2_0. No corpus content moves; the locked rule on ~/.claude/-derived content is respected.
One pytest entry-point per bench-gated module — #197 dedup, #199 enforcement, #201 contradiction, #228 wonder_consolidation, #229 promotion_trigger, #193 sentiment. Each loads the lab corpus via the AELFRICE_CORPUS_ROOT fixture and skips when the detector module isn't yet implemented. Public CI: 6 skipped (corpus absent). Lab CI with corpus mounted: 6 skipped (detectors not yet shipped) — flips to real assertions as each detector lands.
Defaults AELFRICE_CORPUS_ROOT to ~/projects/aelfrice-lab/tests/corpus/v2_0 matching the two-repo layout in CLAUDE.md. Errors clearly when the corpus dir is missing rather than silently skipping. The script lives public-side (path string only, no derived content).
Reviewer's GuideIntroduces a public bench-gate harness for the v2.0 evaluation corpus driven by an AELFRICE_CORPUS_ROOT-based pytest fixture, adds six per-module bench-gated tests that currently stub detector imports but fully exercise corpus loading, wires the new bench_gated marker into pytest config, documents lab-side corpus mounting, and provides a convenience script to run the suite in lab environments while ensuring public CI skips cleanly without the private corpus. Sequence diagram for bench_gate harness execution via run_bench_gate.shsequenceDiagram
actor LabUser
participant run_bench_gate_sh as run_bench_gate_sh
participant Env as environment
participant Pytest as pytest
participant Conftest as tests_conftest
participant CorpusFixture as corpus_root_fixture
participant BenchTests as bench_gate_tests
LabUser->>run_bench_gate_sh: invoke script
run_bench_gate_sh->>Env: read AELFRICE_CORPUS_ROOT
alt env_var_unset
Env-->>run_bench_gate_sh: unset
run_bench_gate_sh->>Env: set default corpus path
else env_var_set
Env-->>run_bench_gate_sh: existing value
end
run_bench_gate_sh->>Env: export AELFRICE_CORPUS_ROOT
run_bench_gate_sh->>Env: check corpus directory exists
alt corpus_dir_missing
run_bench_gate_sh-->>LabUser: print error and exit 1
else corpus_dir_present
run_bench_gate_sh->>Pytest: exec uv run pytest tests/bench_gate/ -m bench_gated
Pytest->>Conftest: load configuration and fixtures
Conftest->>Pytest: register bench_gated marker
Pytest->>CorpusFixture: request corpus_root fixture
CorpusFixture->>Env: read AELFRICE_CORPUS_ROOT
Env-->>CorpusFixture: corpus path
CorpusFixture-->>Pytest: provide corpus_root path
loop each_bench_gate_test
Pytest->>BenchTests: collect and run test
BenchTests->>Conftest: apply autouse bench_gated marker
Conftest->>Env: check AELFRICE_CORPUS_ROOT and per_module_dir
alt corpus_available_for_module
BenchTests->>CorpusFixture: load JSONL from corpus_root
BenchTests-->>Pytest: test passes or fails
else corpus_missing_for_module
Conftest-->>Pytest: mark test as skipped
end
end
Pytest-->>LabUser: report results
end
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughAdds a bench-gate pytest harness that reads a lab-mounted v2.0 corpus via AELFRICE_CORPUS_ROOT, new session fixtures and JSONL loaders, six bench-gated tests that validate detector outputs against labeled corpus rows, a lab-run shell entrypoint, and README updates. Tests skip when corpus is absent. Changes
Sequence Diagram(s)sequenceDiagram
participant LabRunner as Lab Runner (scripts/run_bench_gate.sh)
participant ShellEnv as Shell / Env
participant Pytest as Pytest (uv)
participant Conftest as tests/conftest.py
participant FS as Corpus FS (AELFRICE_CORPUS_ROOT)
participant Detector as aelfrice.<module>
LabRunner->>ShellEnv: set/export AELFRICE_CORPUS_ROOT
ShellEnv->>FS: validate directory exists
LabRunner->>Pytest: invoke pytest tests/bench_gate/ -m bench_gated
Pytest->>Conftest: request session fixtures / autouse guard
Conftest->>FS: load JSONL via load_corpus_module(module)
Pytest->>Detector: import aelfrice.<module> (skip if missing)
Pytest->>Detector: call classify/score on each row
Detector-->>Pytest: prediction
Pytest->>Pytest: compute accuracy, assert threshold or skip/fail
(Note: colored rectangles not required for this simple sequence.) Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related issues
Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- Consider moving
load_corpus_module(and any other reusable helpers) out oftests/conftest.pyinto a dedicated helper module undertests/so test files don’t need to import fromconftest, which is treated specially by pytest. - For the bench-gate tests that
try: from aelfrice import ...and then call.classify/.decide, consider also guarding against missing attributes (e.g., viagetattrandpytest.skipif absent) so partially implemented modules don’t cause hard failures in the bench harness.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Consider moving `load_corpus_module` (and any other reusable helpers) out of `tests/conftest.py` into a dedicated helper module under `tests/` so test files don’t need to import from `conftest`, which is treated specially by pytest.
- For the bench-gate tests that `try: from aelfrice import ...` and then call `.classify` / `.decide`, consider also guarding against missing attributes (e.g., via `getattr` and `pytest.skip` if absent) so partially implemented modules don’t cause hard failures in the bench harness.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/bench_gate/test_contradiction.py`:
- Around line 14-17: Replace the broad try/except ImportError around importing
aelfrice.relationship_detector with a presence check using
importlib.util.find_spec("aelfrice.relationship_detector") and only call
pytest.skip("contradiction detector not yet implemented (`#201`)") if find_spec
returns None; then perform a normal import of aelfrice.relationship_detector so
real import-time errors within relationship_detector are propagated rather than
being masked by the except clause.
In `@tests/bench_gate/test_dedup.py`:
- Around line 29-30: Add a guard to ensure rows is non-empty before computing
accuracy to avoid ZeroDivisionError: assert len(rows) > 0 (or assert rows) with
a clear message, then compute accuracy = correct / len(rows) and assert accuracy
>= 0.5; update the test in test_dedup.py around the accuracy calculation
(variables: accuracy, correct, rows) to perform the non-empty assertion first.
- Around line 19-22: Replace the broad except ImportError around the import of
"from aelfrice import dedup" with a narrow except ModuleNotFoundError that calls
pytest.skip(...), and re-raise any other exceptions so import-time failures
inside aelfrice.dedup surface; apply the same change to the equivalent import
blocks in the six bench-gate test files (test_dedup.py, test_contradiction.py,
test_enforcement.py, test_promotion_trigger.py, test_sentiment.py,
test_wonder_consolidation.py).
In `@tests/conftest.py`:
- Around line 31-44: Add an autouse fixture that enforces skipping any test
marked with `@pytest.mark.bench_gated` when the corpus root is unavailable: create
a new fixture (e.g., skip_bench_gated_without_corpus) with autouse=True that
inspects request.node.get_closest_marker("bench_gated") (or
request.node.iter_markers) and, if present and _corpus_root() returns None,
calls pytest.skip with the same message used in aelfrice_corpus_root
(referencing CORPUS_ENV_VAR and the existing skip message) so tests that forget
to request aelfrice_corpus_root still get skipped on CI.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 03724bed-5a14-4707-b084-6d06be23f93a
📒 Files selected for processing (11)
pyproject.tomlscripts/run_bench_gate.shtests/bench_gate/__init__.pytests/bench_gate/test_contradiction.pytests/bench_gate/test_dedup.pytests/bench_gate/test_enforcement.pytests/bench_gate/test_promotion_trigger.pytests/bench_gate/test_sentiment.pytests/bench_gate/test_wonder_consolidation.pytests/conftest.pytests/corpus/v2_0/README.md
CodeRabbit review on #320: - conftest.py grows an autouse fixture keyed on the bench_gated marker. Backstops the aelfrice_corpus_root fixture so a marker-only test that forgets to request the fixture still skips on public CI. - All six bench-gate tests narrow except ImportError to ModuleNotFoundError scoped to the missing detector module name; any other ImportError (broken submodule import, missing transitive dep) now surfaces instead of being masked as a skip. - test_dedup.py adds an explicit non-empty rows assertion before the accuracy division. uv run pytest tests/bench_gate/ -> 6 skipped, 0 failed (corpus absent).
This was referenced May 3, 2026
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #319.
Public-repo bench-gate harness. No corpus content moves.
What lands
tests/conftest.py—corpus_rootfixture + autousebench_gatedmarker that skips whenAELFRICE_CORPUS_ROOTis unset or the per-module dir is empty.tests/bench_gate/test_*.py— six per-module entry-points (dedup Deduplication module (dedup) — v2.0 evaluation #197, enforcement Enforcement module — directive detection + compliance audit + selective injection #199, contradiction Semantic contradiction detector (relationship_detector) — v2.0 evaluation #201, wonder-consolidation [v2.0 research] wonder-consolidation — phantom generation strategy bake-off (RW vs TC vs STS) #228, promotion-trigger [v2.0] Phantom promotion-trigger rule — three rejected naive triggers, need a benchmarked rule #229, sentiment Evaluate sentiment-from-prose feedback for v2.0 (privacy + determinism review) #193). Detector calls are stubbed; harness just loads JSONL via the fixture.scripts/run_bench_gate.sh— lab-side runner that exportsAELFRICE_CORPUS_ROOTand runs the suite.tests/corpus/v2_0/README.md— points atAELFRICE_CORPUS_ROOT+ lab-mount instructions; clears the Rebuild redesign: eval harness — per-rebuild log + fixed-corpus precision #288 confusion.pyproject.toml— registers thebench_gatedmarker.Verification
uv run pytest tests/bench_gate/→ 6 skipped, 0 failed (corpus absent on public repo). Skip reasons referenceAELFRICE_CORPUS_ROOT.Out of scope
Detector implementations (live in #193/#197/#199/#201/#228/#229). Corpus content (lab repo, governed by #307).
Summary by Sourcery
Introduce a v2.0 bench-gate harness wired to an external evaluation corpus, keeping public CI green by skipping when the lab corpus is unavailable.
New Features:
Enhancements:
Documentation:
Tests:
Summary by CodeRabbit
New Features
Tests
Documentation