refactor: drop benchmarks/-importing targets from aelf bench (closes #342)

[robotrocketscience]

Summary

Closes #342, supersedes #329.

src/aelfrice/cli.py reached into the dev-only benchmarks/ tree at three points (verify-clean, longmemeval-score, posterior-residual). For wheel users those subcommands always failed with the "requires the source tree" hint; for dev users they were a redundant alias for the underlying scripts. Each one tied a runtime CLI surface to dev-script knowledge — deptry flagged this as DEP001 and PR #325 silenced it via an allowlist.

This PR removes the layering violation outright. The three subcommands are replaced with module-level entry points runnable from a source checkout, and the deptry allowlist for benchmarks is dropped.

Behavior

Before	After
aelf bench verify-clean PATH ...	python -m benchmarks.verify_clean PATH ...
aelf bench longmemeval-score PREDS GT JUDGE	python -m benchmarks.longmemeval_score PREDS GT JUDGE
aelf bench posterior-residual [flags]	python -m benchmarks.posterior_ranking [flags]

The unknown-target error in aelf bench lists the three moved entry points so a user typing the old subcommand gets a clean redirect:

$ aelf bench verify-clean
aelf bench verify-clean has moved. Run `python -m benchmarks.verify_clean ...` from a source checkout.

aelf bench (default / synthetic) and the _BENCH_INERT_TARGETS placeholders are unchanged.

Implementation notes

• benchmarks/verify_clean.py and benchmarks/longmemeval_score.py already had if __name__ == "__main__": blocks, so python -m works against them as-is.
• benchmarks/posterior_ranking/ is a package — added __main__.py that mirrors the argparse surface previously living inside cli.py (--fixtures, --seeds, --mrr-threshold, --ece-threshold, --json, --heat-kernel).
• benchmarks/longmemeval_budget_sweep.py had a bare from longmemeval_adapter import ... that worked only because of the DEP001 ignore. Switched to from benchmarks.longmemeval_adapter import ... so deptry resolves it as first-party.
• pyproject.toml: removed DEP001 = ["benchmarks"] from [tool.deptry.per_rule_ignores]. deptry . is clean.
• tests/test_benchmarks_dir.py: replaced the dispatch-to-module assertions with redirect-message assertions; added an unknown-target case.
• tests/test_posterior_ranking_eval.py: rewrote the three CLI integration tests to invoke benchmarks.posterior_ranking.__main__.main directly with the same argparse flags. Dropped the unused _run_cli helper + its imports.
• Doc updates: benchmarks/README.md, docs/BENCHMARKS.md, docs/bayesian_ranking.md, plus CHANGELOG Unreleased entry.

Tradeoffs

Wheel users who somehow had the source tree alongside an installed wheel and were running these three subcommands lose them. Mitigation: the redirect message points them at the new entry points. Acceptable — these subcommands were guarded against the wheel-only case anyway.

Test plan

• uv run pytest (1981 passed, 20 skipped)
• uvx deptry . clean
• discretion grep clean

Summary by Sourcery

Move dev-only benchmark subcommands out of the aelf bench CLI into module-level entry points and update messaging, tests, and docs accordingly.

New Features:

• Add python -m benchmarks.posterior_ranking as the command-line entry point for the posterior ranking benchmark.

Enhancements:

• Replace aelf bench dev-only targets (verify-clean, longmemeval-score, posterior-residual) with redirect messages pointing to the corresponding python -m benchmarks.<module> invocations.
• Simplify the aelf bench unknown-target error to list only supported targets and separately mention the moved dev-only benchmarks.
• Clean up deptry configuration by removing the DEP001 ignore for the benchmarks package and fixing an import in longmemeval_budget_sweep so it resolves as first-party.

Documentation:

• Update benchmark documentation and README to use the new python -m benchmarks.* entry points and describe the new behavior in the changelog.

Tests:

• Adjust benchmark CLI tests to assert redirect messages for moved dev-only targets, update unknown-target coverage, and rework posterior ranking CLI integration tests to call the new module entry point directly.

[sourcery-ai]

Reviewer's Guide

Refactors the aelf bench CLI to stop reaching into the dev-only benchmarks/ package by removing three benchmark-specific subcommands, replacing them with standalone module entry points under benchmarks, updating error/unknown-target messaging, tests, deptry config, and docs accordingly.

Sequence diagram for python -m benchmarks.posterior_ranking CLI entry

sequenceDiagram
    actor User
    participant PythonInterpreter
    participant BenchPosteriorMain as benchmarks_posterior_ranking___main__
    participant PosteriorRun as benchmarks_posterior_ranking_run

    User->>PythonInterpreter: run python -m benchmarks.posterior_ranking [flags]
    PythonInterpreter->>BenchPosteriorMain: import benchmarks.posterior_ranking.__main__
    PythonInterpreter->>BenchPosteriorMain: call main(argv)

    BenchPosteriorMain->>BenchPosteriorMain: parser = ArgumentParser(...)
    BenchPosteriorMain->>BenchPosteriorMain: parser.add_argument(... for fixtures, seeds, thresholds, json_out, heat_kernel)
    BenchPosteriorMain->>BenchPosteriorMain: args = parser.parse_args(argv)

    alt fixtures not provided
        BenchPosteriorMain->>BenchPosteriorMain: fixtures_path = _default_fixtures()
    else fixtures provided
        BenchPosteriorMain->>BenchPosteriorMain: fixtures_path = Path(args.fixtures)
    end

    BenchPosteriorMain->>PosteriorRun: run(fixtures_path, n_seeds=args.seeds, mrr_threshold=args.mrr_threshold, ece_threshold=args.ece_threshold, heat_kernel=args.heat_kernel)
    PosteriorRun-->>BenchPosteriorMain: result dict with mrr, ece, overall_pass

    alt args.json_out is True
        BenchPosteriorMain->>BenchPosteriorMain: payload = json.dumps({mrr, ece, overall_pass})
        BenchPosteriorMain->>User: print payload (JSON)
    else args.json_out is False
        BenchPosteriorMain->>BenchPosteriorMain: format human-readable summary from result
        BenchPosteriorMain->>User: print summary text
    end

    BenchPosteriorMain-->>PythonInterpreter: return exit_code (0 if overall_pass else 1)
    PythonInterpreter-->>User: process exit with exit_code

Loading

Sequence diagram for aelf bench handling moved dev-only targets

sequenceDiagram
    actor User
    participant AelfCli as aelf_cli_entry
    participant CmdBench as _cmd_bench

    User->>AelfCli: run aelf bench target [rest]
    AelfCli->>CmdBench: call _cmd_bench(args, out)

    CmdBench->>CmdBench: parse args.target into target

    alt target in _DEV_TARGETS_MOVED
        CmdBench->>CmdBench: lookup redirect = _DEV_TARGETS_MOVED[target]
        CmdBench->>User: print "aelf bench target has moved. Run `redirect ...` from a source checkout."
        CmdBench-->>AelfCli: return 2
    else target in _BENCH_INERT_TARGETS
        CmdBench->>CmdBench: map target to phase via _BENCH_INERT_TARGETS
        CmdBench->>User: run synthetic benchmark for phase
        CmdBench-->>AelfCli: return synthetic exit code
    else unknown target
        CmdBench->>CmdBench: known_targets = sorted(_BENCH_INERT_TARGETS)
        CmdBench->>CmdBench: moved_targets = sorted(_DEV_TARGETS_MOVED)
        CmdBench->>User: print unknown-target error with known_targets and moved_targets hint
        CmdBench-->>AelfCli: return 2
    end

    AelfCli-->>User: process exit with exit code

Loading

File-Level Changes

Change	Details	Files
Simplify aelf bench CLI by removing dev-only benchmark subcommands and replacing them with redirect messaging to module entry points.	Delete inlined implementations of the verify-clean, longmemeval-score, and posterior-residual targets from _cmd_bench and any associated argparse wiring. Introduce a _DEV_TARGETS_MOVED map for these targets to their new python -m benchmarks.<name> commands and, when invoked, print a relocation message and exit with code 2. Adjust the unknown-target error text to list only the standard targets plus a note that dev-only benchmarks are available via python -m benchmarks.<name> and enumerate the moved targets.	src/aelfrice/cli.py
Create a proper python -m benchmarks.posterior_ranking entry point that mirrors the previous CLI surface.	Add benchmarks/posterior_ranking/__main__.py with an argparse-based main(argv) function exposing --fixtures, --seeds, --mrr-threshold, --ece-threshold, --json, and --heat-kernel flags, delegating work to benchmarks.posterior_ranking.run.run and returning an appropriate exit code. Implement default fixtures path resolution inside __main__.py (mirroring the prior logic in cli.py) and identical human/JSON output formatting to the removed posterior-residual subcommand. Update the module docstring in benchmarks/posterior_ranking/run.py to describe it as the entry point for python -m benchmarks.posterior_ranking rather than aelf bench posterior-residual.	benchmarks/posterior_ranking/__main__.py benchmarks/posterior_ranking/run.py
Update tests to target the new module-level entry point and the changed CLI behavior.	Replace the aelf bench posterior-residual CLI integration tests with tests that import and call benchmarks.posterior_ranking.__main__.main directly, using pytest capsys to capture stdout; drop the now-unused _run_cli helper and imports. Change tests/test_benchmarks_dir.py from asserting dispatch into benchmark modules to asserting that the three dev-only targets print redirect messages referencing the corresponding python -m benchmarks.<name> commands and exit 2. Add a test that unknown aelf bench targets mention the moved dev-only targets in the error output.	tests/test_posterior_ranking_eval.py tests/test_benchmarks_dir.py
Align documentation and changelog with the new benchmark invocation pattern.	Update benchmark usage examples in benchmarks/README.md, docs/BENCHMARKS.md, and docs/bayesian_ranking.md to call python -m benchmarks.verify_clean, python -m benchmarks.longmemeval_score, and python -m benchmarks.posterior_ranking instead of aelf bench .... Add an Unreleased changelog entry describing the removal of dev-only aelf bench targets, their new module entry points, the updated unknown-target behavior, and the removal of the deptry ignore for benchmarks.	benchmarks/README.md docs/BENCHMARKS.md docs/bayesian_ranking.md CHANGELOG.md
Clean up deptry configuration and an internal benchmark import to reflect benchmarks as a first-party package.	Remove the DEP001 = ["benchmarks"] per-rule ignore from the [tool.deptry.per_rule_ignores] section so deptry no longer treats benchmark imports as ignored dangling dependencies. Change benchmarks/longmemeval_budget_sweep.py to import its adapter via from benchmarks.longmemeval_adapter import ... instead of a bare from longmemeval_adapter import ... so deptry sees it as an intra-package import.	pyproject.toml benchmarks/longmemeval_budget_sweep.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#342	Remove the three benchmarks/-importing aelf bench targets (verify-clean, longmemeval-score, posterior-residual) from src/aelfrice/cli.py so the runtime no longer imports dev-only benchmarks/, while keeping the default aelf bench synthetic target and _BENCH_INERT_TARGETS behavior intact and updating the unknown-target messaging appropriately.	✅
#342	Expose the three benchmarks as module entry points runnable from a source checkout with equivalent behavior: python -m benchmarks.verify_clean PATH ..., python -m benchmarks.longmemeval_score PREDS GT JUDGE, and python -m benchmarks.posterior_ranking [--fixtures ... --seeds N --mrr-threshold X --ece-threshold Y --json --heat-kernel].	✅
#342	Remove the deptry DEP001 ignore for benchmarks from pyproject.toml, and update documentation and tests (including CHANGELOG) to use the new python -m benchmarks.<name> entry points instead of the removed aelf bench subcommands, ensuring deptry and existing tests pass.	✅

Possibly linked issues

• refactor: drop benchmarks/-importing targets from aelf bench (supersedes #329) #342: They describe the same refactor: removing benchmarks imports from aelf bench, adding module runners, and fixing deptry.
• refactor: drop benchmarks/-importing targets from aelf bench (supersedes #329) #342: They address the same DEP001 problem by removing cli.py’s benchmarks imports, dropping the allowlist, and preserving benchmark usability.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 44 minutes and 2 seconds before requesting another review.

To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1b227d75-47ab-4d3b-b1de-09b8dd5429ef

📥 Commits

Reviewing files that changed from the base of the PR and between ba0101c and 9d0d86e.

⛔ Files ignored due to path filters (1)

• CHANGELOG.md is excluded by !**/CHANGELOG.md

📒 Files selected for processing (10)

• benchmarks/README.md
• benchmarks/longmemeval_budget_sweep.py
• benchmarks/posterior_ranking/__main__.py
• benchmarks/posterior_ranking/run.py
• docs/BENCHMARKS.md
• docs/bayesian_ranking.md
• pyproject.toml
• src/aelfrice/cli.py
• tests/test_benchmarks_dir.py
• tests/test_posterior_ranking_eval.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch refactor/issue-342-drop-bench-targets

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 44 minutes and 2 seconds.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="CHANGELOG.md" line_range="15" />
<code_context>
+- **Dev-only benchmark targets moved out of the `aelf` CLI** ([#342](https://github.com/robotrocketscience/aelfrice/issues/342)). `aelf bench verify-clean | longmemeval-score | posterior-residual` were always-failing in shipped wheels (the `benchmarks/` tree is dev-only and not packaged) and put dev-script knowledge in the runtime CLI surface. Replaced with module-level entry points runnable from a source checkout: `python -m benchmarks.verify_clean PATH ...`, `python -m benchmarks.longmemeval_score PREDS GT JUDGE`, `python -m benchmarks.posterior_ranking [--fixtures ... --seeds N --mrr-threshold X --ece-threshold Y --json --heat-kernel]`. The unknown-target error in `aelf bench` now points at the new entry points. The default `aelf bench` (synthetic harness in `src/aelfrice/benchmark.py`) and the `_BENCH_INERT_TARGETS` placeholders are unchanged. The deptry `DEP001 = ["benchmarks"]` ignore is removed — `src/aelfrice` no longer imports the dev-only package.
</code_context>
<issue_to_address>
**issue:** The documented arguments for `benchmarks.longmemeval_score` here differ from the other docs in this repo.

Here you show `benchmarks.longmemeval_score` taking three arguments (`PREDS GT JUDGE`), while other docs (e.g., benchmark README and BENCHMARKS) show only two. Please update this entry or the other docs so they all describe the same invocation signature for `longmemeval_score`.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue: The documented arguments for benchmarks.longmemeval_score here differ from the other docs in this repo.

Here you show benchmarks.longmemeval_score taking three arguments (PREDS GT JUDGE), while other docs (e.g., benchmark README and BENCHMARKS) show only two. Please update this entry or the other docs so they all describe the same invocation signature for longmemeval_score.