feat(retrieval): use_intentional_clustering flips to default-on (#436)

[robotrocketscience]

Summary

Flips use_intentional_clustering default False → True per the explicit ship-criteria contract in resolve_use_intentional_clustering():

"Default: False — ships behind the flag at v2.0.0; the bench gate (A2 in docs/feature-intentional-clustering.md) flips the default after lab-side benchmark evidence clears."

That gate cleared on the production multi-store sweep (lab-side, #436 R6 run 2026-05-09): 60/60 PASS at p99 0.328ms — ~15-30× margin under the 5ms A4 latency budget across the top-2 production stores.

Substrate (already on main)

• docs(feature-intentional-clustering): spec memo for #436 #455 — spec memo
• feat(clustering): RetrievalCluster module + multi-fact corpus mount (436) #496 — RetrievalCluster module + multi-fact corpus mount
• feat(retrieval): wire intentional clustering into retrieve_v2 (#436 Phase 2) #498 — wire intentional clustering into retrieve_v2
• feat(test): run_clustering_uplift driver — bench-gate scorer for #436 #504 — run_clustering_uplift bench-gate scorer

What changes

• src/aelfrice/retrieval.py:923 — return False → return True in resolve_use_intentional_clustering(). Precedence (env > kwarg > TOML > default) is unchanged.
• tests/test_clustering_integration.py — three test updates:
  • test_default_is_off → test_default_is_on
  • test_env_garbage_falls_through default-arm assertion flipped
  • test_default_call_byte_identical_to_explicit_off → test_default_call_byte_identical_to_explicit_on (the ON-byte-identity invariant supersedes the OFF-byte-identity invariant)
• CHANGELOG.md — [Unreleased] Changed entry mirroring the v2.1.0 [retrieval] Pipeline composition tracker — unified retrieve() with feature-flag gate #154 default-flip format.

Opt-out

Reversible for v2.0.x parity:

[retrieval]
use_intentional_clustering = false

…or AELFRICE_INTENTIONAL_CLUSTERING=0.

Out of scope

The lab-finding that raising DEFAULT_CLUSTER_EDGE_FLOOR from 0.4 → 0.6 triples uplift on 8 rows (by excluding EDGE_CITES inter-cluster edges) is queued as a separate follow-up so this PR stays minimal and reviewable.

Test plan

• pytest tests/test_clustering_integration.py — 10/10 PASS
• pytest tests/ -k 'clustering or retrieve_v2 or context_rebuilder' — 130/130 PASS, 4 skipped
• aelf-pr-open gate (rebase + full pytest + discretion grep + signed)
• CI green

Summary by Sourcery

Flip intentional clustering retrieval behavior to default-on while preserving existing override precedence and update tests and changelog accordingly.

New Features:

• Enable intentional clustering by default in retrieval flows, with opt-out via config or environment variable.

Enhancements:

• Clarify retrieval flag resolution documentation for the intentional clustering default behavior.

Documentation:

• Document the default-on use_intentional_clustering behavior and opt-out path in the changelog.

Tests:

• Update clustering and compression integration tests to reflect the default-on clustering behavior and its mutex with compression.

[sourcery-ai]

Reviewer's Guide

Flips the retrieval feature flag use_intentional_clustering from default-off to default-on while preserving the existing precedence rules (env > kwarg > TOML > default), updates tests to assert the new default behavior and maintain invariants, adjusts compression-related tests to explicitly disable clustering where the two features are currently mutex, and documents the flip in the changelog.

Sequence diagram for resolve_use_intentional_clustering with default-on behavior

sequenceDiagram
  participant Caller
  participant Retrieval as RetrievalModule
  participant Env as EnvSubsystem
  participant Toml as TomlConfig

  Caller->>Retrieval: resolve_use_intentional_clustering(explicit, start)
  Retrieval->>Env: _env_intentional_clustering_override()
  Env-->>Retrieval: env_value or None

  alt env_value is not None
    Retrieval-->>Caller: env_value
  else env_value is None
    alt explicit is not None
      Retrieval-->>Caller: explicit
    else explicit is None
      Retrieval->>Toml: _read_toml_flag_for(INTENTIONAL_CLUSTERING_FLAG, start)
      Toml-->>Retrieval: toml_value or None
      alt toml_value is not None
        Retrieval-->>Caller: toml_value
      else toml_value is None
        Retrieval-->>Caller: True
      end
    end
  end

Loading

File-Level Changes

Change	Details	Files
Flip use_intentional_clustering default to True while preserving resolution precedence.	Update resolve_use_intentional_clustering() docstring to describe the new default-on behavior and reference the cleared latency bench gate. Change the final fallback return value in resolve_use_intentional_clustering() from False to True, keeping env/kwarg/TOML precedence unchanged.	src/aelfrice/retrieval.py
Align clustering integration tests with the new default-on semantics and invariants.	Rename test_default_is_off to test_default_is_on and update the assertion to expect True when no overrides are present. Update test_env_garbage_falls_through to assert that invalid env content falls through to the new default of True, and that an explicit False kwarg still forces the flag off. Update the byte-identity invariant test to assert that the default call is identical to an explicit use_intentional_clustering=True call, renaming it and its docstring to reflect the ON-byte-identity invariant.	tests/test_clustering_integration.py
Make compression integration tests robust to clustering now being default-on and mutually exclusive with compression.	Pass use_intentional_clustering=False explicitly in compression tests that previously relied on the default-off behavior to satisfy the current mutex between clustering and type-aware compression. Ensure the env-only compression test explicitly disables clustering via the AELFRICE_INTENTIONAL_CLUSTERING=0 env var so it still exercises the "env alone enables compression" path with the new default.	tests/test_compression_integration.py
Document the default flip for use_intentional_clustering in the changelog.	Add a Changed entry under [Unreleased] describing the default flip, referencing prior substrate PRs, the bench gate results, the unchanged precedence ordering, and the opt-out configuration for v2.0.x parity.	CHANGELOG.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 3 minutes and 20 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3b62f105-3140-42b3-a7f4-ea8ca9cbe0de

📥 Commits

Reviewing files that changed from the base of the PR and between cff8df1 and fdf31f4.

⛔ Files ignored due to path filters (1)

• CHANGELOG.md is excluded by !**/CHANGELOG.md

📒 Files selected for processing (3)

• src/aelfrice/retrieval.py
• tests/test_clustering_integration.py
• tests/test_compression_integration.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/issue-436-default-flip

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• In test_env_var_alone_enables_compression, consider using the existing ENV_INTENTIONAL_CLUSTERING constant instead of the raw "AELFRICE_INTENTIONAL_CLUSTERING" string so env key usage stays consistent and refactor-safe.
• Now that use_intentional_clustering defaults to True and is mutex with use_type_aware_compression, it may be worth enforcing this at the retrieve_v2 boundary (e.g., raising or logging when both are enabled) instead of relying on scattered call-site overrides to avoid accidental co-activation.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In `test_env_var_alone_enables_compression`, consider using the existing `ENV_INTENTIONAL_CLUSTERING` constant instead of the raw `"AELFRICE_INTENTIONAL_CLUSTERING"` string so env key usage stays consistent and refactor-safe.
- Now that `use_intentional_clustering` defaults to `True` and is mutex with `use_type_aware_compression`, it may be worth enforcing this at the `retrieve_v2` boundary (e.g., raising or logging when both are enabled) instead of relying on scattered call-site overrides to avoid accidental co-activation.

## Individual Comments

### Comment 1
<location path="tests/test_compression_integration.py" line_range="190-196" />
<code_context>
     monkeypatch: pytest.MonkeyPatch, _isolated_cwd: Path
 ) -> None:
     monkeypatch.setenv(ENV_TYPE_AWARE_COMPRESSION, "1")
+    # Post-#436 default-flip, AELFRICE_INTENTIONAL_CLUSTERING must also
+    # be disabled in this scope to satisfy the v2.0.0 mutex (the cluster
+    # pack accounts in raw tokens; composing it with compressed cost is
+    # tracked as a v2.x follow-up). The test still meaningfully exercises
+    # "env var alone enables compression" — it just makes the clustering
+    # env-disable explicit instead of relying on the (now-flipped) default.
+    monkeypatch.setenv("AELFRICE_INTENTIONAL_CLUSTERING", "0")
     s = _populate_store()
     result = retrieve_v2(s, "sqlite system")  # no explicit kwarg
</code_context>
<issue_to_address>
**nitpick:** Avoid hard-coding the clustering env var name in tests

Use the same constant the implementation relies on (e.g. `ENV_INTENTIONAL_CLUSTERING`) instead of the string literal. This keeps the test aligned with production behavior and ensures it fails if the env var name changes, rather than silently diverging.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

nitpick: Avoid hard-coding the clustering env var name in tests

Use the same constant the implementation relies on (e.g. ENV_INTENTIONAL_CLUSTERING) instead of the string literal. This keeps the test aligned with production behavior and ensures it fails if the env var name changes, rather than silently diverging.

[robotrocketscience]

[claim:review:maxwell:2026-05-10T06:41:24Z]

[robotrocketscience]

[claim:review:faraday:2026-05-10T06:42:54Z]

[robotrocketscience]

[release:review:faraday:2026-05-10T06:42:58Z]

[robotrocketscience]

[claim:review:leibniz:2026-05-10T06:43:37Z]

[robotrocketscience]

[release:review:leibniz:2026-05-10T06:43:42Z]

[github-actions]

This PR is now behind main. Rebase locally so your commit signatures stay intact:

git fetch origin && git checkout 'feat/issue-436-default-flip' && git rebase origin/main
# resolve conflicts if any, then
git push --force-with-lease

Auto-rebase was removed because the bot has no signing key; rebasing as the bot strips author signatures and the required_signatures rule on main then blocks the merge. See #341.

[robotrocketscience]

[claim:review:leibniz:2026-05-10T06:49:43Z]

[robotrocketscience]

[release:review:leibniz:2026-05-10T06:49:47Z]

[robotrocketscience]

[release:review:maxwell:2026-05-10T06:53:20Z]