Cost 8.3b by preflight instead of guessing at it - #201
Merged
Conversation
Ran the episodic preflight -- free, zero provider calls, zero graph writes -- to replace an estimate I had been quoting as "roughly 25x everything else" with a measured figure. A 4-question episodic build is 47 extraction calls and 2.12M input tokens. Scaling to the ~30 questions the decision needs: ~350 calls and ~16M tokens per build, six builds for three per arm across two arms, so roughly 96M input tokens. More useful than the number: a reduced run is not partial progress here, it is forbidden by the task's own pre-registered decision rule. That rule ships a non-Ignore default only if the episodic mean gain exceeds that type's own noise band across three or more builds per arm. At four questions accuracy is quantised into 25% steps, so the band swamps any gain and the rule returns "leave it opt-in" -- a rule-compliant verdict that is really about sample size. That is the same trap 7.6 spent four runs demonstrating: a technically valid negative describing the instrument rather than the feature. Producing one here would be worse, because the pre-registered rule would lend it authority it had not earned. A decision rule is only worth having if it is not run at an n it cannot speak to. So 8.3b stays unrun, and what was a vague "this costs too much" is now a measured budget question someone can answer. Everything else is ready: episodic sampling preflights clean, 8.3a is merged, and 8.1 shipped first as the task requires. Docs only; no source changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
Adds a review note documenting the measured (preflight-estimated) token/call cost of running the episodic default evaluation (8.3b), and explains why running a cheaper/smaller sample would violate the pre-registered decision rule for defaulting.
Changes:
- Add a new review document capturing the preflight-measured extraction call count and estimated input tokens for a 4-question episodic build.
- Document the extrapolated cost for the expected ~30-question decision run and the resulting total budget implication.
- Explain why a reduced-n run is not “partial progress” under the task’s stated decision rule.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+9
to
+12
| ``` | ||
| frozen preparation preflight 47 calls for 182 source sessions | ||
| and 2,123,846 estimated input tokens | ||
| ``` |
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ran the episodic preflight — free, zero provider calls, zero graph writes — to replace an estimate I'd been quoting ("roughly 25× everything else") with a measured figure.
A 4-question episodic build is 47 extraction calls and 2.12M input tokens. Scaling to the ~30 questions the decision needs: ~350 calls and ~16M tokens per build; six builds (3/arm × 2 arms) ≈ 96M input tokens.
More useful than the number: a reduced run isn't partial progress here — it's forbidden by the task's own pre-registered decision rule.
At n=4, accuracy is quantised into 25% steps. The noise band swamps any plausible gain and the rule returns "leave it opt-in" — a rule-compliant verdict that is actually about sample size.
That's the same trap 7.6 spent four runs demonstrating: a technically valid negative describing the instrument rather than the feature. Producing one here would be worse, because the pre-registered rule would lend it authority it hadn't earned. A decision rule is only worth having if it isn't run at an n it can't speak to.
Everything else is ready:
--memory-types episodicpreflights clean, 8.3a is merged, and 8.1 shipped first as the task requires — without it, switching the mode on writes model-generated claims labelled as the user's.Docs only.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE