evidence(ai-search): re-run controlled entity-and-offer questions after the llms/offer mirror went live - #43
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
📝 WalkthroughWalkthroughThe AI-search evidence was rerun on 2026-08-09. Fixtures and the embedded audit were refreshed. Rerun history rules and stricter validation were added. The q5 ground truth now describes the current tinystudio.io offer. ChangesAI-search evidence maintenance
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: efbc355717
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| { | ||
| "schema": "ai-search-evidence/v1", | ||
| "testedOn": "2026-08-06", | ||
| "testedOn": "2026-08-09", |
There was a problem hiding this comment.
When this mixed-date fixture is rendered, public/audit.js labels the entire artifact “Controlled test, 2026-08-09” using only testedOn, while the ChatGPT and Perplexity records still have testedAt: 2026-08-06 and per-run dates are never displayed. The public evidence therefore makes those two stale blockers appear to belong to the August 9 test; either rerun them or render their individual dates/a date range.
Useful? React with 👍 / 👎.
| A re-run of the same question-and-engine pair replaces that pair's run | ||
| with the fresh capture (one run per pair, `testedOn` moved to the capture | ||
| day); the prior capture stays in git history, so the before/after remains |
There was a problem hiding this comment.
When a future rerun is accidentally appended instead of replacing its prior question-and-engine record, neither scripts/check-site.mjs nor the UI tests reject the duplicate; the audit renderer will count and display both captures as current runs. Add uniqueness validation for the (questionId, engine) pair so the newly documented replacement invariant cannot silently corrupt the evidence artifact.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/evidence/ai-search/2026-08-09-controlled-rerun.md`:
- Line 118: Update the fenced code block at the documented verification commands
near line 118 to specify the sh language on its opening fence, resolving the
MD040 markdownlint violation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 9d4e7238-6fac-46b0-8539-0710a5b837f5
📒 Files selected for processing (6)
docs/evidence/ai-search/2026-08-09-controlled-rerun.mdevidence-fixtures/ai-search/README.mdevidence-fixtures/ai-search/evidence.jsonpublic/audit.htmlscripts/check-site.mjsscripts/test-agent-ui.mjs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c890b74ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (!AI_STATES.includes(run.state)) failures.push(`AI-search run has an unknown state: ${run.state}`); | ||
| if (!questionIds.has(run.questionId)) failures.push(`AI-search run references an unknown question: ${run.questionId}`); | ||
| if (!engines.has(run.engine)) failures.push(`AI-search run references an unknown engine: ${run.engine}`); | ||
| if (!run.testedAt) failures.push(`AI-search run must record when it was tested: ${run.questionId}/${run.engine}`); |
There was a problem hiding this comment.
When a future run supplies a truthy non-date such as true, {}, or "unknown", this guard and the matching UI assertion both pass, allowing the published evidence to lack the date required by the fixture contract. Validate that testedAt is a string in the expected date format rather than checking only its truthiness.
Useful? React with 👍 / 👎.
| ], | ||
| "remediation": { | ||
| "text": "The answer listed six different studios, none of them tinystudio.io. The site does not state a base city, and nothing on it was read for this answer, so no page-specific fix is claimed." | ||
| "text": "The answer listed four different studios in four cities, none of them tinystudio.io. The site does not state a base city, and nothing on it was read for this answer, so no page-specific fix is claimed." |
There was a problem hiding this comment.
The captured answer names Jakarta, Paris, London, Manchester, and North Hollywood/Los Angeles—five cities across four studio entries—but the remediation summarizes this as four cities. Because this text is published as part of the evidence artifact, correct the count or avoid stating one so the summary matches the capture.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
|
Refreshed against current main (fleet lane re-verify, 2026-08-11): branch merged up to main head 6f24bff; fixed the outstanding CodeRabbit MD040 nit (verification code fence now tagged |
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
bca445f to
d98c812
Compare
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
d98c812 to
6ce93d6
Compare
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
6ce93d6 to
877bb1a
Compare
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
… after the llms/offer mirror went live Fresh 2026-08-09 captures for the same named questions on the same engines (Google AI Overview q1-q7, Bing q1, DuckDuckGo q1): all nine runs are Wrong, and q7 moves absent -> wrong — the pricing query now produces an AI answer built from the stale Agent Desk index entry, contradicting the live pricing page. No Found transition, so the backlog item honestly stays open; the 2026-08-06 captures remain in git history and nothing was relabelled. Checks now require testedAt on every run and forbid sources on absent runs; the fixture README documents that a re-run replaces the pair's run.
… and merge current main
…, not the retired Agent Desk
The q5 controlled question ("What is tinystudio.io?") grounded its truth in
the self-serve Agent Desk, which is retired and no longer the offer. The
2026-08-09 controlled re-run left this as the open item: all nine fresh runs
were Wrong, and the stale ground truth itself still named the Agent Desk as
part of the offer. The truth now matches the live site's own identity
statement — the free leak audit of high-ticket service homepages and the
human-reviewed desk that closes what the audit finds — so the fixture's
yardstick no longer encodes the retired product. The audit page's embedded
AI-search bundle is regenerated from the fixtures so the drift guard passes,
and a receipt records the alignment.
Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
877bb1a to
ed62202
Compare
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/evidence/ai-search/2026-08-11-q5-ground-truth-alignment.md`:
- Around line 55-57: Revise the unchanged-markup claim in the evidence document
to qualify that source-of-truth public pages were unchanged, while explicitly
noting that public/audit.html was regenerated and its embedded evidence content
changed. Keep the references to llms.txt, offer.md, and existing mirror checks
accurate.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 32999b46-4480-4a08-b64e-83919e4e58a2
📒 Files selected for processing (5)
docs/evidence/ai-search/2026-08-09-controlled-rerun.mddocs/evidence/ai-search/2026-08-11-q5-ground-truth-alignment.mdevidence-fixtures/ai-search/controlled-questions.jsonpublic/audit.htmlscripts/check-site.mjs
🚧 Files skipped from review as they are similar to previous changes (2)
- scripts/check-site.mjs
- docs/evidence/ai-search/2026-08-09-controlled-rerun.md
| - No public page markup, `llms.txt`, `offer.md`, or check script changed; | ||
| every fact in the new truth was already asserted on the live pages and by | ||
| the existing mirror checks. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Correct the claim about unchanged public markup.
public/audit.html changes in this PR. Its embedded ai-search-evidence bundle changes at Line 121, and the surrounding evidence copy changes at Lines 117-120. This conflicts with “No public page markup ... changed” at Lines 55-57. If the statement refers only to source-of-truth pages, qualify it and state that public/audit.html was regenerated.
Proposed wording
- No public page markup, `llms.txt`, `offer.md`, or check script changed;
+ No source-of-truth page markup, `llms.txt`, `offer.md`, or check script changed in this pass; `public/audit.html` was regenerated as the embedded-bundle output described above;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - No public page markup, `llms.txt`, `offer.md`, or check script changed; | |
| every fact in the new truth was already asserted on the live pages and by | |
| the existing mirror checks. | |
| - No source-of-truth page markup, `llms.txt`, `offer.md`, or check script changed in this pass; `public/audit.html` was regenerated as the embedded-bundle output described above; | |
| every fact in the new truth was already asserted on the live pages and by | |
| the existing mirror checks. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/evidence/ai-search/2026-08-11-q5-ground-truth-alignment.md` around lines
55 - 57, Revise the unchanged-markup claim in the evidence document to qualify
that source-of-truth public pages were unchanged, while explicitly noting that
public/audit.html was regenerated and its embedded evidence content changed.
Keep the references to llms.txt, offer.md, and existing mirror checks accurate.
…udit still teaches the retired Agent Desk (#153) Merged PR #43 (AI-search evidence re-run + q5 truth aligned to the current offer) is on origin/main ad9cee3 but not live: release-state pins 5864e39 (deployed before the merge), so live /audit still embeds the 2026-08-06 artifact with the retired Agent Desk framing, contradicting live llms.txt/offer.md. Receipt pins the gap (md5s, testedOn, q5 truth), verifies all other pages match main, names the deployment blocker (expired wrangler auth, Nish-reserved, already NOTIFY'd), and confirms npm run check/test pass on the to-be-shipped revision.
…gainst current GitHub state (#146)
…on current main and live (2026-08-14) (#186) The retired Agent Desk ground truth in the AI-search controlled questions (q5) was already replaced on origin/main (ed62202, merged via PR #43). This receipt re-verifies the replacement against the current head (e9fc96a) and the live deployment: the fixture truth, the audit-page embedded bundle (byte-identical to the fixtures), the homepage identity row, the llms.txt/offer.md mirror, npm run check and the 117-test suite all pass, and the served /audit, /llms.txt and /offer.md bytes match source. No captured run was relabelled. Co-authored-by: nish3451 <nish3451@users.noreply.github.com> Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
…ation closeout (2026-08-14)
…nt main (2026-08-14) (#189) * docs(evidence): re-verify PR #42 + #43 serial-merge closeout on current main (2026-08-14) Re-verifies the review item [unreviewed-by-opus] Merge PR #42 and #43 in serial order (or rebase the loser) against the current GitHub state on 2026-08-14. The 2026-08-12 closeout (PR #146, 4e2b94c) still holds: - PR #42 squash 95d2248 is in origin/main (f9214c1) - PR #43 merge ad9cee3 is in origin/main - git merge-base --is-ancestor 95d2248 ad9cee3 is true (serial order) - PR #42 declaration is on main; q8-conversion-audit (PR #102) layered on top - PR #43 branch tip ed62202 is an ancestor of main with zero unmerged content - No open PR #42 or #43 No code change is needed or proposed. * docs(lane-report): record lane 1 PR #42 + #43 serial-merge re-verification closeout (2026-08-14) --------- Co-authored-by: Nish <nish@tinystudio.io>
…#128) on current main (#211) * reconcile(evidence): land the stranded 2026-08-12 AI-search re-run on current main PR #128 (lane1/ai-search-rerun-entity-offer-20260812) carried the only fresh 2026-08-12 AI-search captures but went CONFLICTING against main and was stranded unowned, so the live /audit panel kept rendering the 2026-08-09 record. This re-lands the measurement on current main: - evidence-fixtures/ai-search/evidence.json: the 2026-08-12 DuckDuckGo (q1/q2/q5) and Bing (q1/q5/q7) captures, all Wrong or Absent, with testedOn moved to 2026-08-12; the 2026-08-06 Google captures and the not-tested runs are retained. - controlled-questions.json: unchanged - main's q5 ground truth stands (the retired Agent Desk wording PR #128 carried predates PR #43 and must not resurrect). - public/audit.html: embedded AI-search bundle regenerated from fixtures (drift guard passes). - scripts/test-agent-ui.mjs: strict-state set now expects absent, which the 2026-08-12 Bing runs reintroduced. - docs/evidence/ai-search/: original 2026-08-12 receipt with a reconciliation note, plus this pass's reconciliation receipt. Co-authored-by: CommandCodeBot <noreply@commandcode.ai> * docs(lane1): add lane report for the PR #128 AI-search reconciliation Co-authored-by: CommandCodeBot <noreply@commandcode.ai> --------- Co-authored-by: nish3451 <nish3451@users.noreply.github.com> Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
…nt main (2026-08-17, lane 1) (#244) Co-authored-by: minimax-vps <minimax-vps@fleet.local>
…nt main (2026-08-21, lane 1) (#269) Co-authored-by: minimax-vps <minimax-vps@local>
…on current main and live (2026-08-21) (#264) Re-verification against the current origin/main head (92d55c3) and the live deployment (Cloudflare release-state pinned at b4d80f1, 2026-08-17). The q5 truth in evidence-fixtures/ai-search/controlled-questions.json has been the current-offer wording since commit ed62202 (2026-08-11, PR #43), and is preserved verbatim on the current head and on the live served bytes (public/llms.txt and public/offer.md md5-match the live URLs). The drift guard, the bundle/fixture equality probe, and the full test suite (132 tests, 0 failures) all pass. No fixture, code, or page markup changed in this pass; the receipt records the closeout on the current head so the item cannot be re-opened by tracker drift. Co-authored-by: minimax-vps <minimax-vps@local> Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 participant
What this does
Controlled re-run of the AI-search evidence fixture (backlog item: "Re-establish verified AI-search entity and offer understanding after the 2026-08-08 15:04 live recheck").
The llms.txt/offer.md identity mirror (PR #19) has been merged and live since 2026-08-09 04:12 IST, but the
/auditAI-search panel still rendered the 2026-08-06 captures. This PR re-runs the same named questions on the same engines (anonymous sessions, English US) and records the fresh captures.Results (all captured verbatim, strict states)
tinystudio.io pricing) movesabsent→wrong: the pricing query now produces an AI Overview built from the stale Agent Desk index entry, claiming the site lists no pricing — contradicting the live pricing page. Verbatim quote truncated before an email address the answer published (the fixture must never capture emails).Foundtransition — the item's accept criterion is not met, so the backlog item honestly stays open. The 2026-08-06 captures remain in git history; nothing was relabelled.Follow-up: q5 ground truth aligned to the current offer (2026-08-11)
The 2026-08-09 re-run's receipt left one open item: the
q5controlled question ("What is tinystudio.io?") still grounded its truth in the retired self-serve Agent Desk. That ground truth is now corrected to the live site's own identity statement — the free leak audit of high-ticket service homepages and the human-reviewed desk that closes what the audit finds — and the audit page's embedded AI-search bundle is regenerated so the drift guard passes. The captured runs are untouched; only the yardstick the fixture checks answers against changed. Receipt:docs/evidence/ai-search/2026-08-11-q5-ground-truth-alignment.md.Also in this PR
scripts/check-site.mjs+scripts/test-agent-ui.mjs: every run must carrytestedAt;absentruns must not carry sources (README contract).evidence-fixtures/ai-search/README.md: a re-run replaces the pair's run; prior capture stays in git history.docs/evidence/ai-search/2026-08-09-controlled-rerun.md: full receipt with methodology, verbatim results, and what remains untested.Verification
npm run checkpassesnpm testpasses (6 headings + 7 sitemap + 53 worker + 15 ui)git diff --checkcleanNo ranking, lead, visibility, or revenue claim is made; this is a measurement recorded honestly.
Summary by CodeRabbit
Documentation
Updates
Validation