fix(public): refresh the study figures to the 2026-08-12 scan and guard the daily-refresh promise - #156
Conversation
The homepage, audit, pricing, and specimen pages promise "this number is today's" and "refreshed daily", but the served figures (53 of 89) came from the 2026-08-06 snapshot. The daily scan has produced newer snapshots (2026-08-09, 2026-08-11, 2026-08-12) that never reached this repo, and the scan's deploy step is skipped (no Cloudflare token env file), so the live claim drifted six days from reality. - Bring the three newer snapshots into study/snapshots/ (the daily record is the asset; render.py --check now passes against the true newest scan). - Re-run study/render.py to rewrite the data-study spans from the 2026-08-12 snapshot: 55 of 90 readable (was 53 of 89), 57 with no price (unchanged), nine blocked by bot protection (was ten). Verified: python3 study/render.py --check passes; npm run check and npm test pass (92 tests, 0 failures); git diff --check clean.
The site promises "refreshed daily" / "this number is today's" on four pages, but the served figures drifted six days (2026-08-06 -> 08-12) before anyone noticed, because nothing failed when the snapshots were not imported into the repo. Two new guards, wired into npm test (and therefore CI): - render.py --check: the pages' data-study spans must match the newest committed snapshot, so a partial import fails the suite. - Freshness: the newest snapshot must be at most 4 days old (the scan runs daily; import is manual), so a multi-day drift turns red instead of silently contradicting the public claim. Verified: npm test passes (94 tests, 0 failures).
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
Included review availability: 3 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour. 📝 WalkthroughWalkthroughThe PR adds eight dated study snapshots, updates public study figures to 91 readable sites, and adds freshness checks to the aggregate test command. ChangesStudy publication and freshness
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: ⚪ Minimal · up to The PR refreshes the published study figures and adds freshness checks to prevent silent drift; no actionable merge-blocking risk remains after normal checks and review. Sequence Diagram(s)sequenceDiagram
participant npm_test
participant test_study_freshness
participant study_render_py
participant study_snapshots
participant public_pages
npm_test->>test_study_freshness: run freshness checks
test_study_freshness->>study_snapshots: discover and validate dated snapshots
test_study_freshness->>study_render_py: run --check
study_render_py->>public_pages: verify rendered pages match the newest snapshot
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/test-study-freshness.mjs`:
- Around line 36-45: Update the freshness test around newestSnapshotFile to read
the selected snapshot JSON and use its scanned_at timestamp instead of the
filename date. Validate that the metadata date matches the YYYY-MM-DD filename,
reject invalid or future scanned_at values, and calculate age from scanned_at
while preserving the existing maximum-age assertion.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 99578cd1-8895-433d-89ec-3cd58dd7a8ea
📒 Files selected for processing (9)
package.jsonpublic/audit.htmlpublic/index.htmlpublic/pricing.htmlpublic/specimen.htmlscripts/test-study-freshness.mjsstudy/snapshots/2026-08-09.jsonstudy/snapshots/2026-08-11.jsonstudy/snapshots/2026-08-12.json
| test("the newest study snapshot is fresh enough to back the daily-refresh promise", () => { | ||
| const file = newestSnapshotFile(); | ||
| const date = file.replace(/\.json$/, ""); | ||
| const snapshotEnd = Date.parse(`${date}T23:59:59Z`); | ||
| assert.ok(Number.isFinite(snapshotEnd), `snapshot filename ${file} must be an ISO date (YYYY-MM-DD.json)`); | ||
|
|
||
| const ageDays = (Date.now() - snapshotEnd) / 86_400_000; | ||
| assert.ok( | ||
| ageDays <= MAX_SNAPSHOT_AGE_DAYS, | ||
| `newest snapshot is ${date} (${Math.floor(ageDays)} days old) but the site promises "refreshed daily" — import the newer snapshots from the scan checkout and re-run study/render.py (max allowed age: ${MAX_SNAPSHOT_AGE_DAYS} days)` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validate freshness from snapshot metadata.
Line 39 uses the end of the filename date instead of the scan time. A snapshot scanned at 01:30:29Z can remain valid for almost five days. A future-dated filename also produces a negative age and passes.
Read the selected JSON file. Validate that its date matches the filename. Calculate age from scanned_at. Reject scan times later than Date.now().
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/test-study-freshness.mjs` around lines 36 - 45, Update the freshness
test around newestSnapshotFile to read the selected snapshot JSON and use its
scanned_at timestamp instead of the filename date. Validate that the metadata
date matches the YYYY-MM-DD filename, reject invalid or future scanned_at
values, and calculate age from scanned_at while preserving the existing
maximum-age assertion.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a97da36342
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const ageDays = (Date.now() - snapshotEnd) / 86_400_000; | ||
| assert.ok( | ||
| ageDays <= MAX_SNAPSHOT_AGE_DAYS, |
There was a problem hiding this comment.
When the repository has no pushes or pull requests for several days, this assertion is never reevaluated: the checked .github/workflows/ci.yml only triggers CI for pull_request and pushes to main, with no scheduled run. The public figures can therefore become indefinitely stale without producing the promised visible failure; add a scheduled workflow that runs test:study (and reports failures) independently of repository activity.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b21a2dac35
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| const ageDays = (Date.now() - snapshotEnd) / 86_400_000; | ||
| assert.ok( | ||
| ageDays <= MAX_SNAPSHOT_AGE_DAYS, |
There was a problem hiding this comment.
If the scan host's clock is wrong or a snapshot is accidentally imported with a future filename, ageDays becomes negative and this upper-bound-only assertion passes until days after that future date, allowing the freshness guard and rendered pages to trust an invalid snapshot. Validate that the parsed snapshot date is not later than the current UTC date before accepting its age.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79aa9f6c6f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const file = newestSnapshotFile(); | ||
| const date = file.replace(/\.json$/, ""); | ||
| const snapshotEnd = Date.parse(`${date}T23:59:59Z`); |
There was a problem hiding this comment.
When a snapshot is manually imported under the wrong filename, this check treats the filename as its acquisition date and never inspects the payload's existing date or scanned_at fields. For example, renaming an older snapshot to today's date and re-rendering its figures makes both new tests pass while the public data is still stale. Validate that the filename matches the embedded date and calculate age from scanned_at so the guard verifies when the scan actually ran.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e35accfc3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| test("public pages are rendered from the newest study snapshot", () => { | ||
| execFileSync("python3", ["study/render.py", "--check"], { cwd: ROOT, stdio: "pipe" }); |
There was a problem hiding this comment.
The imported snapshot can pass this check even though it was produced from a different study set: at this commit study/corpus.json contains 20,005 sites and study/scan.py's default seven-day slice for 2026-08-12 contains 2,858 sites, while 2026-08-12.json contains only 99 sites (just 19 overlap that expected slice). Because the check only compares the pages with whichever snapshot was copied in, artifacts from the stale scan checkout are accepted as today's repository study and publish incompatible results. Validate the snapshot's hosts or corpus identity against the expected current-corpus slice before rendering it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
nish3451 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5fa87c0e27
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| test("public pages are rendered from the newest study snapshot", () => { | ||
| execFileSync("python3", ["study/render.py", "--check"], { cwd: ROOT, stdio: "pipe" }); |
There was a problem hiding this comment.
When a future snapshot changes its industries or geographies totals, this check can still pass while public/specimen.html:130 continues to claim “thirteen industries, eight geographies,” because those values are outside data-study spans even though render.py already exposes industries_word and geographies_word. Tag or otherwise validate these remaining counts so the new synchronization test actually detects drift across every listed public page.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Resolve package.json conflict: keep both the PR's test:study guard and main's test:viewport / test:narrow-pages / test:narrow additions in the aggregate test script.
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Rebase failed
1 participant
What
Tier-1 public-promise gap: the homepage, audit, pricing, and specimen pages promise "this number is today's" and "Method: ... refreshed daily", but the served figures (53 of 89) came from the 2026-08-06 snapshot — six days stale. The daily scan had produced newer snapshots (2026-08-09, 08-11, 08-12) that never reached this repo.
study/snapshots/2026-08-09.json,2026-08-11.json,2026-08-12.json(the daily series is the asset;render.py --checknow passes against the true newest scan).study/render.py: 55 of 90 readable (was 53 of 89), 57 with no price (unchanged), nine blocked by bot protection (was ten). Onlydata-studyspans touched — nothing else.scripts/test-study-freshness.mjswired intonpm test(and CI) fails when (a) the pages drift from the newest committed snapshot, or (b) the newest snapshot is older than 4 days. This is the detector for the exact failure that just happened — the drift was silent for six days because nothing checked.Root cause of the drift (for the record)
The daily scan (
fleet-tinystudio-market-scan.timer→tinystudio-market-scan.sh) runs in a different local checkout (TinyStudio.io-agent-self-serve) and its deploy step is skipped because the Cloudflare token env file at~/.config/cloudflare/envis missing — so figures rendered there never ship, and the snapshots are never imported into GitHub. Fixing the scan/deploy path touches deploy config and secrets (excluded scope); the guard above turns any future drift into a visible CI failure, and this PR lands today's figures on the code path that deploys from GitHub main.Verify
python3 study/render.py --check→pages match snapshot 2026-08-12npm run check→TinyStudio.io checks passed.npm test→ 94 tests, 0 failures (92 existing + 2 new freshness guards)git diff --check→ cleanSummary by CodeRabbit
New Features
Updates
Quality Improvements