Fix/convex pipeline health - #41
Merged
Merged
Conversation
…es after clustering Events no longer go public on article/source counts alone. Publishing is now gated on a successful AI summary: clustering keeps qualifying events in `processing`, and applyEventSummaryResult flips status -> `published` once an event has neutral/reformist/suveranist perspectives + globalImpact (and syncs its public preview). Effective publish bar = the summary bar (3 articles / 2 sources), matching the two thresholds. Summaries are no longer trickled by the 45-min cron alone: clustering, merge, and recluster batches now trigger summarizeQueuedEvents immediately, so an event that just crossed the bar is summarized and published right away. Enqueue selection scans qualifying `processing` events (primary) plus `published` (re-summarize/backfill); shouldResummarize drops its published-only gate. event_summary_batch_size raised 3 -> 8 to keep pace. Verified with a new convex-test suite (promotion, enqueue eligibility, singleton skip, no-op re-enqueue) + full backend suite (166 pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rs Agerpres) Agerpres has no native RSS; it routes through the bazqux createfeed extractor, which scrapes the Agerpres widget live and routinely takes ~7-20s. The flat 15s AbortSignal.timeout aborted it most runs (AbortError), quarantining the national wire service and starving cross-source clustering. Add an optional per-feed fetchTimeoutMs (default 15s), set Agerpres to 30s, and thread it through ingestSingleFeed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One-time cleanup for the publish-on-summary switchover. Reverts `published` events that can never earn an AI summary (below the summary min articles/sources and lacking one) back to `processing`, deleting their public preview, so the new "public only with an AI summary" invariant holds. Qualifying published-but- summary-less events are left for the summary backfill to fill in place. Dry-run by default; idempotent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
Next review available in: 26 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (8)
WalkthroughPublish status is decoupled from clustering thresholds and gated instead on successful AI summarization, with immediate summarization scheduling after clustering/merging, a config default update, a data migration to revert misclassified published events, per-feed fetch timeout support, and a PostHog UI host override. ChangesPublish-on-summary gating and clustering changes
Per-feed fetch timeout override
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant Clustering as clusterEnrichedArticles
participant Scheduler
participant SummarizeJob as summarizationNode.summarizeQueuedEvents
participant Enqueue as enqueueEligibleEventSummaries
participant Apply as applyEventSummaryResult
participant Events as events table
Clustering->>Events: attach articles, preserve event status
alt articles clustered/attached
Clustering->>Scheduler: runAfter(0, summarizeQueuedEvents)
Scheduler->>SummarizeJob: invoke immediately
end
SummarizeJob->>Enqueue: scan processing + published events
Enqueue->>Events: enqueue eligible summary jobs
Events-->>Apply: job returns AI summary result
Apply->>Events: patch summary fields
alt event.status == processing
Apply->>Events: set status = published
end
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/backend/convex/ingestion.ts (1)
1622-1635: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winRetry path drops
fetchTimeoutMsoverride.The retry call to
ingestSingleFeedat Line 1624 omitsfetchTimeoutMs: feed.fetchTimeoutMs, so retried feeds fall back to the 15s default. For the Agerpres feed (30s override), a retry after a timeout will likely time out again, defeating the per-feed override.🐛 Proposed fix
const retryResult = await ctx.runAction( internal.ingestion.ingestSingleFeed, { feedUrl: feed.url, feedName: feed.name, feedDomain: feed.domain, baseBias: feed.baseBias, reliabilityScore: feed.reliabilityScore, mbfcCategory: feed.mbfc.category, mbfcFactual: feed.mbfc.factual, mbfcCredibility: feed.mbfc.credibility, + fetchTimeoutMs: feed.fetchTimeoutMs, }, );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/backend/convex/ingestion.ts` around lines 1622 - 1635, The retry path in ingestion should preserve the per-feed timeout override when calling ingestSingleFeed. Update the ctx.runAction payload inside the feedsToRetry loop to include feed.fetchTimeoutMs alongside the existing feedUrl/feedName/feedDomain and MBFC fields, so retries honor the same timeout configuration as the original ingest path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/web/src/lib/posthog.tsx`:
- Around line 19-23: The PostHog config in posthog.tsx currently mixes regions
by defaulting api_host to the US endpoint while hardcoding ui_host to the EU
endpoint, so update the PostHogProvider setup to keep ui_host aligned with the
actual project region. Make ui_host configurable via an env var in the same
style as POSTHOG_HOST, and ensure the fallback matches the same region used by
api_host so toolbar and session-recording links resolve to the correct project.
In `@packages/backend/convex/summarizationPublishGate.test.ts`:
- Around line 136-166: Add a complementary test in
summarizationPublishGate.test.ts for enqueueEligibleEventSummaries that covers a
published event without a current AI summary, since the existing test only
verifies the skip path for a published event that already has one. Reuse the
existing test setup around convexTest,
internal.summarization.enqueueEligibleEventSummaries, and the events insert
shape, but omit or change the current AI summary fields so the event should
qualify and be queued; then assert the queued count increases as expected.
---
Outside diff comments:
In `@packages/backend/convex/ingestion.ts`:
- Around line 1622-1635: The retry path in ingestion should preserve the
per-feed timeout override when calling ingestSingleFeed. Update the
ctx.runAction payload inside the feedsToRetry loop to include
feed.fetchTimeoutMs alongside the existing feedUrl/feedName/feedDomain and MBFC
fields, so retries honor the same timeout configuration as the original ingest
path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ba284e93-99c0-490f-a520-7718f980ba94
📒 Files selected for processing (8)
apps/web/src/lib/posthog.tsxpackages/backend/convex/clustering.tspackages/backend/convex/config.tspackages/backend/convex/feeds.tspackages/backend/convex/ingestion.tspackages/backend/convex/migrations.tspackages/backend/convex/summarization.tspackages/backend/convex/summarizationPublishGate.test.ts
Comment on lines
+136
to
+166
| test("a published event with a current AI summary is not re-enqueued", async () => { | ||
| const t = convexTest(schema, modules); | ||
| const now = Date.now(); | ||
| await t.run(async (ctx) => { | ||
| await ctx.db.insert("events", { | ||
| title: "Already Summarized", | ||
| slug: "already-summarized", | ||
| status: "published", | ||
| firstPublishedAt: now, | ||
| lastUpdatedAt: now, | ||
| lastArticleAt: now, | ||
| articleCount: 3, | ||
| sourceCount: 2, | ||
| sourceIds: [], | ||
| perspectiveSummaries: { | ||
| neutral: "n", | ||
| reformist: "r", | ||
| suveranist: "s", | ||
| }, | ||
| perspectiveSource: "ai", | ||
| globalImpact: "g", | ||
| lastSummarizedAt: now, | ||
| }); | ||
| }); | ||
|
|
||
| const result = await t.mutation( | ||
| internal.summarization.enqueueEligibleEventSummaries, | ||
| { limit: 10, minArticles: 3, minSources: 2 }, | ||
| ); | ||
| expect(result.queued).toBe(0); | ||
| }); |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Consider adding a test for published events without a current AI summary.
The PR summary states enqueueEligibleEventSummaries now scans both processing and published events. Test 4 only covers a published event that already has a summary (and is correctly skipped). A complementary test for a published event without a current AI summary — which should be enqueued for summarization — would verify the new behavior that enables the migration's "backfill in place" strategy.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/backend/convex/summarizationPublishGate.test.ts` around lines 136 -
166, Add a complementary test in summarizationPublishGate.test.ts for
enqueueEligibleEventSummaries that covers a published event without a current AI
summary, since the existing test only verifies the skip path for a published
event that already has one. Reuse the existing test setup around convexTest,
internal.summarization.enqueueEligibleEventSummaries, and the events insert
shape, but omit or change the current AI summary fields so the event should
qualify and be queued; then assert the queued count increases as expected.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
- posthog.tsx: repair botched ui_host splice (stray duplicate posthog.init +
const wedged inside the object literal); hoist POSTHOG_UI_HOST to module scope.
- bookmark-button.tsx: type useMutation via FunctionReturnType/FunctionArgs so
onSuccess data is known (was inferred as {}).
- source.$sourceId.tsx: pin `data` to the query return type so mapped `article`
isn't implicitly any (loaderData is untyped through the router context).
- __root.tsx: @ts-expect-error the root beforeLoad return-context (TanStack
Router v1.132 types it as `never`; returning context is the correct runtime
contract read via useRouteContext).
- Remove dead code / unused symbols: admin.pipeline pct(), politica Link import,
aiBudget startOfUtcDay(), auth sendAuthEmail's unused actionUrl binding.
Web app + backend both typecheck clean; backend suite 166 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Bug Fixes