fix: restore event publishing (grounding evidence + queue starvation) - #65
Merged
Conversation
…gate) Publishing stopped completely on 2026-08-02: 0 events created since then have ever reached `published`, while 4,273 piled up in `processing` and the summary queue grew past 800 jobs. Publishing is gated solely on a successful AI summary, so every pre-publication check is a publish veto. Root cause was a config change, not code. On 2026-08-01 13:41 event_summary_body_fetch_enabled was set to false. With article_fact_extraction_enabled also false, the L4 grounding corpus collapsed to summary + rssSnippet only — measured at ~109 + ~105 chars per article across 200 live articles, so ~1.3 KB total for a 6-article event — while the prompt still asks for ~420 words / 25-30 sentences. Most sentences could not be grounded, so summaries were blocked_ungrounded (423 of the last 800 audit rows). The same-evening bump of event_grounding_max_unsupported_ratio 0.34 -> 0.5 could not compensate. Thin input also drove the verbatim gate: with only a short snippet available, a faithful sentence reproduces it. Restore evidence via stored facts rather than live fetches: - article_fact_extraction_enabled defaults to true. Enrichment already holds the full article body, so facts are extracted once and stored, and summarization grounds against them without re-fetching bodies on every regeneration. This keeps body fetch off, so no action wall-clock is reintroduced. Remove the NER risk gate entirely: - Its lexicon matched by prefix (no trailing word boundary), so ordinary coverage tripped it — production holds include "viol" matching inside "incendii violente" — and findNamedEntities treated any two adjacent capitalized words as a named entity, bypassing ENTITY_STOPWORDS. One flagged sentence held the whole event, and a pending review row skipped that event from re-enqueue forever, so holds were permanent. - A pending summaryReviewQueue row no longer blocks re-enqueue, so the events held before this change can publish instead of staying stranded. - summaryReviewQueue, holdSummaryForReview and /admin/review are retained so rows held before the removal remain readable. L3 verbatim overlap and L4 grounding remain as pre-publication checks. Backend suite green (273 passed); convex tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012MpkuigW6T4N1exXip6bRp
…neration Follow-up to bbf4205. Body fetch stays off and the grounding budget stays tight — both were deliberate Convex-usage patches — so publishing has to work within thin evidence rather than restore expensive evidence. Grounding: strip, don't block. The gate already stripped unsupported sentences, but only after a ratio check that discarded the whole summary when more than event_grounding_max_unsupported_ ratio of sentences failed. That threw away the supported sentences too and left the event in `processing`. Removed the ratio block. An unsupported sentence still never reaches publication — it is stripped — and the floor stays: if nothing grounded remains in `neutral`, the summary is still blocked. Net effect is strictly more published events and strictly fewer model calls, with the same guarantee about what gets published. event_grounding_max_unsupported_ratio is now unused. Terminal blocks: stop regenerating identical inputs. A blocked summary left no stored summary, so shouldResummarize stayed true and the next cron regenerated the same summary — a model call plus a grounding call, hourly, forever (423 blocked runs across 230 events, up to 8 per event). Both terminal paths now stamp the event's lastSummarySignature, so the existing no_change_since_last_run short-circuit fires before any model call. The signature is derived from the event's articles, so a new article changes it and the event retries on its own. event_summary_enqueue_limit 40 -> 12 to match event_summary_batch_size. A run drains batch_size jobs, so a higher enqueue limit grew the queue every hour (820 of 1000 recent jobs were still queued) — wasted writes that also pushed freshly-qualified events behind days of backlog. Candidates are scanned most-recent-first, so this costs recall of stale events, not fresh ones. Backend suite green (276 passed); convex tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012MpkuigW6T4N1exXip6bRp
listDueSummaryJobs drained strictly oldest-first. A run drains batch_size jobs, so with a backlog that ordering pins fresh events behind old ones indefinitely: production currently holds 824 queued jobs reaching back to 2026-08-03, so an event queued today would wait roughly three days for its first attempt. The feed would keep showing stale news even though publishing had been repaired. Pure newest-first has the opposite failure (the backlog never retires), so the batch is now split: ~70% newest-first so freshly-qualified events are attempted within about one run, the remainder oldest-first so the backlog still drains every run. Selection de-duplicates, since both reads hit the same index range. Tests cover both directions: a job queued today is selected despite 40 older ones, and the oldest job is still selected when 40 newer ones exist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012MpkuigW6T4N1exXip6bRp
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restores event publishing, which has been completely stopped in production since 2026-08-02 (22 days). No new events have reached the feed in that time.
Root cause
#63disabledevent_summary_body_fetch_enabled, andarticle_fact_extraction_enabledwas also off. Those are the only two sources of grounding evidence. With both disabled the grounding corpus collapsed tosummary + rssSnippet(~200 chars/article), so every summary was rejected asblocked_ungrounded. Because summaries gate publishing, nothing could publish.A second, compounding problem:
event_summary_enqueue_limit(40) exceededevent_summary_batch_size(12). Each run enqueued up to 40 jobs but drained only 12, so the queue grew every run and freshly-qualified events were buried behind days of backlog. Production showed 1000+ queued and 1000+ failed jobs with only 1 processing.Changes
Three commits, originally authored 2026-08-15 but never pushed, rebased onto current
main:restore event publishing— use stored atomic facts as grounding evidence; remove the NER risk gate.publish grounded summaries instead of discarding them— stop discarding summaries that passed grounding; stop needless regeneration.stop the summary queue starving freshly-qualified events— keepenqueue_limitat or belowbatch_sizeso the queue drains instead of growing.These were written against pre-#63 code, so they were rebased, not merged — #63's cost-reduction work is fully preserved (
rateLimitError.ts,feedPagination.test.ts,applyCostReductionConfigand the reduced cron cadence all remain). One conflict inconfig.tswas resolved by merging both rationales into a single description rather than dropping #63's I/O context.Production state
Already applied ahead of this PR (reversible config change):
event_summary_enqueue_limit: 40 → 12, to stop queue growth immediately.article_fact_extraction_enabledwas alreadytruein prod.Testing
mainis exactly the 7 files of the fixes.🤖 Generated with Claude Code
Generated by Claude Code