Skip to content

improve bandwidth - #27

Merged
flvvius merged 3 commits into
mainfrom
fix/improve_bandwidth
May 9, 2026
Merged

improve bandwidth#27
flvvius merged 3 commits into
mainfrom
fix/improve_bandwidth

Conversation

@flvvius

@flvvius flvvius commented May 9, 2026

Copy link
Copy Markdown
Owner

Summary by CodeRabbit

  • New Features

    • Enhanced article enrichment pipeline with improved embedding version tracking and fact extraction handling for better accuracy.
    • Increased summary length allowance: center sections now support up to 120 words, while left/right and global impact sections support up to 100 words each.
  • Performance Improvements

    • Optimized AI budget tracking with pre-computed daily aggregations for faster calculations.

Review Change Stack

@vercel

vercel Bot commented May 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
news Error Error May 9, 2026 0:12am
news-web Ready Ready Preview, Comment May 9, 2026 0:12am

@coderabbitai

coderabbitai Bot commented May 9, 2026

Copy link
Copy Markdown
Contributor

Warning

Rate limit exceeded

@flvvius has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 38 minutes and 35 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 84c6d467-b71a-463e-b8bf-2b5335ef5306

📥 Commits

Reviewing files that changed from the base of the PR and between 7540a8f and bb3fc47.

📒 Files selected for processing (4)
  • packages/backend/convex/aiBudget.ts
  • packages/backend/convex/enrichment.ts
  • packages/backend/convex/migrations.ts
  • packages/backend/convex/prompts.ts

Walkthrough

This PR refactors article enrichment and AI budget accounting while centralizing source bias labeling. It introduces new schema fields for tracking enrichment state (needsFactExtraction, latestEmbeddingVersion, needsReenrichment), creates a daily AI budget aggregation table, extracts sourceBiasLabel into a shared utility, and refactors candidate claiming/deferring logic to use derived maintenance fields. Summary word limits are increased, and formatting/line-wrapping changes are applied throughout.

Changes

Article Enrichment, Budget Aggregation, and Source Bias Refactoring

Layer / File(s) Summary
Schema & Data Contracts
packages/backend/convex/schema.ts
Articles table gains optional fields: needsFactExtraction, latestEmbeddingVersion, needsReenrichment, plus indices for status, reenrichment, and fact-extraction queries. New aiBudgetDailyTotal table stores daily aggregated spentUsd, reservedUsd, indexed by date.
Shared Source Bias Utility
packages/backend/convex/lib/sourceBias.ts
New module exports SourceBiasLabel type and sourceBiasLabel(source) function mapping source bias/MBFC category to predefined labels, replacing duplicated logic across multiple files.
AI Budget Accounting
packages/backend/convex/aiBudget.ts
Daily budget parsing tightened to require finite values ≥ 0.1. getDailyBudgetTotals now reads precomputed aiBudgetDailyTotal aggregates when available. New ensureDailyBudgetTotal lazily creates daily aggregate rows. adjustBudgetShard maintains consistency between shard and aggregate rows.
Enrichment Maintenance Helpers
packages/backend/convex/enrichment.ts
New helpers: hasAtomicFacts, needsFactExtraction determine fact-extraction eligibility; deriveArticleMaintenanceFields computes derived booleans after field overrides; getLatestEmbeddingVersion fetches newest embedding version.
Article Candidate Claiming & Deferring
packages/backend/convex/enrichment.ts
claimArticlesForReenrichment and claimArticlesNeedingFactExtraction refactored to use Map-based deduplication across multiple indices and compute latestVersion per candidate. Defer and mark functions patch derived maintenance fields consistently. markArticleEnriched embedding persistence changed to single-row "hot steady state".
Source Bias Label Integration
packages/backend/convex/sources.ts, packages/backend/convex/summarization.ts, packages/backend/convex/claimDivergence.ts
Import and use shared sourceBiasLabel function; removed local duplicate implementations.
Summary Generation Word Limits
packages/backend/convex/summarizationNode.ts
Word caps increased: center 60–120, left/right 25–100, globalImpact 25–100. Schema descriptions updated to match.
Code Formatting & Line Wrapping
packages/backend/convex/clustering.ts, packages/backend/convex/summarization.ts
Reformatted imports, wrapped long expressions, and reflowed multi-line logic without changing control flow or numeric thresholds.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

  • flvvius/news#21: Both PRs modify the enrichment pipeline and refactor candidate claiming/deferring logic in claimArticlesNeedingFactExtraction, deferArticleFactExtraction, and embedding persistence.
  • flvvius/news#18: Main PR's changes to aiBudget.ts (daily-limit parsing, aiBudgetDailyTotal aggregate table, ensureDailyBudgetTotal) directly overlap with that PR's budget accounting refactoring.
  • flvvius/news#14: Both PRs modify clustering.ts event presentation refresh and latestArticlePublishedAt calculations; the retrieved PR queues rendering using that same field.
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title 'improve bandwidth' is too vague and generic. It does not clearly convey what specific changes were made, such as database optimization, caching aggregates, fact extraction refactoring, or budget parsing improvements. Consider using a more descriptive title that reflects the main changes, such as 'Optimize article enrichment pipeline and AI budget aggregation' or similar, to clarify the key improvements for code history.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/improve_bandwidth

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/backend/convex/enrichment.ts (1)

642-670: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Clean up older articleEmbeddings rows before writing the new embedding.

This no longer converges existing multi-row articles to the “single hot row” state described in the comment. Any older rows stay in the vector index, so one article can keep returning duplicate or stale matches.

Proposed fix
-    const latestEmbeddingRow = await ctx.db
+    const embeddingRows = await ctx.db
       .query("articleEmbeddings")
       .withIndex("by_article_version", (q) => q.eq("articleId", articleId))
       .order("desc")
-      .first();
+      .collect();
+    const [latestEmbeddingRow, ...staleEmbeddingRows] = embeddingRows;
+
+    for (const row of staleEmbeddingRows) {
+      await ctx.db.delete(row._id);
+    }

     // Keep a single hot embedding row per article in steady state.
     if (latestEmbeddingRow) {
       await ctx.db.patch(latestEmbeddingRow._id, {
         embedding,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/backend/convex/enrichment.ts` around lines 642 - 670, The code
updates or inserts a single "hot" article embedding but does not remove older
articleEmbeddings rows, leaving stale duplicates in the vector index; modify the
logic around latestEmbeddingRow/articleEmbeddings to remove all other rows for
the same articleId after you patch or insert: query all rows via
ctx.db.query("articleEmbeddings").withIndex("by_article_version", ...) (or reuse
the existing result set), keep the one you just patched/inserted
(latestEmbeddingRow._id or the newly returned id), and call ctx.db.delete for
any other rows where _id !== keptId so only a single hot row remains; ensure
this cleanup runs both in the branch that calls ctx.db.patch and in the branch
that calls ctx.db.insert.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/backend/convex/aiBudget.ts`:
- Around line 214-220: The daily aggregate update is using the original
requested deltas (deltaSpent/deltaReserved) instead of the actual clamped shard
changes, allowing aiBudgetDailyTotal to drift; change the code so that after
computing the clamped shard adjustments you compute the actual applied deltas
(e.g., appliedSpent = newShardSpent - oldShardSpent, appliedReserved =
newShardReserved - oldShardReserved) and use those appliedSpent/appliedReserved
when computing nextTotalSpent/nextTotalReserved and when updating
aiBudgetDailyTotal (also apply the same fix in the second identical block around
the aiBudgetDailyTotal updates at the later section referenced). Ensure you
reference ensureDailyBudgetTotal, aiBudgetDailyTotal, and the shard
variables/roundUsd calculations when making the change.

---

Outside diff comments:
In `@packages/backend/convex/enrichment.ts`:
- Around line 642-670: The code updates or inserts a single "hot" article
embedding but does not remove older articleEmbeddings rows, leaving stale
duplicates in the vector index; modify the logic around
latestEmbeddingRow/articleEmbeddings to remove all other rows for the same
articleId after you patch or insert: query all rows via
ctx.db.query("articleEmbeddings").withIndex("by_article_version", ...) (or reuse
the existing result set), keep the one you just patched/inserted
(latestEmbeddingRow._id or the newly returned id), and call ctx.db.delete for
any other rows where _id !== keptId so only a single hot row remains; ensure
this cleanup runs both in the branch that calls ctx.db.patch and in the branch
that calls ctx.db.insert.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f9f5abd1-79f6-446a-bc8b-c7c2bac975d3

📥 Commits

Reviewing files that changed from the base of the PR and between bab5e11 and 7540a8f.

⛔ Files ignored due to path filters (1)
  • packages/backend/convex/_generated/api.d.ts is excluded by !**/_generated/**, !**/_generated/**
📒 Files selected for processing (9)
  • packages/backend/convex/aiBudget.ts
  • packages/backend/convex/claimDivergence.ts
  • packages/backend/convex/clustering.ts
  • packages/backend/convex/enrichment.ts
  • packages/backend/convex/lib/sourceBias.ts
  • packages/backend/convex/schema.ts
  • packages/backend/convex/sources.ts
  • packages/backend/convex/summarization.ts
  • packages/backend/convex/summarizationNode.ts

Comment thread packages/backend/convex/aiBudget.ts
@flvvius
flvvius merged commit b873470 into main May 9, 2026
3 of 4 checks passed
@coderabbitai coderabbitai Bot mentioned this pull request May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant