fix(embeddings): stamp legacy embedding_model + stop liveness probe killing the embeddings service

[buremba]

Two fixes for the embeddings-backfill incident found during the #1066-#1070 prod rollout verification.

1. Migration: stamp legacy NULL embedding_model rows (recall regression)

20260526120000 (#1069) added event_embeddings.embedding_model but left existing rows NULL. The model-scoped vector search excludes NULL stamps, so on deploy the entire legacy corpus (prod: 1,231,945 rows) dropped out of vector search — full semantic-recall regression (text search unaffected). Hotfixed in prod with the same UPDATE; this migration makes it reproducible (fresh env / PITR restore). The stamp is accurate (legacy rows were all the default model) — no re-embedding. Idempotent; no-op once stamped.

2. Chart: tolerant liveness probe + CPU for the embeddings service (backfill 0/100)

Root-caused the failing backfill (0/100, restart loop): the embeddings service runs the model inline on a single-threaded event loop, so a heavy batch blocks /health. It shared the stateless app's probe (timeout 3s x3), so k8s killed a busy-but-healthy service mid-request (exit 137, "failed liveness probe", /health context deadline exceeded) → in-flight embedding requests died with fetch failed → restart loop. Not OOM, not data, not a code bug — an infra misconfig.

Fix (no code workaround): the embeddings service gets its own tolerant probe (liveness must detect a dead process, not a busy one: timeout 15s x8) and cpu 500m→1000m so batches finish faster. Chart.yaml 9.4.1→9.4.2 so Flux ChartVersion reconcile ships it (release-please doesn't manage the chart).

Validation

• Migration: idempotent data UPDATE; CI migrations job validates apply. No committed schema.sql.
• Chart: helm lint clean; helm template renders the new probe (liveness timeout 15 / failureThreshold 8) + cpu 1000m.

Notes

• Prod recall is already restored (manual stamp). This PR makes that reproducible AND permanently fixes the embeddings restart loop.
• Until the chart ships, the backfill still intermittently restarts the embeddings pod (~0.3% of events lack embeddings); merging + shipping the chart stops the churn.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced embeddings service stability with refined health check configuration, improving reliability during high-load operations
  • Increased CPU resource allocation for embeddings service to optimize performance during batch processing
  • Applied database migration to ensure consistent embedding model version tracking

• Chores

  • Updated Helm chart version to 9.4.2

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f3aebee7-88c0-42b3-b25f-d52ecd94c7d7

📥 Commits

Reviewing files that changed from the base of the PR and between 6eb19f4 and 870d620.

📒 Files selected for processing (4)

• charts/lobu/Chart.yaml
• charts/lobu/templates/embeddings-deployment.yaml
• charts/lobu/values.yaml
• db/migrations/20260526170000_backfill_legacy_embedding_model_stamp.sql

---

📝 Walkthrough

Walkthrough

This PR improves embeddings service reliability through coordinated database and Kubernetes configuration changes. A SQL migration backfills missing embedding model identifiers, while Helm chart updates increase CPU allocation and introduce embeddings-specific health check timeouts to prevent premature service termination during heavy batch processing.

Changes

Embeddings Service Reliability

Layer / File(s)	Summary
Database migration for embedding model backfill db/migrations/20260526170000_backfill_legacy_embedding_model_stamp.sql	Migration sets embedding_model to 'Xenova/bge-base-en-v1.5' for all event_embeddings rows where the column is NULL; down migration is a no-op to avoid reintroducing incorrect behavior.
Embeddings service Helm configuration and deployment charts/lobu/values.yaml, charts/lobu/templates/embeddings-deployment.yaml, charts/lobu/Chart.yaml	Helm values.yaml increases embeddings CPU request to 1000m and adds embeddings-scoped healthCheck with more tolerant readinessProbe/livenessProbe timings; deployment template references embeddings-specific health check values; chart version bumped to 9.4.2.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 A migration to stamp what was lost,
More CPU to power embeddings through cost,
Health checks now patient, no false alarms—
The service breathes easy in embedding farms! 🌾

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/fix-embedding-backfill-followup

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!