fix: handle FK violation in observation_history during parallel consolidation

[handnewb]

Summary

Under parallel/batched consolidation, _append_observation_history can fail with:

insert or update on table "observation_history" violates foreign key constraint "observation_history_observation_id_fkey"

One batch deletes/replaces an observation row while another writes its history — a race condition. Instead of failing the entire consolidation task, catch the ForeignKeyViolationError and skip the history entry.

Change

• Add import asyncpg to consolidation module
• Wrap the INSERT in _append_observation_history with try/except asyncpg.exceptions.ForeignKeyViolationError
• Log a warning when the FK violation occurs (observation was removed before history)

Impact

• Consolidation tasks no longer fail due to this race condition
• Observation history entries are best-effort (safe to skip — the current observation state is the source of truth)
• No schema changes required

Reproduction

Triggered by parallel consolidation (consolidation_llm_parallelism > 1). Sequential consolidation (consolidation_llm_parallelism = 1) never hits this, confirming the race condition.

Closes #2597
Closes #2506

[yingliang-zhang]

Thanks for this fix — it matches exactly what I was about to PR myself.

I hit the same ForeignKeyViolationError on observation_history_observation_id_fkey and traced the root cause. One additional finding worth noting:

This race is not limited to parallel consolidation (consolidation_llm_parallelism > 1). It can also occur within a single batch in sequential mode:

Looking at _process_memory_batch (around L1566-1625 in consolidator.py), deletes execute first (L1588-1598), then updates (L1600-1625). If the LLM returns both a delete and an update action for the same observation_id within one batch:

1. The delete succeeds — observation row is removed from memory_units
2. The update proceeds to _execute_update_action → _append_observation_history (L1895)
3. The INSERT INTO observation_history fails because the observation no longer exists in memory_units

The security check at L1592 (observation must be present in the unioned recall) only guards the delete path, and the check at L1605 (observation must be in per_fact_obs_ids) only guards the update path — neither cross-references the other action list within the same batch.

So the try/except ForeignKeyViolationError approach in this PR is the correct and sufficient fix — it handles both the parallel race and the same-batch delete-then-update sequence. The observation_history entry is best-effort by nature (the current observation state is the source of truth), so skipping it on FK violation is safe.

Reproduced on v0.8.4 with sequential consolidation (consolidation_llm_parallelism=1). After applying this patch locally, the 2 failed consolidation tasks stopped recurring.