Repository navigation
fix: preserve semantic image evidence tables #303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 9 commits
0269b9e
3cd2d6e
e055463
f5b2345
2702fd6
fe0a4f2
e5d0221
b7e6e82
497d9e3
817681b
a9b4be2
7e4cc10
ba71bc1
d8e8ede
b9bba3b
93c8314
fc61f60
a4a6008
1c861f4
bdf3716
7727cb1
497ac12
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
|
devin-ai-integration[bot] marked this conversation as resolved.
Resolved
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -76,12 +76,24 @@ async def persist_post_content( | |
|
|
||
| Provider calls happen before the short database transaction. A failed or | ||
| unavailable embedding call writes no vector row; it never writes a zero or | ||
| guessed vector. The raw body remains in ``source_post`` for future retry. | ||
| guessed vector. A same-image retry cannot replace non-empty persisted OCR | ||
| with an empty result. The raw body remains in ``source_post`` for future | ||
| retry. | ||
| """ | ||
| normalized = normalized_result or normalize_post_body(body, vision_client) | ||
| chunks = chunk_by_source_body(body) | ||
| image_results = {result.chunk_index: result for result in normalized.image_results} | ||
| formatting = {hint.chunk_index: hint.style for hint in normalized.formatting_hints} | ||
| image_ocr_by_sha256: dict[str, bool] = {} | ||
| for chunk in chunks: | ||
| if chunk.unit_type != "image" or chunk.image_data is None: | ||
| continue | ||
| result = image_results.get(chunk.index) | ||
| description = result.description if result else None | ||
| content_sha256 = hashlib.sha256(chunk.image_data).hexdigest() | ||
| image_ocr_by_sha256[content_sha256] = image_ocr_by_sha256.get( | ||
| content_sha256, False | ||
| ) or bool(description and description.extracted_text.strip()) | ||
|
Comment on lines
+99
to
+108
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: OCR-preservation guard protects whole-image OCR only, not region OCR The guard computes Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+99
to
+108
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: OCR-preservation guard keying and OR-aggregation verified correct The guard builds Was this helpful? React with 👍 or 👎 to provide feedback. |
||
|
|
||
| prepared: list[tuple[Chunk, str, str | None]] = [] | ||
| for chunk in chunks: | ||
|
|
@@ -196,6 +208,29 @@ async def persist_post_content( | |
| continue | ||
|
|
||
| async with conn.transaction(): | ||
| if image_ocr_by_sha256: | ||
| await conn.fetchval( | ||
| "select post_id from source_post where post_id = $1 for update", | ||
| post_id, | ||
| ) | ||
|
Comment on lines
+245
to
+249
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: OCR guard only takes the source_post FOR UPDATE lock when the post contains images The Was this helpful? React with 👍 or 👎 to provide feedback. |
||
| previous_images = await conn.fetch( | ||
| """ | ||
| select image.content_sha256, image.extracted_text | ||
| from post_content_unit unit | ||
| join post_content_image image using (post_content_unit_id) | ||
| where unit.post_id = $1 | ||
| and nullif(btrim(image.extracted_text), '') is not null | ||
| """, | ||
| post_id, | ||
| ) | ||
| if any( | ||
| row["content_sha256"] in image_ocr_by_sha256 | ||
| and not image_ocr_by_sha256[row["content_sha256"]] | ||
| for row in previous_images | ||
| ): | ||
| raise RuntimeError( | ||
| "refusing to replace non-empty image OCR with an empty retry result" | ||
| ) | ||
|
Comment on lines
+245
to
+267
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: OCR guard adds a lock + fetch on every image-bearing persist The guard at post_content_persistence.py runs Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+245
to
+267
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔍 Normalize-only backfill silently skips image posts with existing OCR The preservation guard raises when a same-hash image previously had OCR but the current run yields none. Under Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+260
to
+267
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: OCR preservation can permanently block reprocessing of an image that legitimately lost its text The new guard in post_content_persistence.py refuses to persist when a previously-stored image (matched by Was this helpful? React with 👍 or 👎 to provide feedback. |
||
| await conn.execute("delete from post_content_unit where post_id = $1", post_id) | ||
| unit_ids: dict[int, str] = {} | ||
| for chunk, unit_text, style in prepared: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -54,6 +54,15 @@ def _parser() -> argparse.ArgumentParser: | |
| return parser | ||
|
|
||
|
|
||
| async def _ensure_open_connection( | ||
| conn: asyncpg.Connection, target_dsn: str | ||
| ) -> asyncpg.Connection: | ||
| """Reconnect after a database restart without repeating VISION work.""" | ||
| if not conn.is_closed(): | ||
| return conn | ||
| return await asyncpg.connect(target_dsn) | ||
|
|
||
|
|
||
| async def backfill_post_content( | ||
| target_dsn: str, | ||
| raw_post_ids: list[str] | None, | ||
|
|
@@ -216,6 +225,7 @@ async def backfill_post_content( | |
| if described_images == 0 and not normalized.text.strip(): | ||
| result["skipped_posts"] += 1 | ||
| continue | ||
| conn = await _ensure_open_connection(conn, target_dsn) | ||
|
devin-ai-integration[bot] marked this conversation as resolved.
Resolved
Comment on lines
229
to
+232
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: Reconnect guard relies on asyncpg is_closed, which only reflects locally-detected closure
Was this helpful? React with 👍 or 👎 to provide feedback. |
||
| await persist_post_content( | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📝 Info: Backfill reconnect only covers the VISION gap, not a drop during persist's embedding work
Was this helpful? React with 👍 or 👎 to provide feedback. |
||
| conn, | ||
| str(row["post_id"]), | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| """Operator backfill connection recovery contracts.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
|
|
||
| from scripts import backfill_post_content | ||
|
|
||
|
|
||
| def test_reconnects_only_after_database_connection_closes(monkeypatch) -> None: | ||
| replacement_connection = object() | ||
| connected_dsns: list[str] = [] | ||
|
|
||
| class Connection: | ||
| def __init__(self, closed: bool) -> None: | ||
| self._closed = closed | ||
|
|
||
| def is_closed(self) -> bool: | ||
| return self._closed | ||
|
|
||
| async def connect(dsn: str): | ||
| connected_dsns.append(dsn) | ||
| return replacement_connection | ||
|
|
||
| monkeypatch.setattr(backfill_post_content.asyncpg, "connect", connect) | ||
|
|
||
| current_connection = Connection(False) | ||
| assert ( | ||
| asyncio.run(backfill_post_content._ensure_open_connection(current_connection, "dsn")) | ||
| is current_connection | ||
| ) | ||
| assert asyncio.run(backfill_post_content._ensure_open_connection(Connection(True), "dsn")) is replacement_connection | ||
| assert connected_dsns == ["dsn"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📝 Info: _pop_first_env strips exactly one matched outer quote pair
start.py strips a single pair of matching outer quotes so Docker env-file quoting (e.g.
KEY='value') no longer corrupts the credential or the/v1URL suffix check. It only removes one layer; a doubly-quoted value like''value''would retain inner quotes, and an intentionally empty quoted value ("") collapses to empty and falls through to the next alias. Real credentials are unlikely to legitimately start and end with the same quote char, so the practical risk is low.Was this helpful? React with 👍 or 👎 to provide feedback.