feat(embeddings): batch-tolerant import embeddings via pg-llm-batch submodule - #969
Closed
seonghobae wants to merge 1 commit into
Closed
feat(embeddings): batch-tolerant import embeddings via pg-llm-batch submodule#969seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
…ubmodule Integrate the org's standalone pg_tiktoken Postgres batch engine as naruon's first git submodule (vendor/pg-llm-batch, pinned SHA) and wire the batch-tolerant email-import embedding hotspot to it, with graceful degradation to the existing per-item path when the submodule is uninitialized or batching is unconfigured. - Add .gitmodules + vendor/pg-llm-batch submodule pinned at 9f40192 (Apache-2.0 extracted core; permissive deps only). naruon runs unchanged without it. - New services/batch_embedding_service.py: resolves per-tenant batch config from the Fernet DB (tenant_configs, never os.getenv), loads the submodule behind an import guard, uses the engine's TokenCounter/BatchAccumulator to token/byte/ record-partition texts, records LlmBatchJob/LlmBatchItem, and returns fitted vectors — or None to fall back. - Wire _generate_import_embeddings via an EmailImportBatchContext threaded from import_email_uploads; interactive draft/translate paths are untouched. - ORM models LlmBatchJob (llm_batch_jobs) + LlmBatchItem (llm_batch_items) and per-tenant batch config columns on tenant_configs (batch_embedding_dsn is a Fernet EncryptedString), modeled on provider_writeback_retry_items. Migration 0010_llm_batch_embedding. - Provider creds continue to resolve via resolve_runtime_llm_provider. - Opt-in docker-compose.pg-llm-batch.yml (pg_tiktoken image) + README note. - Fast mocked tests: batch routing, graceful fallback (disabled / submodule missing), and config-from-Fernet-DB (not env). Human action to enable: init the submodule + install it, bring up the batch Postgres, and seed batch config in the Fernet DB (tenant_configs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
Contributor
Author
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+148
to
+149
| except Exception: # pragma: no cover - defensive | ||
| pass |
Contributor
|
PR governance metadata gate is not ready for
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headb61111b6408853b8b7f8284e72adef5595e7a198. -
Head SHA:
b61111b6408853b8b7f8284e72adef5595e7a198 -
Workflow run: 28914293258
-
Workflow attempt: 1
Coverage evidence
Coverage Evidence
- Head SHA:
b61111b6408853b8b7f8284e72adef5595e7a198 - Required test evidence: supported repository test suites must pass.
- Required docstring evidence: repository-owned docstring gates must pass when configured; otherwise docstring coverage is advisory.
Python project dependencies (backend/requirements.txt)
Defaulting to user installation because normal site-packages is not writeable
Collecting fastapi==0.138.2 (from -r requirements.txt (line 1))
Downloading fastapi-0.138.2-py3-none-any.whl.metadata (26 kB)
Collecting starlette==1.3.1 (from -r requirements.txt (line 2))
Downloading starlette-1.3.1-py3-none-any.whl.metadata (6.4 kB)
Collecting uvicorn==0.49.0 (from -r requirements.txt (line 3))
Downloading uvicorn-0.49.0-py3-none-any.whl.metadata (6.7 kB)
Requirement already satisfied: pytest==9.1.1 in /home/runner/.local/lib/python3.12/site-packages (from -r requirements.txt (line 4)) (9.1.1)
Collecting httpx==0.28.1 (from -r requirements.txt (line 5))
Downloading httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB)
Collecting pydantic-settings==2.14.2 (from -r requirements.txt (line 6))
Downloading pydantic_settings-2.14.2-py3-none-any.whl.metadata (3.4 kB)
Collecting aiosmtplib==5.1.2 (from -r requirements.txt (line 7))
Downloading aiosmtplib-5.1.2-py3-none-any.whl.metadata (3.6 kB)
Collecting aioimaplib==2.0.1 (from -r requirements.txt (line 8))
Downloading aioimaplib-2.0.1-py3-none-any.whl.metadata (9.5 kB)
Collecting sqlalchemy==2.0.51 (from -r requirements.txt (line 9))
Downloading sqlalchemy-2.0.51-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (9.5 kB)
Collecting alembic==1.18.5 (from -r requirements.txt (line 10))
Downloading alembic-1.18.5-py3-none-any.whl.metadata (7.2 kB)
Collecting greenlet==3.5.3 (from -r requirements.txt (line 11))
Downloading greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (3.8 kB)
Collecting asyncpg==0.31.0 (from -r requirements.txt (line 12))
Downloading asyncpg-0.31.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (4.4 kB)
Collecting pgvector==0.4.2 (from -r requirements.txt (line 13))
Downloading pgvector-0.4.2-py3-none-any.whl.metadata (19 kB)
Collecting pytest-asyncio==1.4.0 (from -r requirements.txt (line 14))
Downloading pytest_asyncio-1.4.0-py3-none-any.whl.metadata (4.1 kB)
Collecting openai==2.44.0 (from -r requirements.txt (line 15))
Downloading openai-2.44.0-py3-none-any.whl.metadata (34 kB)
Collecting langchain-text-splitters==1.1.2 (from -r requirements.txt (line 16))
Downloading langchain_text_splitters-1.1.2-py3-none-any.whl.metadata (3.3 kB)
Collecting tiktoken==0.13.0 (from -r requirements.txt (line 17))
Downloading tiktoken-0.13.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (6.7 kB)
Collecting google-api-python-client==2.198.0 (from -r requirements.txt (line 18))
Downloading google_api_python_client-2.198.0-py3-none-any.whl.metadata (7.0 kB)
Collecting google-auth-httplib2==0.4.0 (from -r requirements.txt (line 19))
Downloading google_auth_httplib2-0.4.0-py3-none-any.whl.metadata (3.0 kB)
Collecting google-auth-oauthlib==1.4.0 (from -r requirements.txt (line 20))
Downloading google_auth_oauthlib-1.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting email-validator==2.3.0 (from -r requirements.txt (line 21))
Downloading email_validator-2.3.0-py3-none-any.whl.metadata (26 kB)
Collecting cryptography==49.0.0 (from -r requirements.txt (line 22))
Downloading cryptography-49.0.0-cp311-abi3-manylinux_2_34_x86_64.whl.metadata (4.3 kB)
Collecting python-multipart==0.0.32 (from -r requirements.txt (line 23))
Downloading python_multipart-0.0.32-py3-none-any.whl.metadata (2.1 kB)
Collecting prometheus-fastapi-instrumentator==8.0.2 (from -r requirements.txt (line 24))
Downloading prometheus_fastapi_instrumentator-8.0.2-py3-none-any.whl.metadata (13 kB)
Collecting opentelemetry-api==1.43.0 (from -r requirements.txt (line 25))
Downloading opentelemetry_api-1.43.0-py3-none-any.whl.metadata (1.4 kB)
Collecting opentelemetry-sdk==1.43.0 (from -r requirements.txt (line 26))
Downloading opentelemetry_sdk-1.43.0-py3-none-any.whl.metadata (1.7 kB)
Collecting opentelemetry-instrumentation-fastapi==0.64b0 (from -r requirements.txt (line 27))
Downloading opentelemetry_instrumentation_fastapi-0.64b0-py3-none-any.whl.metadata (2.2 kB)
Collecting opentelemetry-exporter-otlp==1.43.0 (from -r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp-1.43.0-py3-none-any.whl.metadata (2.4 kB)
Collecting protobuf==7.35.1 (from -r requirements.txt (line 29))
Downloading protobuf-7.35.1-cp310-abi3-manylinux2014_x86_64.whl.metadata (595 bytes)
Collecting setuptools==82.0.1 (from -r requirements.txt (line 30))
Downloading setuptools-82.0.1-py3-none-any.whl.metadata (6.5 kB)
Collecting wheel==0.47.0 (from -r requirements.txt (line 31))
Downloading wheel-0.47.0-py3-none-any.whl.metadata (2.3 kB)
Collecting websockets==16.0 (from -r requirements.txt (line 32))
Downloading websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (6.8 kB)
Collecting PyJWT==2.13.0 (from -r requirements.txt (line 33))
Downloading pyjwt-2.13.0-py3-none-any.whl.metadata (3.4 kB)
Collecting icalendar==7.2.0 (from -r requirements.txt (line 34))
Downloading icalendar-7.2.0-py3-none-any.whl.metadata (6.9 kB)
Collecting ruff==0.15.20 (from -r requirements.txt (line 35))
Downloading ruff-0.15.20-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (26 kB)
Collecting defusedxml==0.7.1 (from -r requirements.txt (line 36))
Downloading defusedxml-0.7.1-py2.py3-none-any.whl.metadata (32 kB)
Collecting pydantic>=2.9.0 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading pydantic-2.13.4-py3-none-any.whl.metadata (109 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.4/109.4 kB 319.0 kB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/lib/python3/dist-packages (from fastapi==0.138.2->-r requirements.txt (line 1)) (4.10.0)
Collecting typing-inspection>=0.4.2 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)
Collecting annotated-doc>=0.0.2 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB)
Collecting anyio<5,>=3.6.2 (from starlette==1.3.1->-r requirements.txt (line 2))
Downloading anyio-4.14.1-py3-none-any.whl.metadata (4.6 kB)
Requirement already satisfied: click>=7.0 in /usr/lib/python3/dist-packages (from uvicorn==0.49.0->-r requirements.txt (line 3)) (8.1.6)
Collecting h11>=0.8 (from uvicorn==0.49.0->-r requirements.txt (line 3))
Downloading h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)
Requirement already satisfied: iniconfig>=1.0.1 in /home/runner/.local/lib/python3.12/site-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (2.3.0)
Requirement already satisfied: packaging>=22 in /usr/lib/python3/dist-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (24.0)
Requirement already satisfied: pluggy<2,>=1.5 in /home/runner/.local/lib/python3.12/site-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (1.6.0)
Requirement already satisfied: pygments>=2.7.2 in /usr/lib/python3/dist-packages (from pytest==9.1.1->-r requirements.txt (line 4)) (2.17.2)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from httpx==0.28.1->-r requirements.txt (line 5)) (2023.11.17)
Collecting httpcore==1.* (from httpx==0.28.1->-r requirements.txt (line 5))
Downloading httpcore-1.0.9-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: idna in /usr/lib/python3/dist-packages (from httpx==0.28.1->-r requirements.txt (line 5)) (3.6)
Collecting python-dotenv>=0.21.0 (from pydantic-settings==2.14.2->-r requirements.txt (line 6))
Downloading python_dotenv-1.2.2-py3-none-any.whl.metadata (27 kB)
Collecting Mako (from alembic==1.18.5->-r requirements.txt (line 10))
Downloading mako-1.3.12-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.8.0 (from fastapi==0.138.2->-r requirements.txt (line 1))
Downloading typing_extensions-4.16.0-py3-none-any.whl.metadata (3.3 kB)
Collecting numpy (from pgvector==0.4.2->-r requirements.txt (line 13))
Downloading numpy-2.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai==2.44.0->-r requirements.txt (line 15)) (1.9.0)
Collecting jiter<1,>=0.10.0 (from openai==2.44.0->-r requirements.txt (line 15))
Downloading jiter-0.16.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.2 kB)
Collecting sniffio (from openai==2.44.0->-r requirements.txt (line 15))
Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting tqdm>4 (from openai==2.44.0->-r requirements.txt (line 15))
Downloading tqdm-4.68.4-py3-none-any.whl.metadata (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.4/57.4 kB 16.8 MB/s eta 0:00:00
Collecting langchain-core<2.0.0,>=1.2.31 (from langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading langchain_core-1.4.8-py3-none-any.whl.metadata (4.7 kB)
Collecting regex (from tiktoken==0.13.0->-r requirements.txt (line 17))
Downloading regex-2026.6.28-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.5/40.5 kB 6.5 MB/s eta 0:00:00
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from tiktoken==0.13.0->-r requirements.txt (line 17)) (2.31.0)
Requirement already satisfied: httplib2<1.0.0,>=0.19.0 in /usr/lib/python3/dist-packages (from google-api-python-client==2.198.0->-r requirements.txt (line 18)) (0.20.4)
Collecting google-auth!=2.24.0,!=2.25.0,<3.0.0,>=1.32.0 (from google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading google_auth-2.55.2-py3-none-any.whl.metadata (5.2 kB)
Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0,>=1.31.5 (from google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading google_api_core-2.31.0-py3-none-any.whl.metadata (3.2 kB)
Collecting uritemplate<5,>=3.0.1 (from google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading uritemplate-4.2.0-py3-none-any.whl.metadata (2.6 kB)
Collecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib==1.4.0->-r requirements.txt (line 20))
Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl.metadata (11 kB)
Collecting dnspython>=2.0.0 (from email-validator==2.3.0->-r requirements.txt (line 21))
Downloading dnspython-2.8.0-py3-none-any.whl.metadata (5.7 kB)
Collecting cffi>=2.0.0 (from cryptography==49.0.0->-r requirements.txt (line 22))
Downloading cffi-2.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.5 kB)
Collecting prometheus-client<1.0.0,>=0.8.0 (from prometheus-fastapi-instrumentator==8.0.2->-r requirements.txt (line 24))
Downloading prometheus_client-0.25.0-py3-none-any.whl.metadata (2.1 kB)
Collecting opentelemetry-semantic-conventions==0.64b0 (from opentelemetry-sdk==1.43.0->-r requirements.txt (line 26))
Downloading opentelemetry_semantic_conventions-0.64b0-py3-none-any.whl.metadata (2.4 kB)
Collecting opentelemetry-instrumentation-asgi==0.64b0 (from opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading opentelemetry_instrumentation_asgi-0.64b0-py3-none-any.whl.metadata (2.0 kB)
Collecting opentelemetry-instrumentation==0.64b0 (from opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading opentelemetry_instrumentation-0.64b0-py3-none-any.whl.metadata (7.2 kB)
Collecting opentelemetry-util-http==0.64b0 (from opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading opentelemetry_util_http-0.64b0-py3-none-any.whl.metadata (2.6 kB)
Collecting opentelemetry-exporter-otlp-proto-grpc==1.43.0 (from opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp_proto_grpc-1.43.0-py3-none-any.whl.metadata (2.6 kB)
Collecting opentelemetry-exporter-otlp-proto-http==1.43.0 (from opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp_proto_http-1.43.0-py3-none-any.whl.metadata (2.4 kB)
Requirement already satisfied: python-dateutil in /usr/lib/python3/dist-packages (from icalendar==7.2.0->-r requirements.txt (line 34)) (2.8.2)
Collecting tzdata>=2025.3 (from icalendar==7.2.0->-r requirements.txt (line 34))
Downloading tzdata-2026.2-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting googleapis-common-protos~=1.57 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading googleapis_common_protos-1.75.0-py3-none-any.whl.metadata (8.6 kB)
Collecting grpcio<2.0.0,>=1.63.2 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading grpcio-1.81.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (3.7 kB)
Collecting opentelemetry-exporter-otlp-proto-common==1.43.0 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_exporter_otlp_proto_common-1.43.0-py3-none-any.whl.metadata (1.8 kB)
Collecting opentelemetry-proto==1.43.0 (from opentelemetry-exporter-otlp-proto-grpc==1.43.0->opentelemetry-exporter-otlp==1.43.0->-r requirements.txt (line 28))
Downloading opentelemetry_proto-1.43.0-py3-none-any.whl.metadata (2.3 kB)
Collecting wrapt<3.0.0,>=1.0.0 (from opentelemetry-instrumentation==0.64b0->opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading wrapt-2.2.2-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (7.4 kB)
Collecting asgiref~=3.0 (from opentelemetry-instrumentation-asgi==0.64b0->opentelemetry-instrumentation-fastapi==0.64b0->-r requirements.txt (line 27))
Downloading asgiref-3.11.1-py3-none-any.whl.metadata (9.3 kB)
Collecting pycparser (from cffi>=2.0.0->cryptography==49.0.0->-r requirements.txt (line 22))
Downloading pycparser-3.0-py3-none-any.whl.metadata (8.2 kB)
Collecting proto-plus<2.0.0,>=1.24.0 (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0,>=1.31.5->google-api-python-client==2.198.0->-r requirements.txt (line 18))
Downloading proto_plus-1.28.0-py3-none-any.whl.metadata (2.2 kB)
Collecting requests (from tiktoken==0.13.0->-r requirements.txt (line 17))
Downloading requests-2.34.2-py3-none-any.whl.metadata (4.8 kB)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth!=2.24.0,!=2.25.0,<3.0.0,>=1.32.0->google-api-python-client==2.198.0->-r requirements.txt (line 18)) (0.2.8)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /usr/lib/python3/dist-packages (from httplib2<1.0.0,>=0.19.0->google-api-python-client==2.198.0->-r requirements.txt (line 18)) (3.1.1)
Collecting jsonpatch<2.0.0,>=1.33.0 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading jsonpatch-1.33-py2.py3-none-any.whl.metadata (3.0 kB)
Collecting langchain-protocol>=0.0.17 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading langchain_protocol-0.0.18-py3-none-any.whl.metadata (2.4 kB)
Collecting langsmith<1.0.0,>=0.3.45 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading langsmith-0.9.8-py3-none-any.whl.metadata (22 kB)
Requirement already satisfied: pyyaml<7.0.0,>=5.3.0 in /usr/lib/python3/dist-packages (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16)) (6.0.1)
Collecting tenacity!=8.4.0,<10.0.0,>=8.1.0 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading tenacity-9.1.4-py3-none-any.whl.metadata (1.2 kB)
Collecting uuid-utils<1.0,>=0.12.0 (from langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading uuid_utils-0.16.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.4 kB)
Collecting annotated-types>=0.6.0 (from pydantic>=2.9.0->fastapi==0.138.2->-r requirements.txt (line 1))
Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
Collecting pydantic-core==2.46.4 (from pydantic>=2.9.0->fastapi==0.138.2->-r requirements.txt (line 1))
Downloading pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting charset_normalizer<4,>=2 (from requests->tiktoken==0.13.0->-r requirements.txt (line 17))
Downloading charset_normalizer-3.4.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (41 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.7/41.7 kB 13.0 MB/s eta 0:00:00
Requirement already satisfied: urllib3<3,>=1.26 in /usr/lib/python3/dist-packages (from requests->tiktoken==0.13.0->-r requirements.txt (line 17)) (2.0.7)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/lib/python3/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib==1.4.0->-r requirements.txt (line 20)) (3.2.2)
Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/lib/python3/dist-packages (from Mako->alembic==1.18.5->-r requirements.txt (line 10)) (2.1.5)
Requirement already satisfied: jsonpointer>=1.9 in /usr/lib/python3/dist-packages (from jsonpatch<2.0.0,>=1.33.0->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16)) (2.0)
Collecting orjson>=3.9.14 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading orjson-3.11.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (41 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.0/42.0 kB 13.6 MB/s eta 0:00:00
Collecting requests-toolbelt>=1.0.0 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading requests_toolbelt-1.0.0-py2.py3-none-any.whl.metadata (14 kB)
Collecting xxhash>=3.0.0 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading xxhash-3.8.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (15 kB)
Collecting zstandard>=0.23.0 (from langsmith<1.0.0,>=0.3.45->langchain-core<2.0.0,>=1.2.31->langchain-text-splitters==1.1.2->-r requirements.txt (line 16))
Downloading zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (3.3 kB)
Downloading fastapi-0.138.2-py3-none-any.whl (129 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.3/129.3 kB 19.6 MB/s eta 0:00:00
Downloading starlette-1.3.1-py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.6/73.6 kB 25.4 MB/s eta 0:00:00
Downloading uvicorn-0.49.0-py3-none-any.whl (71 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.4/71.4 kB 27.3 MB/s eta 0:00:00
Downloading httpx-0.28.1-py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.5/73.5 kB 26.7 MB/s eta 0:00:00
Downloading pydantic_settings-2.14.2-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.7/61.7 kB 24.9 MB/s eta 0:00:00
Downloading aiosmtplib-5.1.2-py3-none-any.whl (28 kB)
Downloading aioimaplib-2.0.1-py3-none-any.whl (34 kB)
Downloading sqlalchemy-2.0.51-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 140.2 MB/s eta 0:00:00
Downloading alembic-1.18.5-py3-none-any.whl (264 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 264.7/264.7 kB 80.3 MB/s eta 0:00:00
Downloading greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (614 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 614.2/614.2 kB 121.5 MB/s eta 0:00:00
Downloading asyncpg-0.31.0-cp312-cp312-manylinux_2_28_x86_64.whl (3.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 136.8 MB/s eta 0:00:00
Downloading pgvector-0.4.2-py3-none-any.whl (27 kB)
Downloading pytest_asyncio-1.4.0-py3-none-any.whl (16 kB)
Downloading openai-2.44.0-py3-none-any.whl (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 139.8 MB/s eta 0:00:00
- Result: PASS
Python coverage with missing-line report (backend)
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/runner/work/naruon/naruon/pr-head/backend
configfile: pytest.ini
## Changed-File Evidence Map
```mermaid
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (4 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (4 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (6 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (6 files)"]
R2 --> V2["backend tests"]
Contributor
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage Evidence
Python project dependencies (backend/requirements.txt)
Python coverage with missing-line report (backend) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates the org's standalone
pg_tiktokenPostgres batch engine(pg-llm-batch) as naruon's first git submodule and wires the
batch-tolerant email-import embedding hotspot to it. The feature is fully
optional and degrades gracefully: naruon runs unchanged with the submodule
uninitialized or batching unconfigured, falling back to the existing per-item
embedding path.
What changed
.gitmodules+vendor/pg-llm-batchpinned at9f40192(Apache-2.0 extracted core; all deps permissive/commercially sellable). Sets
the submodule convention (naruon had none).
backend/services/batch_embedding_service.py:tenant_configs),never
os.getenv. The batch DSN is a FernetEncryptedStringat rest.ImportError-> fall back).TokenCounter/BatchAccumulatorto token/byte/record-partition the texts, records a job/item audit trail, returns fitted vectors
or
None(fall back).email_import_service._generate_import_embeddingsroutesthrough an
EmailImportBatchContextthreaded fromimport_email_uploads.Interactive draft/translate paths are intentionally untouched. Provider creds
still resolve via
resolve_runtime_llm_provider(Fernet DB, per-tenant).provider_writeback_retry_items):llm_batch_jobs,llm_batch_items, plustenant_configsbatch columns(
batch_embedding_enabled,batch_embedding_dsn,batch_embedding_endpoint,batch_embedding_model). Migration0010_llm_batch_embedding.docker-compose.pg-llm-batch.yml(pg_tiktoken image via thesubmodule build context) + README section.
Tests
New fast, fully-mocked unit tests (
backend/tests/test_batch_embedding_service.py):routing through the batch component, graceful fallback when disabled / submodule
missing, and config-from-Fernet-DB (not env) with a ciphertext-at-rest
assertion. Plus a
0010migration assertion. Full backend suite green(
1145 passed; 2 pre-existingtest_release_governancefailures are unrelated —missing
.github/workflows/*.ymlon the branch, reproduced on a clean tree).CodeGraph
Indexed the fresh naruon clone (
codegraph init) and located the call sites viacodegraph_explore "email_import_service _generate_import_embeddings embedding.py resolve_runtime_llm_provider llm_provider_selection", which returned the verbatimsources and blast radius (4 callers of
_generate_import_embeddings; 11 ofresolve_runtime_llm_provider) used to scope the wiring.Human action required (to enable; off by default)
git submodule update --init vendor/pg-llm-batchandpip install -e vendor/pg-llm-batch.docker compose -f docker-compose.yml -f docker-compose.pg-llm-batch.yml up.tenant_configs):batch_embedding_enabled = true, Fernet-encryptedbatch_embedding_dsn,optional
batch_embedding_endpoint/batch_embedding_model.🤖 Generated with Claude Code