feat: add pdf ingestion lane - #16
Conversation
WalkthroughAdds a new pdf-ingest service and its Dockerfile, app, and requirements; introduces env vars and docker-compose wiring; extends Makefile with smoke tests and loaders; documents PDF ingestion; adds a new “ingest.document.ready.v1” topic and schema; updates topics catalog; and includes supporting examples and configs. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor U as Client
participant I as pdf-ingest (FastAPI)
participant S as MinIO/S3
participant L as langextract
participant X as extract-worker
participant N as NATS
participant V as Vector/Search
U->>I: POST /pdf/ingest {bucket,key,namespace,title}
I->>S: GET object (bucket,key)
S-->>I: PDF bytes
I->>I: Parse PDF (limit pages), build text
I->>L: Extract chunks/errors (text)
L-->>I: chunks[], errors[]
I->>X: POST /ingest {doc, chunks}
X-->>I: ingest result
alt publish_events=true
I->>N: publish ingest.file.added.v1
I->>N: publish ingest.document.ready.v1
end
X->>V: Upsert chunks (downstream)
I-->>U: {doc_id, file_id, uri, chunk_count, error_count, ingest_result}
note over I,N: New topics include ingest.document.ready.v1
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Free ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (9)
Tip 👮 Agentic pre-merge checks are now available in preview!Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.
Please see the documentation for more information. Example: reviews:
pre_merge_checks:
custom_checks:
- name: "Undocumented Breaking Changes"
mode: "warning"
instructions: |
Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).Please share your feedback with us on this Discord post. Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
Codex Review: Here are some suggestions.
Reply with @codex fix comments to fix any unresolved comments.
About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".
| pdf-ingest: | ||
| build: | ||
| context: . | ||
| dockerfile: services/pdf-ingest/Dockerfile | ||
| restart: unless-stopped | ||
| env_file: [.env] | ||
| environment: | ||
| - MINIO_ENDPOINT=${MINIO_ENDPOINT:-minio:9000} | ||
| - MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY} | ||
| - MINIO_SECRET_KEY=${MINIO_SECRET_KEY} | ||
| - MINIO_SECURE=${MINIO_SECURE:-false} | ||
| - PDF_DEFAULT_BUCKET=${PDF_DEFAULT_BUCKET:-assets} | ||
| - PDF_DEFAULT_NAMESPACE=${PDF_DEFAULT_NAMESPACE:-pmoves} | ||
| - PDF_MAX_PAGES=${PDF_MAX_PAGES:-0} | ||
| - PDF_INGEST_EXTRACT_URL=${PDF_INGEST_EXTRACT_URL:-http://extract-worker:8083/ingest} | ||
| - NATS_URL=${NATS_URL:-nats://nats:4222} | ||
| depends_on: | ||
| extract-worker: | ||
| condition: service_started | ||
| minio: | ||
| condition: service_started | ||
| ports: ["8092:8092"] | ||
| profiles: ["workers","orchestration"] | ||
| networks: [pmoves] |
There was a problem hiding this comment.
[P1] Expose pdf-ingest on host port already used by publisher-discord
The new pdf-ingest service maps ports: ["8092:8092"] while publisher-discord already binds the same host port in the orchestration profile. Bringing up both profiles together (docker compose --profile orchestration up) will fail with a port-collision error and neither service will start. Consider assigning a different host port or making one of the bindings optional to avoid blocking the compose stack.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Summary
- Updated the
pdf-ingestservice’s host port mapping to 8095 and aligned the ingestion documentation with the new endpoint so it no longer conflicts withpublisher-discord. pmoves/docker-compose.ymlL176-L189 pmoves/docs/DATA_IMPORT.mdL30-L41 - Pointed the
smoke-pdfMakefile target at the new pdf-ingest host port for both verification steps. pmoves/MakefileL60-L76 - Scheduled NATS publishes on the application’s event loop so the synchronous endpoint can emit events without runtime errors. pmoves/services/pdf-ingest/app.pyL55-L116
Testing
- ✅
python -m compileall pmoves/services/pdf-ingest
There was a problem hiding this comment.
Summary
- Remapped the pdf-ingest service’s host port to 8094 in docker-compose so it no longer collides with publisher-discord when orchestration is enabled. pmoves/docker-compose.ymlL176-L189
- Updated the
smoke-pdfMakefile target to call the pdf-ingest endpoint on the new host port throughout the smoke flow. pmoves/MakefileL60-L76 - Adjusted the data import guide to reference the revised pdf-ingest port in the example ingestion command. pmoves/docs/DATA_IMPORT.mdL38-L41
Testing
⚠️ Tests not run (not requested)
…tion-status feat: add pdf ingestion lane
Update setup_mcp.py to work with different platforms
…836) * docs(audit): resolve AB-9, update dashboard with Mar 9 findings - AB-9 (runner queue starvation) RESOLVED: 3/4 runners online, CI queue healthy, CodeQL completing in ~4min - PRs #834/#835 merge tracking added - Dependabot: 0 open (medium alert resolved) - Trivy failure triage: agent-zero timeout (infra), archon/deepresearch upstream dep pins needed, pmoves-yt urllib3 quick fix - Docker Bench Security unblocked by AB-9 resolution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(security): refresh P2 tracker with tiered prioritization - Update date from 2026-02-26 to 2026-03-09 - Re-prioritize 14 open items into 3 tiers: 4 production-blocking, 6 tracked improvements, 5 cosmetic - Add "Blocks Production?" column with rationale per item - Confirm no P2s fixed by PRs #827-#835 (CI/docs/build-gate only) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(docs): correct P2 open-item count from 14 to 15 Items #1-#14 and #16 are open (#15 is closed), totaling 15. Fixes count in both P2 tracker and dashboard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Shaela Bello <slbello@uncg.edu> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…p (camera-fit + viewer-hardening) (#1681) Records the fork commit production runs after merging fork #15 (embed camera fit) and #16 (#1667 concern-B surfaceFn guard) into PMOVES.AI-Edition-Hardened. c3c2b39 -> bf3d922. website/hyperdim/ is generated/gitignored and re-syncs from this pin at deploy; this keeps main's recorded pointer in step with the live deploy for reproducibility. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Summary
pdf-ingestFastAPI service that downloads PDFs from MinIO, extracts them withlibs.langextract, forwards chunks toextract-worker, and emits ingestion eventsingest.document.ready.v1contract, wire the service into docker-compose, provide defaults/smoke coverage, and document the MinIO-to-Qdrant flowTesting
https://chatgpt.com/codex/tasks/task_b_68c9dec59e5c8324a68fa4f8a8ad6b97
Summary by CodeRabbit