feat(pdf-dom): recognize PDF DOM via NewsDOM sidecar into the content graph - #965
Conversation
… graph
Integrate the NewsDOM PDF recognition sidecar so naruon can turn scanned/native
PDFs into a structured DOM and land it in the content graph.
- newsdom_client.py: SSRF-safe, address-pinned async httpx client that POSTs
multipart PDF bytes to {base_url}/parse with language/mode form fields and an
optional bearer token. Validates against a new ALLOWED_NEWSDOM_HOSTS allowlist
(mirrors ALLOWED_LLM_BASE_URL_HOSTS) with an ALLOW_LOCAL_NEWSDOM_PROVIDERS
escape hatch for the docker sidecar.
- NewsdomProvider model + migration 0010: base_url in plaintext, api_token as
EncryptedString (Fernet), read from the DB per-org — never os.getenv at
runtime, mirroring LLMProvider.
- Attachment path: application/pdf now maps to a deferred 'pdf' descriptor
(parse_status pdf_dom_recognition_pending) instead of unsupported_binary, so
heavy OCR/MinerU never runs inline during import. The worker maps the returned
pages->articles->body_blocks tree into Attachment.parse_content (embeddings)
and a document->section->paragraph ContentNode/ContentSegment graph.
- Data-documents path: pdf-dom-recognition intent endpoint (status
pdf_dom_recognition_pending) mirroring the HWP conversion intent, plus a binary
PDF upload variant; executed by the worker calling newsdom_client.
- Submodule + sidecar: vendor/newsdom-api pinned submodule and a 'newsdom'
docker-compose service (healthcheck on /health). naruon degrades gracefully
without the sidecar (PDFs stay pending).
- Tests: fast mocked unit tests for the pdf descriptor, the DOM->content-graph
mapping, config-resolved-from-DB, the recognition worker with a mocked client,
and SSRF allowlist rejection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
리뷰 참고 — 사람 조치 및 후속 항목 (검증 결과)사람 조치:
후속(비차단):
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
PR governance metadata gate is not ready for
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The coverage-evidence CI gate runs `docker compose config` + `docker compose build` whenever docker-compose.yml changes. The newsdom service builds from context ./vendor/newsdom-api, a submodule that is not checked out in that job, so the build failed with: target newsdom: failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory newsdom is an optional PDF->DOM sidecar; the backend already degrades gracefully when it is absent. Put it behind a 'newsdom' compose profile so the default build/config path skips it, and make the backend's depends_on required: false so config/build does not fail when the profile is inactive (dropped from the dependency graph) while still waiting for the sidecar when the profile is enabled. Bring it up intentionally with: git submodule update --init vendor/newsdom-api docker compose --profile newsdom up Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
Remove the unused module-level `from services import newsdom_client` import (ruff F401) and replace the `startswith` prefix check with an exact-equality assertion on the normalized URL. The prefix check triggered CodeQL py/incomplete-url-substring-sanitization since it would also accept hosts like newsdom.example.com.evil.com; the deterministic normalizer output lets the test assert the full value instead.
|
@copilot resolve the merge conflicts on this branch. |
Head branch was pushed to by a user without write access
…dar descriptors and ALLOWED_NEWSDOM_HOSTS + ALLOWED_SCOPEWEAVE_HOSTS
Merge conflicts resolved. Three files had conflicts:
All 30 affected tests pass (commit |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head2559283c7ab1c5865691db37d62293865988d67a. -
Head SHA:
2559283c7ab1c5865691db37d62293865988d67a -
Workflow run: 29205347662
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (3 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (3 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (15 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (15 files)"]
R2 --> V2["backend tests"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage evidence job did not run or did not publish coverage evidence. Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (3 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (3 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (15 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (15 files)"]
R2 --> V2["backend tests"]
|
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head4e257f5fc6cf9983be2521af1542235631f2eb99. -
Head SHA:
4e257f5fc6cf9983be2521af1542235631f2eb99 -
Workflow run: 29213382582
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (3 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (3 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (15 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (15 files)"]
R2 --> V2["backend tests"]
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head8986e738afd9441d023c73134aeb85a119e03432. -
Head SHA:
8986e738afd9441d023c73134aeb85a119e03432 -
Workflow run: 29215344235
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (3 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (3 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (15 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (15 files)"]
R2 --> V2["backend tests"]
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headf99a0e967f954de8431de6b7bae320d94baa9143. -
Head SHA:
f99a0e967f954de8431de6b7bae320d94baa9143 -
Workflow run: 29220153338
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (3 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (3 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (15 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (15 files)"]
R2 --> V2["backend tests"]
…odex/naruon-965-audit
|
리뷰 후속 항목을 current-head 기준으로 재검증했습니다.
로컬 검증: backend |
…odex/naruon-965-audit # Conflicts: # backend/tests/test_repo_hygiene.py # frontend/screenshot.cjs
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including backend/alembic/versions/0010_newsdom_providers.py, backend/alembic/versions/0015_merge_newsdom_email_heads.py, backend/alembic/versions/0016_document_org_scope.py, backend/alembic/versions/0017_merge_newsdom_carddav_heads.py, backend/api/data.py, and 26 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects backend/alembic/versions/0010_newsdom_providers.py to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: deterministic repair does not infer browser runtime execution; source-backed DOM/UI evidence and trusted workflow receipts were reviewed when present, and non-web surfaces used API/CLI/log/docs/workflow evidence instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
Adversarial validation
{"status":"passed","probes":[{"path":"backend/services/newsdom_worker.py","line":367,"hypothesis":"Worker may fail to handle concurrent sweeps","attack_or_counterexample":"Simulated concurrent sweeps","evidence":"Worker uses advisory locks to prevent concurrent sweeps","outcome":"falsified"},{"path":"backend/services/newsdom_pdf_recognition.py","line":63,"hypothesis":"Missing provider config may cause errors","attack_or_counterexample":"Null provider input","evidence":"Function gracefully returns None for missing/inactive providers","outcome":"falsified"}],"residual_risk":"Low risk of edge cases in provider configuration handling"}- Result: APPROVE
- Reason: PR integrates PDF DOM recognition with comprehensive tests and coverage
- Head SHA:
9fcd5d17e38a7bc7cff8bcfd2479de195a1f36c8 - Workflow run: 29255409437
- Workflow attempt: 1
github-project-automation
Bot
moved this from In Progress
to Done
in naruon Platform Roadmap
Summary
Integrates the NewsDOM PDF recognition sidecar so naruon recognizes PDF DOM structure and lands the structured DOM into the content graph. PDFs were previously
unsupported_binary; they are now recognized (pages → articles → body_blocks) intodocument → section → paragraphContentNode/ContentSegmenttrees plus flat text for embeddings.Targets the generalized NewsDOM
/parsecontract (multipartfile+language/modeform fields + optional bearer) being landed in the siblingnewsdom-apibranchfeat/generalize-pdf-dom. Extra form fields / auth header are ignored by the current sidecar, so this is forward-compatible.What's included
services/newsdom_client.py— SSRF-safe, address-pinned async httpx client (mirrorsservices/llm_provider_urls.py). POSTs the PDF to{base_url}/parse, validates the host against a newALLOWED_NEWSDOM_HOSTSallowlist, resolves + pins IPs, rejects private/loopback/IP-literal/userinfo/DNS-rebind.ALLOW_LOCAL_NEWSDOM_PROVIDERSunlocks the docker container hostname for dev.NewsdomProvidermodel + migration0010_newsdom_providers—base_urlplaintext,api_tokenasEncryptedString(Fernet), resolved from the DB per organization. Noos.getenvfor service config/secrets in consumer code.application/pdfmaps to a new deferredpdfdescriptor with statuspdf_dom_recognition_pending; heavy OCR/MinerU never runs inline during import. The worker maps the recognized tree intoAttachment.parse_content+ the content graph.POST /documents/{id}/pdf-dom-recognition-intent(mirrors the HWP conversion intent) + a binary PDF upload variantPOST /documents/pdf-dom-recognition; executed by the worker callingnewsdom_client.vendor/newsdom-apipinned git submodule (ContextualWisdomLab/newsdom-api@ develop4a68893) and anewsdomdocker-compose service (build from submodule, healthcheck on/health). Backend allowlistsnewsdomand depends on it.Graceful degradation
Without the sidecar (or with no active
NewsdomProvider),resolve_newsdom_runtime_configreturnsNone, PDFs stay in the clearpdf_dom_recognition_pendingstate, and imports never fail.Test status
Full backend suite: 1159 passed, 22 skipped, 2 failed. The 2 failures are pre-existing
test_release_governance.pychecks about missing.github/workflows/*files, unrelated to this change (no governance files were touched). The 28 new tests all pass.Human action required
NewsdomProviderrow (per organization) withbase_url+api_token(token is Fernet-encrypted at rest) and add its host toALLOWED_NEWSDOM_HOSTS. Nothing is read from the environment at request time.git submodule update --init --recursive. Re-pinvendor/newsdom-apionce the siblingfeat/generalize-pdf-dom/parsegeneralization lands on newsdom-apidevelop.🤖 Generated with Claude Code