feat: preserve OCR page structure and sanitize parser failures

ARCHITECTURE.md

@@ -13,9 +13,11 @@ orchestration, MinerU process execution, and DOM normalization.
 - `src/newsdom_api/service.py` orchestrates PDF parsing,
   temporary files, and response construction.
 - `src/newsdom_api/mineru_runner.py` shells out to the MinerU CLI,
-  collects JSON outputs, and surfaces process errors.
+  collects JSON outputs, and translates runtime or incomplete-output
+  failures into typed sanitized exceptions.
 - `src/newsdom_api/dom_builder.py` converts MinerU `content_list`
-  blocks into the canonical NewsDOM response model.
+  blocks plus page model metadata into the canonical NewsDOM response
+  model.
 - `src/newsdom_api/schemas.py` defines the public response schema.
 - `src/newsdom_api/synthetic.py` and
   `src/newsdom_api/equivalence.py` support synthetic fixture
@@ -27,9 +29,13 @@ orchestration, MinerU process execution, and DOM normalization.
 2. `src/newsdom_api/service.py` writes the upload to a temporary
    workspace and calls MinerU.
 3. `src/newsdom_api/mineru_runner.py` resolves the executable, runs
-   the OCR pipeline, and loads generated JSON artifacts.
-4. `src/newsdom_api/dom_builder.py` normalizes OCR blocks into the canonical response.
-5. FastAPI returns typed JSON from `src/newsdom_api/schemas.py`.
+   the OCR pipeline, loads generated JSON artifacts, and raises typed
+   sanitized errors for runtime-unavailable or incomplete-output cases.
+4. `src/newsdom_api/dom_builder.py` normalizes OCR blocks into the
+   canonical response while preserving page-aware structure from
+   MinerU model metadata.
+5. FastAPI returns typed JSON from `src/newsdom_api/schemas.py` and
+   maps MinerU runtime failures to 503 and incomplete output to 502.
 
 ## Supporting systems
 

README.md

@@ -44,14 +44,13 @@ docker build -t newsdom-api .
 docker run -p 8000:8000 newsdom-api
 ```
 
-The default image exposes the REST API on port `8000` as a lean multi-arch
-service image. It is suitable for `linux/amd64` and `linux/arm64`, including
-Apple Silicon hosts running the API service inside Docker.
+The default image exposes the REST API on port `8000` as a multi-arch service
+image. It is suitable for `linux/amd64` and `linux/arm64`, including Apple
+Silicon hosts running the API service inside Docker.
 
-The lean image is intentionally a REST API shell: `/health`, `/docs`, and
-OpenAPI endpoints are available immediately, while real `/parse` execution
-still requires a compatible MinerU runtime to be available inside the
-container image.
+The default image already includes the MinerU runtime and sets
+`NEWSDOM_MINERU_BIN=mineru`, so `/parse` is available without layering an extra
+OCR package into the container at runtime.
 
 For heavier parsing deployments, build the optional NVIDIA-oriented variant:
 
@@ -75,6 +74,12 @@ provide the CUDA user-space/runtime stack required by MinerU.
 curl -F "file=@sample.pdf" http://127.0.0.1:8000/parse
 ```
 
+Each request is written to a request-scoped temporary directory before MinerU
+runs, and those temporary files are removed after the response completes.
+Sanitized parse failures return `503 MinerU runtime unavailable` when the
+runtime cannot be executed and `502 MinerU output was incomplete` when MinerU
+finishes without the required output artifacts.
+
 ### Run tests
 
 ```bash

docs/operations/deploy-runbook.md

@@ -1,8 +1,9 @@
 # Deploy runbook
 
 This repository does not ship a long-lived production cluster
-definition in-tree, so deployment verification focuses on reproducible
-local and GitHub-hosted delivery paths.
+definition in-tree and has no in-tree Kubernetes manifests, so
+deployment verification focuses on reproducible local and GitHub-hosted
+delivery paths.
 
 ## Local API smoke
 
@@ -17,7 +18,15 @@ local and GitHub-hosted delivery paths.
 
 1. `docker build -t newsdom-api .`
 2. `docker run --rm -p 18080:8000 newsdom-api`
-3. `curl -fsS http://127.0.0.1:18080/health`
+3. Verify:
+   - `curl -fsS http://127.0.0.1:18080/health`
+   - `curl -fsS -F "file=@sample.pdf" http://127.0.0.1:18080/parse`
+
+The default image already includes the MinerU runtime and sets
+`NEWSDOM_MINERU_BIN=mineru`, so container smoke should treat `/parse`
+as part of the supported contract instead of a docs-only shell.
+
+`/health` proves the API process is serving but does not validate a full `/parse` round-trip, MinerU execution, or OCR artifact production.
 
 ## Release smoke
 
@@ -30,5 +39,8 @@ local and GitHub-hosted delivery paths.
 ## Failure handling
 
 - Capture sanitized logs outside `tmp/` when a delivery path fails.
+- Expect `/parse` failures to stay sanitized: `503 MinerU runtime
+  unavailable` when the runtime cannot execute, and `502 MinerU output
+  was incomplete` when required OCR artifacts are missing or invalid.
 - Reconcile the failure against `README.md`, `CHANGELOG.md`, and the
   relevant workflow before closing the task.

docs/plans/2026-04-23-ocr-accuracy-program-followthrough-design.md

@@ -0,0 +1,235 @@
+# Design: OCR accuracy program follow-through on fresh develop baseline
+
+## Context
+
+- This worktree is `feature/ocr-accuracy-program-followthrough-3`, created from
+  the latest `origin/develop` reference (`27d0c05`).
+- Fresh baseline verification already passed in this worktree:
+  - `uv sync --frozen --all-extras`
+  - `uv run pytest` → `139 passed`
+- Current `develop` still has the page-collapse root cause:
+  - `src/newsdom_api/service.py` forwards only `content_list`
+  - `src/newsdom_api/dom_builder.py` hardcodes one page and ignores MinerU
+    `model` metadata
+  - `src/newsdom_api/schemas.py` exposes richer structures than the builder
+    actually populates
+- Current runtime/docs state is still partially inconsistent:
+  - `Dockerfile` installs MinerU in the default image
+  - `README.md` and `manual/api-reference.md` still contain stale runtime/run
+    contract wording
+  - deploy docs verify `/health` but do not meaningfully exercise `/parse`
+- Current GitHub OCR program already has a canonical hierarchy and milestone:
+  - milestone `v0.2.0 OCR accuracy and runtime readiness`
+  - meta issue `#61`
+  - sub-issues `#60`, `#62`, `#63`, `#64`
+- Current open PR topology relevant to this work:
+  - `#57` MinerU/mkdocs-material/pyinstaller dependency bump, milestone-linked but
+    not linked to `#60-#64` in body/title
+  - `#58` GitHub Actions dependency bump, no milestone/issue linkage
+  - `#51` pypdf dependency bump, no milestone/issue linkage
+- Local-only private OCR inputs exist at
+  `/Users/seonghobae/Documents/newsdom-api/tests/copyrighted_fixtures_for_realities`
+  as five flat PDF files with no adjacent redacted outputs or sidecars.
+
+## Constraints
+
+- Never commit copyrighted source PDFs, OCR text, rendered crops, or secrets.
+- Durable evidence must be redacted and structural only: counts, warning types,
+  status, sanitized summaries, and synthetic fixtures are allowed.
+- Repository-local policy requires:
+  - branch from `develop`, target `develop`
+  - TDD for every behavior change
+  - explicit GitHub issue/PR/workflow state checks before acting
+- Acceptance still requires:
+  - `uv run pytest`
+  - `uv run pytest --cov=src/newsdom_api --cov-branch --cov-report=term-missing --cov-fail-under=100`
+  - `uv run mkdocs build --strict` when user-facing docs change
+  - targeted smoke checks for changed delivery paths
+- API hardening remains in scope because `/parse` is an externally exposed OCR
+  path:
+  - do not leak private paths or raw subprocess errors
+  - document and test sanitized failure behavior
+  - keep abuse/rate/resource implications visible even if not fully solved in
+    this slice
+- Kubernetes/runtime concerns stay in scope for docs, smoke evidence, and deploy
+  readiness even if the first implementation slice is application-level.
+- Non-interactive default applies: do not stop for human review; use robot review
+  and recorded evidence instead.
+
+## Approaches considered
+
+### 1. Start with private-fixture runtime debugging
+
+- Pros:
+  - immediately attacks the real-document runtime uncertainty
+  - may surface host/container/runtime blockers early
+- Cons:
+  - leaves the already-proven page-collapse bug unfixed
+  - depends on environment-specific runtime state before the deterministic code
+    root cause is repaired
+  - makes later benchmark interpretation noisier
+- Verdict: useful follow-up, not the first implementation slice.
+
+### 2. Restore page-aware normalization first, then sanitize runtime and docs (recommended)
+
+- Pros:
+  - fixes the known root cause described by `#62`
+  - keeps the first change set deterministic and testable
+  - allows real-document evidence and engine comparisons to run against the right
+    DOM contract
+  - fits the existing issue hierarchy in `#61`
+- Cons:
+  - runtime-safe failure handling and docs drift still need additional steps
+  - requires schema/doc updates to keep the public contract honest
+- Verdict: recommended.
+
+### 3. Bundle normalization, runtime failure mapping, docs reconciliation, and benchmark workflow in one large change
+
+- Pros:
+  - one branch can narrate the whole OCR/runtime program
+  - fewer rebases between related tasks
+- Cons:
+  - larger review and verification blast radius
+  - mixes deterministic unit-testable work with environment-sensitive
+    investigation and GitHub topology maintenance
+- Verdict: acceptable as a branch-level sequence, not as one undivided task.
+
+## Recommended design
+
+Execute the program in five layers:
+
+1. **Restore page-aware NewsDOM normalization (`#62`)**
+2. **Add sanitized runtime/output failure handling for `/parse`**
+3. **Reconcile runtime/manual/Kubernetes contract drift (`#64`)**
+4. **Capture redacted local-only OCR evidence and benchmark prerequisites (`#60`, `#63`)**
+5. **Refresh GitHub issue/PR continuity for adjacent open PRs and follow-up tasks**
+
+### Layer 1: Page-aware normalization contract
+
+- Update `src/newsdom_api/service.py` so `build_dom(...)` receives both
+  `content_list` and `model`.
+- Update `src/newsdom_api/dom_builder.py` to:
+  - group blocks by `page_idx`
+  - use `model[*].page_info.page_no`, width, and height when present
+  - preserve deterministic page order
+  - keep `header`, `footer`, `page_number`, and `ad` as page-scoped metadata
+  - preserve caption/footnote/media metadata for image/table/chart blocks
+  - emit quality warnings when page assignment is ambiguous or content/model
+    structure diverges
+- Expand `src/newsdom_api/schemas.py` only as far as required to represent the
+  preserved data truthfully.
+
+### Layer 2: Sanitized runtime failure handling
+
+- Introduce typed MinerU runtime/output exceptions.
+- `src/newsdom_api/mineru_runner.py` should wrap:
+  - missing executable failures
+  - `subprocess.CalledProcessError`
+  - incomplete output artifact cases
+- `src/newsdom_api/main.py` should map them to stable sanitized API responses:
+  - `503` for runtime/unavailable parser failures
+  - `502` for incomplete parser outputs
+- Keep private file paths and raw stderr out of public responses.
+
+### Layer 3: Runtime/manual/Kubernetes contract reconciliation
+
+- Align these truth sources with the actual code path:
+  - `README.md`
+  - `manual/api-reference.md`
+  - `docs/operations/deploy-runbook.md`
+  - any touched architecture/agent docs when the runtime contract changes
+- Document default image behavior, temp-storage expectations, `/health`, and any
+  relevant CPU/GPU or multi-arch delivery caveats.
+- Add or extend smoke evidence so `/parse` contract drift is harder to miss.
+
+### Layer 4: Local-only OCR evidence and benchmark prerequisites
+
+- Run at least one private PDF through the updated branch.
+- Store only redacted durable evidence:
+  - page counts
+  - block counts
+  - success/failure state
+  - warning categories
+  - sanitized runtime summary if failure remains
+- Use that evidence to support the next actions for `#60` and `#63`.
+
+#### 2026-04-23 local-only evidence snapshot
+
+- Input class: single private PDF fixture from
+  `/Users/seonghobae/Documents/newsdom-api/tests/copyrighted_fixtures_for_realities`
+  (`content_file_3998.pdf`)
+- Structural source evidence:
+  - page count: 16
+  - PDF encryption: yes
+- Current-branch local-only parse result:
+  - state: failure
+  - block count: unavailable because MinerU runtime did not start and no DOM was
+    produced
+  - warning categories: none observed before failure
+  - sanitized runtime summary: `MinerU runtime unavailable`
+- Readiness implication:
+  - `#60` remains blocked on a host or container environment that actually has
+    the MinerU executable available for private-fixture benchmarking
+  - `#63` can still use this snapshot as proof that the remaining gap is runtime
+    availability rather than additional unredacted output handling
+
+### Layer 5: GitHub topology continuity
+
+- Preserve `#61` as the O1 canonical program anchor.
+- Keep `#60`, `#62`, `#63`, `#64` as the active O2 task set.
+- Treat open PRs as follows:
+  - `#57`: related OCR/runtime follow-up PR; keep milestone and add explicit
+    issue linkage if missing
+  - `#58`: delivery-governance follow-up; classify as adjacent/non-core
+  - `#51`: PDF-dependency follow-up; classify as adjacent and only milestone-link
+    if current-code evidence justifies it
+- Do not create duplicate PRs for the same branch; use PR continuity before
+  opening a new PR.
+
+## Data flow
+
+1. `POST /parse` receives an uploaded PDF.
+2. `src/newsdom_api/service.py` writes a sanitized temporary filename.
+3. `src/newsdom_api/mineru_runner.py` invokes MinerU and returns structured data
+   or raises a typed sanitized exception.
+4. `src/newsdom_api/dom_builder.py` constructs a multi-page NewsDOM response from
+   `content_list` plus `model`.
+5. `src/newsdom_api.main` returns a `ParseResponse` or a stable sanitized HTTP
+   error.
+6. Local-only private-fixture reruns produce only redacted structural evidence.
+
+## Error handling
+
+- Parser runtime failures must not leak temp paths or raw subprocess details.
+- Incomplete MinerU output is a typed backend failure, not a raw file error.
+- Page-structure mismatches that still permit normalization belong in
+  `ParseQuality.warnings`, not HTTP errors.
+- Stale docs that overpromise schema/runtime behavior are acceptance failures.
+
+## Testing strategy
+
+- **TDD layer 1**
+  - failing synthetic multi-page regressions in `tests/test_dom_builder.py` and
+    `tests/test_service.py`
+- **TDD layer 2**
+  - failing runner/error-path regressions in `tests/test_mineru_runner_paths.py`
+    and parse endpoint tests
+- **Docs/runtime alignment**
+  - existing docs/manual/governance tests where applicable
+  - `uv run mkdocs build --strict`
+- **Repository gates**
+  - `uv run pytest`
+  - `uv run pytest --cov=src/newsdom_api --cov-branch --cov-report=term-missing --cov-fail-under=100`
+- **Targeted smokes**
+  - parser-related smoke for changed delivery paths
+  - local-only private fixture rerun with redacted evidence only
+
+## Decisions
+
+- Prioritize the deterministic `#62` root-cause fix before broader OCR engine or
+  environment benchmarking.
+- Keep runtime failures explicit and sanitized rather than silent or generic.
+- Treat docs/Kubernetes/runtime reconciliation as part of the OCR delivery path,
+  not optional cleanup.
+- Preserve the existing `#61` hierarchy and repair adjacent PR linkage rather
+  than reshaping the issue tree unnecessarily.