Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
c69efca
docs: Add Korean Web Manual and GitHub Pages deployment workflow
seonghobae Apr 8, 2026
cd441ea
docs: Enhance web manual with concrete API schemas, architecture, and…
seonghobae Apr 8, 2026
c24751c
docs: Massive rewrite of web manual to be ultra-specific with exact s…
seonghobae Apr 9, 2026
2c92e6a
test: add enforced quality gate (#2)
seonghobae Apr 9, 2026
d726acd
ci: keep Node24 forcing without tripping scorecard checks (#16)
seonghobae Apr 9, 2026
6721ac4
ci: keep Node24 forcing without tripping scorecard checks (#17)
seonghobae Apr 9, 2026
860e851
ci: scope workflow write permissions to the jobs that need them (#18)
seonghobae Apr 9, 2026
ed8637d
chore(deps): bump the github-actions group with 9 updates (#15)
dependabot[bot] Apr 9, 2026
86187be
chore(release): prepare initial 0.1.0 changelog metadata (#23)
seonghobae Apr 9, 2026
5da1029
chore(deps-dev): bump the python group with 3 updates (#14)
dependabot[bot] Apr 9, 2026
da52912
fix(actions): vendor Pages artifact upload on node24 (#24)
seonghobae Apr 9, 2026
bf430a1
ci: close immediate in-repo OpenSSF Scorecard gaps (#26)
seonghobae Apr 9, 2026
25febfc
ci: ship lean multi-arch images with optional NVIDIA publish (#27)
seonghobae Apr 9, 2026
09d3472
ci: add clusterfuzzlite smoke integration for dom normalization (#28)
seonghobae Apr 9, 2026
a3edbce
chore: pin new Docker and fuzz dependencies by digest (#30)
seonghobae Apr 9, 2026
63a6d27
ci: expand CodeQL coverage and tighten repo guardrails (#35)
seonghobae Apr 11, 2026
cfadf87
release: back-merge v0.1.1 metadata (#48)
seonghobae Apr 11, 2026
b251fdf
feat: Resolve missing Mineru, fix deprecations & K8s compatibility (#56)
seonghobae Apr 21, 2026
27d0c05
fix: remediate CI test failures caused by github-actions bumps (#59)
seonghobae Apr 21, 2026
90d7e98
fix: preserve OCR page-aware structure and baselines (#69)
seonghobae Apr 23, 2026
46bfa28
feat: Add harness for deriving local OCR baselines (#71)
seonghobae Apr 23, 2026
a3595d4
docs: Document local OCR accuracy evidence workflow (#72)
seonghobae Apr 23, 2026
5aec91d
fix: Add robust timeout and error handling to mineru OCR process (#74)
seonghobae Apr 23, 2026
c6beec4
chore(deps): bump pypdf in the uv group across 1 directory (#51)
dependabot[bot] Apr 23, 2026
573d302
fix: Add robust error handling to OCR harness script (#75)
seonghobae Apr 23, 2026
562eae2
ci: Implement Prebuilt Image for stable test pipeline (#83)
seonghobae Apr 24, 2026
0a0aaf1
feat(tools): Implement OCR benchmark harness (#84)
seonghobae Apr 24, 2026
533d7ba
test: Add redacted structural benchmark results artifact (#86)
seonghobae Apr 24, 2026
6da3860
feat: preserve OCR page structure and sanitize parser failures (#65)
seonghobae Apr 24, 2026
84864e9
chore: release v0.2.0
seonghobae Apr 24, 2026
d22a32b
Merge remote-tracking branch 'origin/main' into release/v0.2.0
seonghobae Apr 24, 2026
f981088
Fix tests and lockfile after merge
seonghobae Apr 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/build-ci-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Build CI Environment Image

on:
push:
branches: [main, develop]
paths:
- 'Dockerfile.test'
- '.github/workflows/build-ci-image.yml'
pull_request:
paths:
- 'Dockerfile.test'
- '.github/workflows/build-ci-image.yml'

permissions:
contents: read
packages: write

jobs:
build-and-push:
runs-on: ubuntu-latest
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Log in to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

# Convert repository owner to lowercase for Docker compatibility
- name: Lowercase repository owner
id: lowercase_owner
run: |
echo "owner=${GITHUB_REPOSITORY_OWNER,,}" >> $GITHUB_OUTPUT

- name: Extract metadata for Docker
id: meta
uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051
with:
images: ghcr.io/${{ steps.lowercase_owner.outputs.owner }}/newsdom-api/ci-env
tags: |
type=raw,value=latest
type=sha

- name: Build and push test image
uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25
with:
context: .
file: ./Dockerfile.test
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
2 changes: 2 additions & 0 deletions .github/workflows/dependency-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,5 @@ jobs:

- name: Dependency review
uses: actions/dependency-review-action@2031cfc080254a8a887f58cffee85186f0e49e48
with:
allow-ghsas: "GHSA-69w3-r845-3855"
12 changes: 7 additions & 5 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ on:

permissions:
contents: read
packages: read

jobs:
pytest:
Expand All @@ -18,7 +19,7 @@ jobs:
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd

- name: Setup Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
with:
python-version: '3.10'

Expand All @@ -27,10 +28,11 @@ jobs:
with:
version: '0.11.3'

- name: Install package
run: uv sync --locked --extra dev
- name: Install dependencies
run: uv sync --frozen --all-extras

- name: Run tests with warnings as errors
- name: Run tests with coverage
env:
PYTHONWARNINGS: error
run: uv run pytest
PYTHONPATH: src
run: uv run pytest --cov=src/newsdom_api --cov-branch --cov-report=term-missing --cov-fail-under=100
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ site/
*.pyc
private/
tmp/
site/
registered_agents.json
task_agent_mapping.json
content_file_3982.pdf
page1_300dpi-1.png
.worktrees/
16 changes: 11 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,11 @@ orchestration, MinerU process execution, and DOM normalization.
- `src/newsdom_api/service.py` orchestrates PDF parsing,
temporary files, and response construction.
- `src/newsdom_api/mineru_runner.py` shells out to the MinerU CLI,
collects JSON outputs, and surfaces process errors.
collects JSON outputs, and translates runtime or incomplete-output
failures into typed sanitized exceptions.
- `src/newsdom_api/dom_builder.py` converts MinerU `content_list`
blocks into the canonical NewsDOM response model.
blocks plus page model metadata into the canonical NewsDOM response
model.
- `src/newsdom_api/schemas.py` defines the public response schema.
- `src/newsdom_api/synthetic.py` and
`src/newsdom_api/equivalence.py` support synthetic fixture
Expand All @@ -27,9 +29,13 @@ orchestration, MinerU process execution, and DOM normalization.
2. `src/newsdom_api/service.py` writes the upload to a temporary
workspace and calls MinerU.
3. `src/newsdom_api/mineru_runner.py` resolves the executable, runs
the OCR pipeline, and loads generated JSON artifacts.
4. `src/newsdom_api/dom_builder.py` normalizes OCR blocks into the canonical response.
5. FastAPI returns typed JSON from `src/newsdom_api/schemas.py`.
the OCR pipeline, loads generated JSON artifacts, and raises typed
sanitized errors for runtime-unavailable or incomplete-output cases.
4. `src/newsdom_api/dom_builder.py` normalizes OCR blocks into the
canonical response while preserving page-aware structure from
MinerU model metadata.
5. FastAPI returns typed JSON from `src/newsdom_api/schemas.py` and
maps MinerU runtime failures to 503 and incomplete output to 502.

## Supporting systems

Expand Down
20 changes: 19 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.2.0] - 2026-04-24

### Added

- Added `benchmark_ocr.py` tool to measure OCR engine performance and structural accuracy on private datasets.
- Deployed a GHCR prebuilt CI container image (`ghcr.io/seongho-bae/newsdom-api/ci-env`) to stabilize test environments and resolve timeout/dependency installation issues.

### Changed

- Updated `dom_builder.py` to preserve multi-page MinerU structure instead of collapsing multi-page outputs into a single page.
- Adjusted the `/parse` endpoint to return specific HTTP error codes (`502` and `503`) mapped to `MineruIncompleteOutputError` and `MineruRuntimeUnavailableError` rather than raw `500` errors.

### Fixed

- Mitigated infinite hang issues when processing specific PDFs by enforcing a strict timeout (300 seconds) in the `mineru` subprocess runner.
- Resolved permission (`EACCES`) issues in GitHub Actions by running tests locally instead of inside a non-root container context for GitHub's restricted runner environment.

## [0.1.1] - 2026-04-11

### Added
Expand All @@ -31,6 +48,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Synthetic newspaper fixture generation and structural equivalence checks
- Protected-branch CI, security gates, release provenance workflow, and Git Flow documentation

[Unreleased]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.1...HEAD
[Unreleased]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.2.0...HEAD
[0.2.0]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.1...v0.2.0
[0.1.1]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/Seongho-Bae/newsdom-api/releases/tag/v0.1.0
17 changes: 10 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,16 @@ ENV PYTHONDONTWRITEBYTECODE=1 \

WORKDIR /app

RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates && \
rm -rf /var/lib/apt/lists/*

COPY --from=uv-bin /uv /uvx /bin/

COPY pyproject.toml uv.lock README.md ./
COPY src/ src/

RUN uv sync --frozen --no-dev
RUN uv sync --frozen --no-dev --extra mineru

FROM ${PYTHON_BASE} AS runtime

Expand All @@ -27,17 +31,16 @@ ENV PYTHONDONTWRITEBYTECODE=1 \

WORKDIR /app

RUN useradd --create-home --home-dir /home/newsdom --shell /usr/sbin/nologin newsdom
RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates && \
rm -rf /var/lib/apt/lists/*

COPY --from=builder /app /app
RUN useradd --create-home --home-dir /home/newsdom --shell /usr/sbin/nologin newsdom

RUN chown -R newsdom:newsdom /app
COPY --from=builder --chown=newsdom:newsdom /app /app

USER newsdom

EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=5 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health').read()"

CMD ["uvicorn", "newsdom_api.main:app", "--host", "0.0.0.0", "--port", "8000"]
36 changes: 36 additions & 0 deletions Dockerfile.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
FROM python:3.10-slim

ENV UV_VERSION=0.11.3

# Install system dependencies
# Note: We do NOT switch to a non-root user (like `USER ciuser`) here.
# GitHub Actions runs `container` jobs by bind-mounting workspace and runner temp directories.
# If the container runs as a non-root user by default, it will not have permission to write
# to the GitHub Actions runner directories (like /__w/_temp/_runner_file_commands/), causing EACCES errors.
# @coderabbitai ignore
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
libgl1 \
libglib2.0-0 \
curl \
git \
&& rm -rf /var/lib/apt/lists/*

# Install uv (pin to match .github/workflows/tests.yml)
RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh

ENV PATH="/usr/local/bin:${PATH}"

WORKDIR /app

# Pre-warm dependencies only (source is mounted at runtime by Actions)
COPY pyproject.toml README.md ./
COPY src/newsdom_api/__init__.py ./src/newsdom_api/__init__.py

# Install dependencies using uv
# Install to system Python so that it's accessible without activating a venv
RUN uv pip install --system -e ".[dev,mineru]"

ENV PYTHONPATH=/app/src

CMD ["pytest"]
19 changes: 12 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,13 @@ docker build -t newsdom-api .
docker run -p 8000:8000 newsdom-api
```

The default image exposes the REST API on port `8000` as a lean multi-arch
service image. It is suitable for `linux/amd64` and `linux/arm64`, including
Apple Silicon hosts running the API service inside Docker.
The default image exposes the REST API on port `8000` as a multi-arch service
image. It is suitable for `linux/amd64` and `linux/arm64`, including Apple
Silicon hosts running the API service inside Docker.

The lean image is intentionally a REST API shell: `/health`, `/docs`, and
OpenAPI endpoints are available immediately, while real `/parse` execution
still requires a compatible MinerU runtime to be available inside the
container image.
The default image already includes the MinerU runtime and sets
`NEWSDOM_MINERU_BIN=mineru`, so `/parse` is available without layering an extra
OCR package into the container at runtime.

For heavier parsing deployments, build the optional NVIDIA-oriented variant:

Expand All @@ -75,6 +74,12 @@ provide the CUDA user-space/runtime stack required by MinerU.
curl -F "file=@sample.pdf" http://127.0.0.1:8000/parse
```

Each request is written to a request-scoped temporary directory before MinerU
runs, and those temporary files are removed after the response completes.
Sanitized parse failures return `503 MinerU runtime unavailable` when the
runtime cannot be executed and `502 MinerU output was incomplete` when MinerU
finishes without the required output artifacts.

### Run tests

```bash
Expand Down
18 changes: 15 additions & 3 deletions docs/operations/deploy-runbook.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Deploy runbook

This repository does not ship a long-lived production cluster
definition in-tree, so deployment verification focuses on reproducible
local and GitHub-hosted delivery paths.
definition in-tree and has no in-tree Kubernetes manifests, so
deployment verification focuses on reproducible local and GitHub-hosted
delivery paths.

## Local API smoke

Expand All @@ -17,7 +18,15 @@ local and GitHub-hosted delivery paths.

1. `docker build -t newsdom-api .`
2. `docker run --rm -p 18080:8000 newsdom-api`
3. `curl -fsS http://127.0.0.1:18080/health`
3. Verify:
- `curl -fsS http://127.0.0.1:18080/health`
- `curl -fsS -F "file=@sample.pdf" http://127.0.0.1:18080/parse`

The default image already includes the MinerU runtime and sets
`NEWSDOM_MINERU_BIN=mineru`, so container smoke should treat `/parse`
as part of the supported contract instead of a docs-only shell.

`/health` proves the API process is serving but does not validate a full `/parse` round-trip, MinerU execution, or OCR artifact production.

## Release smoke

Expand All @@ -30,5 +39,8 @@ local and GitHub-hosted delivery paths.
## Failure handling

- Capture sanitized logs outside `tmp/` when a delivery path fails.
- Expect `/parse` failures to stay sanitized: `503 MinerU runtime
unavailable` when the runtime cannot execute, and `502 MinerU output
was incomplete` when required OCR artifacts are missing or invalid.
- Reconcile the failure against `README.md`, `CHANGELOG.md`, and the
relevant workflow before closing the task.
Loading
Loading