Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
c69efca
docs: Add Korean Web Manual and GitHub Pages deployment workflow
seonghobae Apr 8, 2026
cd441ea
docs: Enhance web manual with concrete API schemas, architecture, and…
seonghobae Apr 8, 2026
c24751c
docs: Massive rewrite of web manual to be ultra-specific with exact s…
seonghobae Apr 9, 2026
2c92e6a
test: add enforced quality gate (#2)
seonghobae Apr 9, 2026
d726acd
ci: keep Node24 forcing without tripping scorecard checks (#16)
seonghobae Apr 9, 2026
6721ac4
ci: keep Node24 forcing without tripping scorecard checks (#17)
seonghobae Apr 9, 2026
860e851
ci: scope workflow write permissions to the jobs that need them (#18)
seonghobae Apr 9, 2026
ed8637d
chore(deps): bump the github-actions group with 9 updates (#15)
dependabot[bot] Apr 9, 2026
86187be
chore(release): prepare initial 0.1.0 changelog metadata (#23)
seonghobae Apr 9, 2026
5da1029
chore(deps-dev): bump the python group with 3 updates (#14)
dependabot[bot] Apr 9, 2026
da52912
fix(actions): vendor Pages artifact upload on node24 (#24)
seonghobae Apr 9, 2026
bf430a1
ci: close immediate in-repo OpenSSF Scorecard gaps (#26)
seonghobae Apr 9, 2026
25febfc
ci: ship lean multi-arch images with optional NVIDIA publish (#27)
seonghobae Apr 9, 2026
09d3472
ci: add clusterfuzzlite smoke integration for dom normalization (#28)
seonghobae Apr 9, 2026
a3edbce
chore: pin new Docker and fuzz dependencies by digest (#30)
seonghobae Apr 9, 2026
63a6d27
ci: expand CodeQL coverage and tighten repo guardrails (#35)
seonghobae Apr 11, 2026
287a389
docs: record v0.1.1 release design
seonghobae Apr 11, 2026
c549219
docs: record v0.1.1 release plan
seonghobae Apr 11, 2026
5f72104
test: add failing v0.1.1 release metadata checks
seonghobae Apr 11, 2026
20b207a
chore(release): prepare v0.1.1 metadata
seonghobae Apr 11, 2026
2160f92
test: keep release metadata lockstep
seonghobae Apr 11, 2026
39684d4
Merge remote-tracking branch 'origin/main' into release/v0.1.1
seonghobae Apr 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,8 @@ site/
*.pyc
private/
tmp/
site/
registered_agents.json
task_agent_mapping.json
content_file_3982.pdf
page1_300dpi-1.png
10 changes: 10 additions & 0 deletions .markdownlint-cli2.jsonc
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"ignores": [
"docs/plans/2026-04-08-git-flow-design.md",
"docs/plans/2026-04-08-git-flow.md",
"docs/plans/2026-04-08-newsdom-design.md",
"docs/plans/2026-04-08-newsdom-implementation.md",
"docs/plans/2026-04-08-quality-gate-design.md",
"docs/plans/2026-04-08-quality-gate.md"
]
}
57 changes: 57 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# AGENTS.md

## Project overview

- Repository: `newsdom-api`
- Product: FastAPI service that converts MinerU OCR output into
canonical NewsDOM JSON.
- Primary branch model: manual Git Flow (`develop` integration,
`main` stable).

## Authoritative docs

Read these first when making repository changes:

- `docs/engineering/canonical-docs.md`
- `docs/engineering/execution-policy.md`
- `docs/engineering/acceptance-criteria.md`
- `docs/engineering/harness-engineering.md`
- `docs/engineering/review-policy.md`
- `docs/engineering/runtime-data-policy.md`
- `docs/engineering/skills-subagents-mcp.md`
- `docs/workflow/git-flow.md`
- `docs/workflow/pr-continuity.md`
- `docs/workflow/one-day-delivery-plan.md`
- `docs/operations/deploy-runbook.md`
- `docs/security/api-security-checklist.md`
- `docs/coderabbit/review-commands.md`
- `ARCHITECTURE.md`

## Setup and verification defaults

- Install: `uv sync --frozen --all-extras`
- Test: `uv run pytest`
- Coverage gate:
`uv run pytest --cov=src/newsdom_api --cov-branch --cov-report=term-missing --cov-fail-under=100`
- Docs build: `uv run mkdocs build --strict`
- Local API: `uv run uvicorn --app-dir src newsdom_api.main:app --reload`

## Delivery defaults

- Branch normal work from `develop` unless the task is a `main`-only
release or hotfix path.
- Keep PR continuity explicit with `gh pr view` / `gh pr list` /
`pr_continuity` before opening duplicates.
- Treat CodeRabbit as advisory automation; required human approvals
still follow the repository ruleset.
- When PRs are blocked externally, continue local adjacent tasks
instead of stopping.

## Safety rules

- Keep synthetic fixtures public and private reference inputs
local-only.
- Do not commit secrets, credentials, or copyrighted source
newspaper material.
- Prefer durable evidence in tracked docs, tests, workflow runs,
PR comments, and release assets over scratch notes.
49 changes: 49 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Architecture

## Runtime shape

`newsdom-api` is a small service-oriented Python application with a
thin FastAPI entrypoint and explicit separation between request
orchestration, MinerU process execution, and DOM normalization.

## Primary modules

- `src/newsdom_api/main.py` exposes `/health` and `/parse`
through FastAPI.
- `src/newsdom_api/service.py` orchestrates PDF parsing,
temporary files, and response construction.
- `src/newsdom_api/mineru_runner.py` shells out to the MinerU CLI,
collects JSON outputs, and surfaces process errors.
- `src/newsdom_api/dom_builder.py` converts MinerU `content_list`
blocks into the canonical NewsDOM response model.
- `src/newsdom_api/schemas.py` defines the public response schema.
- `src/newsdom_api/synthetic.py` and
`src/newsdom_api/equivalence.py` support synthetic fixture
generation and structural comparisons.

## Request flow

1. `src/newsdom_api/main.py` receives an uploaded PDF.
2. `src/newsdom_api/service.py` writes the upload to a temporary
workspace and calls MinerU.
3. `src/newsdom_api/mineru_runner.py` resolves the executable, runs
the OCR pipeline, and loads generated JSON artifacts.
4. `src/newsdom_api/dom_builder.py` normalizes OCR blocks into the canonical response.
5. FastAPI returns typed JSON from `src/newsdom_api/schemas.py`.

## Supporting systems

- `tests/fixtures` holds synthetic PDFs, JSON baselines, and
provenance notes; private reference inputs stay out of git.
- `manual/` is the published user manual rendered by MkDocs.
- `.github/workflows/` encodes CI, security scanning, Pages,
release, and image-delivery policy.
- `scripts/release/` builds release manifests and exports GitHub attestation bundles.

## Delivery boundaries

- `develop` is the integration line for normal feature, fix,
and chore work.
- `main` is the stable release line that receives tagged releases.
- The service is production-grade only when code, docs, workflows,
and release evidence agree.
17 changes: 12 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.1.1] - 2026-04-11

### Added

- Lean multi-arch container delivery workflow with an optional NVIDIA image publishing path
- ClusterFuzzLite DOM-normalization fuzzing harness with a locked fuzz toolchain
- GHCR-ready multi-arch API image delivery, ClusterFuzzLite coverage, and exported `*.intoto.jsonl` provenance bundles for stable releases
- Verified `/docs` and `/redoc` manual screenshots plus canonical engineering policy docs that describe the live repository workflow

### Changed

- Release automation now exports `*.intoto.jsonl` provenance bundles alongside build artifacts
- Tests now pin `uv` setup more tightly, and Pages rebuilds docs when lockfile or local Pages action inputs change
- Protected-branch governance documentation now reflects the current single-maintainer exception while preserving required checks and history protections
- Public setup guidance, docs-toolchain policy, and markdownlint scope now match the merged `develop` / `main` delivery paths

### Fixed

- Patched `pypdf` lockfile coverage to `6.10.0` for GHSA-3crg-w4f6-42mx / CVE-2026-40260

## [0.1.0] - 2026-04-09

Expand All @@ -25,5 +31,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Synthetic newspaper fixture generation and structural equivalence checks
- Protected-branch CI, security gates, release provenance workflow, and Git Flow documentation

[Unreleased]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.0...HEAD
[Unreleased]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.1...HEAD
[0.1.1]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/Seongho-Bae/newsdom-api/releases/tag/v0.1.0
73 changes: 59 additions & 14 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,46 +2,76 @@

## Development setup

Install `uv` first if it is not already available in your `PATH`, then sync the
repository-managed virtual environment:

```bash
python3.10 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
uv sync --frozen --all-extras
```

Install the parser stack only when you need live MinerU execution:

```bash
pip install "mineru[pipeline]==3.0.9"
uv pip install --python .venv/bin/python "mineru[pipeline]==3.0.9"
```

On Windows, replace `.venv/bin/python` with `.venv\Scripts\python.exe`.

## Test commands

```bash
pytest
PYTHONWARNINGS=error pytest
pytest --cov=src/newsdom_api --cov-branch --cov-report=term-missing --cov-fail-under=100
uv run pytest
PYTHONWARNINGS=error uv run pytest
uv run pytest --cov=src/newsdom_api --cov-branch --cov-report=term-missing --cov-fail-under=100
```

CI installs dependencies from `uv.lock`, and workflow actions are pinned by immutable commit SHA. Keep both policies intact when editing `.github/` automation.
CI installs dependencies from `uv.lock`, and workflow actions are
pinned by immutable commit SHA. Keep both policies intact when editing
`.github/` automation.

CircleCI parity is defined in `.circleci/config.yml` and mirrors the same uv-locked warnings-as-errors and 100% coverage quality gate.
CircleCI parity is defined in `.circleci/config.yml` and mirrors the
same uv-locked warnings-as-errors and 100% coverage quality gate.

## Documentation build

The GitHub Pages workflow installs documentation tooling from `uv.lock` via the optional `docs` extra. For local maintainer work, sync all extras so the docs build does not drop the test toolchain from the active environment.
The GitHub Pages workflow installs documentation tooling from
`uv.lock` via the optional `docs` extra. For local maintainer work,
sync all extras so the docs build does not drop the test toolchain
from the active environment.

The supported docs toolchain stays on the MkDocs 1.x line for now.
Keep `mkdocs<2.0` and `mkdocs-material<9.7` in place until the
upstream Material team publishes a workable migration path or this
repository validates a replacement docs stack. `uv.lock` is the source
of truth for the currently supported docs build.

```bash
uv sync --frozen --all-extras
uv run mkdocs build --strict
```

Tagged releases use `.github/workflows/release.yml` to build artifacts, generate SHA256 checksums, emit a JSON manifest, export `*.intoto.jsonl` provenance bundles, and publish a GitHub Release with provenance attestation.
Use markdownlint for `AGENTS.md`, `ARCHITECTURE.md`, `CONTRIBUTING.md`,
and `docs/**/*.md`. The sweep excludes only the legacy
`git-flow-design`, `git-flow`, `newsdom-design`,
`newsdom-implementation`, `quality-gate-design`, and `quality-gate`
planning notes that predate the current markdown style policy.

Tagged releases use `.github/workflows/release.yml` to build
artifacts, generate SHA256 checksums, emit a JSON manifest, export
`*.intoto.jsonl` provenance bundles, and publish a GitHub Release with
provenance attestation.

For full OpenSSF Scorecard branch-protection visibility against classic GitHub branch protection rules, set a repository secret named `SCORECARD_TOKEN` with the fine-grained administration-read scope recommended by the Scorecard Action documentation. Without that secret, Scorecard still runs but may report the Branch-Protection check as inconclusive.
For full OpenSSF Scorecard branch-protection visibility against
classic GitHub branch protection rules, set a repository secret named
`SCORECARD_TOKEN` with the fine-grained administration-read scope
recommended by the Scorecard Action documentation. Without that
secret, Scorecard still runs but may report the Branch-Protection
check as inconclusive.

## Fixture policy

This project intentionally separates public test artifacts from private validation material.
This project intentionally separates public test artifacts from
private validation material.

Allowed in the repository:

Expand All @@ -57,7 +87,8 @@ Not allowed in the repository:

## Private baseline refresh

If you maintain a private reference page locally, refresh only the derived structural baseline:
If you maintain a private reference page locally, refresh only the
derived structural baseline:

```bash
python tools/derive_private_baseline.py tests/fixtures/private_page_baseline.json
Expand All @@ -67,10 +98,24 @@ The source page must remain local.

## Documentation split

- `AGENTS.md`: repository-local execution bootstrap for autonomous
maintenance
- `ARCHITECTURE.md`: runtime structure and module boundaries
- `README.md`: user-facing overview and quickstart
- `CONTRIBUTING.md`: maintainer workflow and safety rules
- `SECURITY.md`: vulnerability reporting and supported-branch policy
- `manual/`: published end-user manual rendered by MkDocs
- `docs/agents/README.md`: agent-specific read order and
repository-local behavior notes
- `docs/coderabbit/review-commands.md`: supported CodeRabbit review
commands used in PRs
- `docs/engineering/`: canonical maintainer policies and acceptance criteria
- `docs/operations/`: delivery and verification runbooks
- `docs/security/api-security-checklist.md`: API hardening checklist
for FastAPI surface changes
- `docs/workflow/git-flow.md`: canonical branch workflow
- `docs/workflow/pr-continuity.md`: canonical PR-selection and
stacked-PR guidance
- `tests/fixtures/README.md`: fixture provenance and regeneration notes
- `docs/plans/`: design and implementation planning notes

Expand Down
54 changes: 38 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,28 +8,33 @@ NewsDOM API parses scanned Japanese newspaper PDFs into DOM-like article trees.

- Primary engine: `MinerU` pipeline backend
- Service wrapper: FastAPI
- Output: canonical JSON with pages, articles, headlines, body blocks, images, captions, and quality metadata
- Output: canonical JSON with pages, articles, headlines, body
blocks, images, captions, and quality metadata

## Quickstart

### Install

Install `uv` first if it is not already available in your `PATH`, then sync the
repository-managed virtual environment:

```bash
python3.10 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
uv sync --frozen --all-extras
```

To enable real parsing with MinerU, install the MinerU CLI separately in the environment that will execute parsing:
To enable real parsing with MinerU, install the MinerU CLI separately in the
same `.venv` that `uv sync` created:

```bash
pip install "mineru[pipeline]==3.0.9"
uv pip install --python .venv/bin/python "mineru[pipeline]==3.0.9"
```

On Windows, replace `.venv/bin/python` with `.venv\Scripts\python.exe`.

### Run

```bash
uvicorn newsdom_api.main:app --reload
uv run uvicorn --app-dir src newsdom_api.main:app --reload
```

### Docker
Expand All @@ -39,9 +44,14 @@ docker build -t newsdom-api .
docker run -p 8000:8000 newsdom-api
```

The default image exposes the REST API on port `8000` as a lean multi-arch service image. It is suitable for `linux/amd64` and `linux/arm64`, including Apple Silicon hosts running the API service inside Docker.
The default image exposes the REST API on port `8000` as a lean multi-arch
service image. It is suitable for `linux/amd64` and `linux/arm64`, including
Apple Silicon hosts running the API service inside Docker.

The lean image is intentionally a REST API shell: `/health`, `/docs`, and OpenAPI endpoints are available immediately, while real `/parse` execution still requires a compatible MinerU runtime to be available inside the container image.
The lean image is intentionally a REST API shell: `/health`, `/docs`, and
OpenAPI endpoints are available immediately, while real `/parse` execution
still requires a compatible MinerU runtime to be available inside the
container image.

For heavier parsing deployments, build the optional NVIDIA-oriented variant:

Expand All @@ -50,7 +60,14 @@ docker build -f Dockerfile.nvidia -t newsdom-api:nvidia .
docker run --gpus all -p 8000:8000 newsdom-api:nvidia
```

`Dockerfile.nvidia` is intended for Linux/NVIDIA environments and is `linux/amd64`-only for hosts that can provide the CUDA user-space/runtime stack required by MinerU. Apple Silicon can run the lean API image, but Docker Desktop does not expose Apple GPU acceleration to Linux containers, so real GPU-accelerated parsing should stay on a native Apple Silicon path instead of the containerized runtime.
`Dockerfile.nvidia` is intended for Linux/NVIDIA environments and is
`linux/amd64`-only. Apple Silicon can run the lean API image, but Docker
Desktop does not expose Apple GPU acceleration to Linux containers, so real
GPU-accelerated parsing should stay on a native Apple Silicon path instead of
the containerized runtime.

The NVIDIA variant is `linux/amd64`-only and is meant for hosts that can
provide the CUDA user-space/runtime stack required by MinerU.

### Parse a PDF

Expand All @@ -61,27 +78,32 @@ curl -F "file=@sample.pdf" http://127.0.0.1:8000/parse
### Run tests

```bash
pytest
uv run pytest
```

### Fuzzing smoke

```bash
./.venv/bin/python fuzzers/dom_builder_fuzzer.py --smoke tests/fixtures/mineru_sample.json
uv run python fuzzers/dom_builder_fuzzer.py --smoke tests/fixtures/mineru_sample.json
```

The repository also enforces a `quality-gate` workflow with 100% source coverage and docstring audit coverage.
The repository also enforces a `quality-gate` workflow with 100% source
coverage and docstring audit coverage.

## Fixtures and provenance

This repository ships only synthetic test fixtures and derived structural baselines. For fixture provenance and regeneration notes, see `tests/fixtures/README.md`.
This repository ships only synthetic test fixtures and derived structural
baselines. For fixture provenance and regeneration notes, see
`tests/fixtures/README.md`.

## Development

Development setup, fixture handling rules, and local-only baseline maintenance are documented in `CONTRIBUTING.md`.
Development setup, fixture handling rules, and local-only baseline
maintenance are documented in `CONTRIBUTING.md`.

Security reporting guidance is documented in `SECURITY.md`.
Version tags trigger a GitHub-native release workflow that builds distribution artifacts, checksums, and provenance attestations.
Version tags trigger a GitHub-native release workflow that builds
distribution artifacts, checksums, and provenance attestations.

Project history is tracked in `CHANGELOG.md`.

Expand Down
Loading
Loading