Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a6c2db5
ci: expand CodeQL coverage and tighten repo guardrails
seonghobae Apr 10, 2026
7945aec
fix: unblock fuzz CI and harden governance tests
seonghobae Apr 10, 2026
bb1cb40
fix: forward libFuzzer flags so ClusterFuzzLite fuzz jobs run
seonghobae Apr 10, 2026
551b374
docs: align repository truth sources with current workflow state
seonghobae Apr 10, 2026
a9d0ebb
Merge pull request #37 from Seongho-Bae/chore/truth-source-alignment
seonghobae Apr 10, 2026
1d263a8
docs: add canonical engineering truth sources
seonghobae Apr 10, 2026
3e7ae79
fix: harden workflow attestation and fuzz builder paths
seonghobae Apr 10, 2026
70383e4
Merge pull request #38 from Seongho-Bae/chore/canonical-engineering-docs
seonghobae Apr 10, 2026
737696b
Merge pull request #39 from Seongho-Bae/chore/workflow-robustness-har…
seonghobae Apr 10, 2026
7fa1b7f
docs: scope markdownlint around active repository docs
seonghobae Apr 10, 2026
1af74b0
Merge pull request #42 from Seongho-Bae/chore/markdownlint-scope-policy
seonghobae Apr 10, 2026
a0e3ac9
docs: pin the supported MkDocs toolchain stance
seonghobae Apr 10, 2026
83c84a1
Merge pull request #43 from Seongho-Bae/chore/docs-toolchain-hold
seonghobae Apr 10, 2026
b3d84ae
docs: align public setup guidance with uv defaults
seonghobae Apr 10, 2026
ae12473
Merge pull request #44 from Seongho-Bae/chore/uv-first-public-setup
seonghobae Apr 10, 2026
06ba722
fix: lock pypdf to patched release
seonghobae Apr 11, 2026
fbb13ef
Merge pull request #46 from Seongho-Bae/fix/pypdf-6-10-lock
seonghobae Apr 11, 2026
bbf0dbc
docs: record reviewer-capacity ruleset alignment plan
seonghobae Apr 11, 2026
c042621
docs: align governance truth with single-maintainer exception
seonghobae Apr 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
* @Seongho-Bae

# Security and workflow ownership
.github/ @Seongho-Bae
docs/ @Seongho-Bae
manual/ @Seongho-Bae
4 changes: 2 additions & 2 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ permissions:

jobs:
analyze:
name: codeql (python)
name: codeql (python, actions)
runs-on: ubuntu-latest
permissions:
contents: read
Expand All @@ -27,7 +27,7 @@ jobs:
- name: Initialize CodeQL
uses: github/codeql-action/init@c10b8064de6f491fea524254123dbe5e09572f13
with:
languages: python
languages: python, actions

- name: Autobuild
uses: github/codeql-action/autobuild@c10b8064de6f491fea524254123dbe5e09572f13
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,8 @@ build/
*.pyc
private/
tmp/
site/
registered_agents.json
task_agent_mapping.json
content_file_3982.pdf
page1_300dpi-1.png
39 changes: 28 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ NewsDOM API parses scanned Japanese newspaper PDFs into DOM-like article trees.

- Primary engine: `MinerU` pipeline backend
- Service wrapper: FastAPI
- Output: canonical JSON with pages, articles, headlines, body blocks, images, captions, and quality metadata
- Output: canonical JSON with pages, articles, headlines, body
blocks, images, captions, and quality metadata

## Quickstart

Expand All @@ -20,7 +21,8 @@ source .venv/bin/activate
pip install -e ".[dev]"
```

To enable real parsing with MinerU, install the MinerU CLI separately in the environment that will execute parsing:
To enable real parsing with MinerU, install the MinerU CLI separately in the
environment that will execute parsing:

```bash
pip install "mineru[pipeline]==3.0.9"
Expand All @@ -29,7 +31,7 @@ pip install "mineru[pipeline]==3.0.9"
### Run

```bash
uvicorn newsdom_api.main:app --reload
uvicorn --app-dir src newsdom_api.main:app --reload
```

### Docker
Expand All @@ -39,9 +41,14 @@ docker build -t newsdom-api .
docker run -p 8000:8000 newsdom-api
```

The default image exposes the REST API on port `8000` as a lean multi-arch service image. It is suitable for `linux/amd64` and `linux/arm64`, including Apple Silicon hosts running the API service inside Docker.
The default image exposes the REST API on port `8000` as a lean multi-arch
service image. It is suitable for `linux/amd64` and `linux/arm64`, including
Apple Silicon hosts running the API service inside Docker.

The lean image is intentionally a REST API shell: `/health`, `/docs`, and OpenAPI endpoints are available immediately, while real `/parse` execution still requires a compatible MinerU runtime to be available inside the container image.
The lean image is intentionally a REST API shell: `/health`, `/docs`, and
OpenAPI endpoints are available immediately, while real `/parse` execution
still requires a compatible MinerU runtime to be available inside the
container image.

For heavier parsing deployments, build the optional NVIDIA-oriented variant:

Expand All @@ -50,9 +57,14 @@ docker build -f Dockerfile.nvidia -t newsdom-api:nvidia .
docker run --gpus all -p 8000:8000 newsdom-api:nvidia
```

`Dockerfile.nvidia` is intended for Linux/NVIDIA environments and is `linux/amd64`-only. Apple Silicon can run the lean API image, but Docker Desktop does not expose Apple GPU acceleration to Linux containers, so real GPU-accelerated parsing should stay on a native Apple Silicon path instead of the containerized runtime.
`Dockerfile.nvidia` is intended for Linux/NVIDIA environments and is
`linux/amd64`-only. Apple Silicon can run the lean API image, but Docker
Desktop does not expose Apple GPU acceleration to Linux containers, so real
GPU-accelerated parsing should stay on a native Apple Silicon path instead of
the containerized runtime.

The NVIDIA variant is `linux/amd64`-only and is meant for hosts that can provide the CUDA user-space/runtime stack required by MinerU.
The NVIDIA variant is `linux/amd64`-only and is meant for hosts that can
provide the CUDA user-space/runtime stack required by MinerU.

### Parse a PDF

Expand All @@ -72,18 +84,23 @@ pytest
python fuzzers/dom_builder_fuzzer.py --smoke tests/fixtures/mineru_sample.json
```

The repository also enforces a `quality-gate` workflow with 100% source coverage and docstring audit coverage.
The repository also enforces a `quality-gate` workflow with 100% source
coverage and docstring audit coverage.

## Fixtures and provenance

This repository ships only synthetic test fixtures and derived structural baselines. For fixture provenance and regeneration notes, see `tests/fixtures/README.md`.
This repository ships only synthetic test fixtures and derived structural
baselines. For fixture provenance and regeneration notes, see
`tests/fixtures/README.md`.

## Development

Development setup, fixture handling rules, and local-only baseline maintenance are documented in `CONTRIBUTING.md`.
Development setup, fixture handling rules, and local-only baseline
maintenance are documented in `CONTRIBUTING.md`.

Security reporting guidance is documented in `SECURITY.md`.
Version tags trigger a GitHub-native release workflow that builds distribution artifacts, checksums, and provenance attestations.
Version tags trigger a GitHub-native release workflow that builds
distribution artifacts, checksums, and provenance attestations.

Project history is tracked in `CHANGELOG.md`.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Code Scanning Hardening and Manual Screenshots Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to
> implement this plan task-by-task.

**Goal:** Close the highest-priority executable security and code-scanning
gaps by hardening repository review protections and CodeQL coverage, then
update the Korean manual with verified screenshots.

**Architecture:** Add repository-owned governance artifacts (`CODEOWNERS`)
and expand workflow coverage so GitHub code scanning evaluates both Python
and GitHub Actions. Keep verification split between local TDD for repo files
and live GitHub API checks for ruleset state, then regenerate manual
screenshots from localhost-served pages so docs reflect the current product
surface.

**Tech Stack:** Python, pytest, GitHub Actions, GitHub REST API (`gh api`),
MkDocs Material, FastAPI, Playwright

---

## Task 1: Add failing tests for security workflow and manual screenshot expectations

**Files:**

- Create: `tests/test_repository_governance.py`
- Keep: `tests/test_readme.py` unchanged

### Step 1: Write the failing test

Add tests that assert:

- `.github/CODEOWNERS` exists and references the maintainer path ownership.
- `.github/workflows/codeql.yml` includes `actions` in the configured
languages.
- `manual/api-reference.md` references screenshot assets.
- `manual/development.md` documents enforced review gates.
- `.gitignore` excludes generated runtime artifacts.

### Step 2: Run test to verify it fails

Run: `uv run pytest tests/test_repository_governance.py -q`

Expected: FAIL because `CODEOWNERS` is missing, CodeQL scans only `python`,
screenshot references do not exist, review-gate docs are incomplete, and
generated runtime artifacts are not ignored.

### Step 3: Write minimal implementation

Create `CODEOWNERS`, update `codeql.yml`, and add screenshot references to
the manual with matching assets.

### Step 4: Run test to verify it passes

Run: `uv run pytest tests/test_repository_governance.py -q`

Expected: PASS

### Step 5: Commit

```bash
git add tests/test_repository_governance.py tests/test_readme.py \
.github/CODEOWNERS .github/workflows/codeql.yml \
manual/api-reference.md manual/assets/
git commit -m "security: harden code scanning and document UI evidence"
```

## Task 2: Generate verified manual screenshots from localhost services

**Files:**

- Create: `manual/assets/swagger-ui.png`
- Create: `manual/assets/redoc.png`
- Modify: `manual/api-reference.md`

### Task 2 Step 1: Write the failing test

Extend the governance and manual test to assert the screenshot files exist,
carry real PNG signatures, and are referenced from the API manual.

### Task 2 Step 2: Run test to verify it fails

Run:
`uv run pytest tests/test_repository_governance.py::`
`test_api_manual_references_screenshot_assets -q`

Expected: FAIL because the screenshot files do not exist yet.

### Task 2 Step 3: Write minimal implementation

Start the FastAPI app locally, open `/docs` and `/redoc` with Playwright, and
capture screenshots into `manual/assets/`, then add Markdown image embeds plus
explanatory captions in `manual/api-reference.md`.

### Task 2 Step 4: Run test to verify it passes

Run:
`uv run pytest tests/test_repository_governance.py::`
`test_api_manual_references_screenshot_assets -q`

Expected: PASS

### Task 2 Step 5: Commit

```bash
git add manual/api-reference.md manual/assets/swagger-ui.png \
manual/assets/redoc.png tests/test_repository_governance.py
git commit -m "docs: add verified API console screenshots"
```

## Task 3: Verify live GitHub ruleset posture and align remediation evidence

**Files:**

- Modify: `.github/CODEOWNERS`
- Modify: `.github/workflows/codeql.yml`
- Modify: `manual/development.md`

### Task 3 Step 1: Write the failing verification target

Record the desired live conditions:

- active ruleset requires 2 approving reviews
- code owner review is required
- last-push approval remains required

### Task 3 Step 2: Run verification to show current gap

Run: `gh api repos/Seongho-Bae/newsdom-api/rulesets/14875805`

Expected: JSON shows `required_approving_review_count` is `1` and
`require_code_owner_review` is `false`.

### Task 3 Step 3: Write minimal implementation

Patch the repository ruleset via `gh api --method PUT ...` so it requires 2
approvals and code owner review, then document the strengthened governance
path in `manual/development.md`.

### Task 3 Step 4: Run verification to prove it passes

Run: `gh api repos/Seongho-Bae/newsdom-api/rulesets/14875805`

Expected: JSON shows `required_approving_review_count` is `2`,
`require_code_owner_review` is `true`, and `require_last_push_approval`
remains `true`.

### Task 3 Step 5: Commit

```bash
git add .github/CODEOWNERS .github/workflows/codeql.yml manual/development.md
git commit -m "security: strengthen review gates for protected branches"
```

## Task 4: Full verification and delivery evidence

**Files:**

- Modify: `README.md`
- Modify: `manual/index.md`

### Task 4 Step 1: Run repository verification

Run:

- `uv sync --extra dev`
- `uv run pytest -q`
- `python3 scripts/prompt_checks/validate_canonical_doc_refs.py --root .`
(skip if the script is not present in this repository)

Expected: all repo checks required for this task pass.

### Task 4 Step 2: Run live and manual verification

Run:

- `gh api repos/Seongho-Bae/newsdom-api/code-scanning/alerts?state=open&per_page=100`
- review the latest Scorecard and CodeQL posture after the push-triggered
workflow

Expected: branch protection alert evidence reflects the tightened ruleset, and
new workflow coverage is present for future scans.

### Task 4 Step 3: Update docs if verification reveals drift

Refresh manual and README wording so documentation matches the verified
workflow and review posture.

### Task 4 Step 4: Commit

```bash
git add README.md manual/index.md
git commit -m "docs: align manual with security verification evidence"
```
6 changes: 5 additions & 1 deletion fuzzers/dom_builder_fuzzer.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,11 @@
from pathlib import Path
from typing import Any

from newsdom_api.dom_builder import build_dom
SRC_ROOT = Path(__file__).resolve().parents[1] / "src"
if str(SRC_ROOT) not in sys.path:
sys.path.insert(0, str(SRC_ROOT))

from newsdom_api.dom_builder import build_dom # noqa: E402


def _coerce_content_list(candidate: Any) -> list[dict[str, Any]]:
Expand Down
34 changes: 28 additions & 6 deletions manual/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ FastAPI 서버를 실행하고, 스캔된 신문 PDF를 업로드하여 기사 D

```bash
# 개발 시 핫-리로딩(--reload) 모드 사용
uvicorn newsdom_api.main:app --reload
uvicorn --app-dir src newsdom_api.main:app --reload
```

## 2. 인터랙티브 웹 UI
Expand All @@ -19,6 +19,18 @@ FastAPI는 OpenAPI 기반의 대화형 API 문서를 자동으로 생성합니
- **Swagger UI**: `http://127.0.0.1:8000/docs`
- **ReDoc**: `http://127.0.0.1:8000/redoc`

실제 로컬 서버에서 검증한 UI 캡처는 아래와 같습니다.
스크린샷 파일은 `manual/assets/swagger-ui.png`,
`manual/assets/redoc.png` 경로에 보관합니다.
이 이미지는 `uv run uvicorn --app-dir src newsdom_api.main:app --host
127.0.0.1 --port 8000` 명령으로 서버를 띄운 뒤 `/docs`와 `/redoc`를
실제로 캡처한 결과입니다. 일반 실행은 활성화된 가상환경에서
`uvicorn --app-dir src newsdom_api.main:app --reload`를 사용하면 됩니다.

![Swagger UI 화면](assets/swagger-ui.png)

![ReDoc 화면](assets/redoc.png)

해당 페이지에서 **`/parse`** 버튼을 클릭한 후, `Try it out` 기능을 통해 파일을 직접 첨부하고 결과물을 테스트해 볼 수 있습니다.

---
Expand All @@ -27,9 +39,12 @@ FastAPI는 OpenAPI 기반의 대화형 API 문서를 자동으로 생성합니

### `POST /parse`

스캔된 일본어 신문 PDF 문서를 업로드 받아, 임시 디렉토리에 저장 후 `mineru` 파이프라인을 백그라운드로 실행하고 변환 결과를 기사 단위 DOM 구조가 담긴 JSON 형태로 반환합니다.
스캔된 일본어 신문 PDF 문서를 업로드 받아, 임시 디렉토리에 저장한 뒤
`mineru` 파이프라인을 백그라운드로 실행하고 변환 결과를 기사 단위 DOM
구조가 담긴 JSON 형태로 반환합니다.

#### 요청 매개변수 (Request Body)

- **`file`** (`UploadFile`, 필수): 변환할 PDF 바이너리 파일 데이터 (`multipart/form-data`)

#### cURL 테스트 예제
Expand Down Expand Up @@ -135,10 +150,17 @@ else:
```

#### 스키마 주요 노드 설명
1. **`BoundingBox` (`bbox`)**: 요소가 위치한 직사각형 영역 `(x0, y0, x1, y1)`. 신문 지면 상에서의 절대적 물리 좌표를 나타냅니다.
2. **`PageNode`**: 단일 신문 지면입니다. 여러 개의 기사(`articles`)와 광고(`ads`), 상단 헤더 정보(`headers`)를 포함합니다.
3. **`ArticleNode`**: 가장 핵심적인 DOM 요소로, 제목(`headline`), 문단 블록 배열(`body_blocks`), 기사에 속한 이미지 목록(`images`), 기사 내 독립된 사진 설명(`captions`)으로 구성됩니다.
4. **`ParseQuality`**: 변환 결과를 검증하기 위한 상태, 사용된 파서 엔진 정보(`mineru`), 주의해야할 품질 문제(`warnings`) 등의 메타데이터를 갖습니다.

1. **`BoundingBox` (`bbox`)**: 요소가 위치한 직사각형 영역
`(x0, y0, x1, y1)`입니다. 신문 지면 상의 절대 물리 좌표를 나타냅니다.
2. **`PageNode`**: 단일 신문 지면입니다. 여러 개의 기사(`articles`),
광고(`ads`), 상단 헤더 정보(`headers`)를 포함합니다.
3. **`ArticleNode`**: 가장 핵심적인 DOM 요소로, 제목(`headline`), 문단
블록 배열(`body_blocks`), 기사에 속한 이미지 목록(`images`), 기사 내
독립된 사진 설명(`captions`)으로 구성됩니다.
4. **`ParseQuality`**: 변환 결과를 검증하기 위한 상태, 사용된 파서 엔진
정보(`mineru`), 주의해야 할 품질 문제(`warnings`) 등의 메타데이터를
갖습니다.

---

Expand Down
Binary file added manual/assets/redoc.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added manual/assets/swagger-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading