Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .jules/bolt.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,3 @@
## 2026-06-30 - Regex over Generator `any` string loops
**Learning:** Using `any(...)` with a generator comprehension in string evaluation paths allocates a new generator and adds Python-level loop overhead for every character.
**Action:** Replace `any()` generators with a pre-compiled regex (`re.compile().search()`) to evaluate string patterns in C, achieving a ~7x speedup for text-heavy operations.
## 2026-07-01 - Precompile MinerU Argument Allowlist
**Learning:** Recompiling the same allowlist pattern through repeated `re.match(...)` calls adds avoidable overhead in MinerU command argument validation.
**Action:** Keep the strict allowlist semantics, but compile the regex once at module load and validate each argument with `Pattern.fullmatch()`.
5 changes: 0 additions & 5 deletions .jules/sentinel.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,3 @@
**Vulnerability:** Unhandled FastAPI exceptions can produce sanitized 500 responses without the same defense-in-depth headers applied by normal middleware responses.
**Learning:** Error response paths need explicit coverage because exception handlers can bypass or duplicate header logic differently from successful request paths.
**Prevention:** Route both middleware responses and global 500 exception responses through a shared security-header helper.

## 2025-02-14 - Fix Command Injection Risk in CLI arguments
**Vulnerability:** Command arguments filtering relied on a denylist of shell control characters (`_SHELL_CONTROL_CHARS = frozenset("&;|`$<>")`), which can be bypassed. Furthermore, certain valid file path characters could still trigger issues or be blocked incorrectly.
**Learning:** Checking for bad characters (denylist) is brittle and frequently bypassable. CLI tools and scripts should strictly validate user inputs (such as file paths) using regex allowlists rather than mutating paths or using weak character blocklists.
**Prevention:** Apply a strict regex allowlist (e.g., `^[\w\s/\\.:~()-]+$`) to validate user-supplied file paths and arguments. Ensure the allowlist permits valid cross-OS path characters like colons, tildes, and parentheses to avoid functional regressions while maintaining security.
16 changes: 6 additions & 10 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Performance
- `newsdom_api.dom_builder._html_safe_text` 함수에 early return과 타입 체크를 도입하여 불필요한 `str()` 캐스팅을 제거함으로써 처리 속도를 개선했습니다.
- `mineru_runner.py`의 MinerU 명령어 인자 allowlist 검증 정규식을 사전 컴파일하여 반복 호출 비용을 줄였습니다.

### Added
- Manual now includes a dedicated security reporting page so vulnerability disclosure, supported branches, and safe evidence handling are visible from the documentation navigation.
- Package metadata now publishes Homepage, Documentation, Repository, Issues, and Security URLs so package indexes and developer tools show the same live project surfaces as the manual and OpenAPI docs.
- Public documentation metadata now points to the current `ContextualWisdomLab/newsdom-api` repository, keeping the MkDocs header, OpenAPI contact link, Scorecard badges, and security reporting links aligned with the live project.
- API reference manual now embeds the checked-in Swagger UI and ReDoc screenshots so the interactive documentation section renders with visible product context instead of text-only placeholders.
- OpenAPI/ReDoc response samples now include concrete `HealthResponse`, `ParseResponse`, and error examples so visual API documentation no longer renders placeholder property names.
- [CLI] 파싱된 DOM JSON 파일이 `ParseResponse` 스키마 형식을 만족하는지 검사하는 `tools/validate_dom.py` 도구를 추가했습니다.
- [CLI] 파싱된 DOM JSON 파일에서 headline과 body_blocks 등 순수 텍스트만 추출하는 `tools/extract_text.py` 도구를 추가했습니다.
- [CLI] 파싱된 NewsDOM JSON을 Markdown 포맷으로 변환하는 `tools/export_markdown.py` 도구를 추가했습니다.
- [CLI] `tools/batch_parse_pdf.py`에 하위 디렉터리의 PDF를 일괄 처리하고 상대 경로로 JSON을 저장하는 `--recursive` 옵션을 추가했습니다.
- OpenAPI 문서에 contact 및 MIT license metadata를 추가하여 API 소비자가 maintainer와 라이선스 정보를 더 쉽게 확인할 수 있도록 개선
Expand Down Expand Up @@ -86,7 +82,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Synthetic newspaper fixture generation and structural equivalence checks
- Protected-branch CI, security gates, release provenance workflow, and Git Flow documentation

[Unreleased]: https://github.com/ContextualWisdomLab/newsdom-api/compare/v0.2.0...HEAD
[0.2.0]: https://github.com/ContextualWisdomLab/newsdom-api/compare/v0.1.1...v0.2.0
[0.1.1]: https://github.com/ContextualWisdomLab/newsdom-api/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/ContextualWisdomLab/newsdom-api/releases/tag/v0.1.0
[Unreleased]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.2.0...HEAD
[0.2.0]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.1...v0.2.0
[0.1.1]: https://github.com/Seongho-Bae/newsdom-api/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/Seongho-Bae/newsdom-api/releases/tag/v0.1.0
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# NewsDOM API

[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/ContextualWisdomLab/newsdom-api/badge)](https://securityscorecards.dev/viewer/?uri=github.com/ContextualWisdomLab/newsdom-api)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/Seongho-Bae/newsdom-api/badge)](https://securityscorecards.dev/viewer/?uri=github.com/Seongho-Bae/newsdom-api)

NewsDOM API parses scanned Japanese newspaper PDFs into DOM-like article trees.

Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Please report a vulnerability privately by opening a GitHub Security
Advisory draft for this repository:

- [GitHub Security Advisory draft](https://github.com/ContextualWisdomLab/newsdom-api/security/advisories/new)
- [GitHub Security Advisory draft](https://github.com/Seongho-Bae/newsdom-api/security/advisories/new)
- [Maintainer profile](https://github.com/seonghobae)

Do not open a public issue for an unpatched vulnerability.
Expand Down
8 changes: 0 additions & 8 deletions manual/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,6 @@ FastAPI는 OpenAPI 기반의 대화형 API 문서를 자동으로 생성합니

해당 페이지에서 **`/parse`** 버튼을 클릭한 후, `Try it out` 기능을 통해 파일을 직접 첨부하고 결과물을 테스트해 볼 수 있습니다.

### 문서 UI 미리보기

아래 캡처는 `/docs`와 `/redoc`의 기본 배치를 빠르게 확인하기 위한 정적 레이아웃 예시입니다. 실제 응답 예시는 실행 중인 서버의 `/openapi.json` 스키마에서 생성됩니다.

![Swagger UI에서 health와 parse 엔드포인트가 보이는 화면](assets/swagger-ui.png)

![ReDoc에서 NewsDOM API 엔드포인트 문서가 보이는 화면](assets/redoc.png)

---

## 3. API 엔드포인트 세부 설명
Expand Down
4 changes: 2 additions & 2 deletions manual/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# NewsDOM API 개요 및 아키텍처

[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/ContextualWisdomLab/newsdom-api/badge)](https://securityscorecards.dev/viewer/?uri=github.com/ContextualWisdomLab/newsdom-api)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/Seongho-Bae/newsdom-api/badge)](https://securityscorecards.dev/viewer/?uri=github.com/Seongho-Bae/newsdom-api)

**NewsDOM API**는 스캔된 일본어 신문 PDF 문서를 분석하여, 웹 브라우저의 DOM(Document Object Model)과 유사한 기사(Article) 단위의 트리 구조로 파싱해주는 API 서비스입니다.

Expand Down Expand Up @@ -37,4 +37,4 @@ MinerU가 뱉어낸 선형적인(Linear) 블록 리스트(`content_list.json`)

---

👉 **[설치 가이드](installation.md)**를 읽고 직접 환경을 구성해 보세요.
👉 **[설치 가이드](installation.md)**를 읽고 직접 환경을 구성해 보세요.
32 changes: 0 additions & 32 deletions manual/security.md

This file was deleted.

2 changes: 1 addition & 1 deletion manual/workflow/ocr-accuracy-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ This will overwrite the existing placeholder fixture with a new baseline derived

As of 2026-04-23, the execution of this harness is **BLOCKED**. The underlying `mineru` OCR process hangs indefinitely when processing the private PDF files.

This issue is tracked in **[#70](https://github.com/ContextualWisdomLab/newsdom-api/issues/70)**.
This issue is tracked in **[#70](https://github.com/Seongho-Bae/newsdom-api/issues/70)**.

Until this blocker is resolved, generating a new `private_page_baseline.json` from the local corpus is not possible. The harness script and tests have been merged so that the tooling is in place once the blocker is fixed.
7 changes: 3 additions & 4 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
site_name: NewsDOM API 매뉴얼
site_url: https://contextwisdomlab.github.io/newsdom-api/
site_url: https://seongho-bae.github.io/newsdom-api/
site_description: 일본어 신문 스캔 PDF를 DOM 형태의 트리로 파싱하는 API
repo_url: https://github.com/ContextualWisdomLab/newsdom-api
repo_name: ContextualWisdomLab/newsdom-api
repo_url: https://github.com/Seongho-Bae/newsdom-api
repo_name: Seongho-Bae/newsdom-api

docs_dir: manual

Expand Down Expand Up @@ -33,7 +33,6 @@ nav:
- 시작하기: index.md
- 설치 가이드: installation.md
- 사용 방법 및 API: api-reference.md
- 보안 제보: security.md
- 개발 및 기여:
- development.md
- 워크플로우:
Expand Down
7 changes: 0 additions & 7 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,6 @@ dependencies = [
"pypdf>=6.13.3,<7.0",
]

[project.urls]
Homepage = "https://contextwisdomlab.github.io/newsdom-api/"
Documentation = "https://contextwisdomlab.github.io/newsdom-api/"
Repository = "https://github.com/ContextualWisdomLab/newsdom-api"
Issues = "https://github.com/ContextualWisdomLab/newsdom-api/issues"
Security = "https://github.com/ContextualWisdomLab/newsdom-api/security/advisories/new"

[project.optional-dependencies]
dev = [
"pytest>=8.3,<10.0",
Expand Down
12 changes: 6 additions & 6 deletions src/newsdom_api/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from pypdf.errors import PdfReadError

from .errors import MineruIncompleteOutputError, MineruRuntimeUnavailableError
from .schemas import ErrorResponse, HealthResponse, ParseResponse
from .schemas import HealthResponse, ParseResponse
from .service import parse_pdf_bytes

MAX_PARSE_UPLOAD_BYTES = 20 * 1024 * 1024
Expand All @@ -38,7 +38,7 @@
version="0.2.0",
contact={
"name": "Seongho Bae",
"url": "https://github.com/ContextualWisdomLab/newsdom-api",
"url": "https://github.com/Seongho-Bae/newsdom-api",
},
license_info={
"name": "MIT License",
Expand Down Expand Up @@ -132,10 +132,10 @@ def _validate_pdf_structure(pdf_bytes: bytes) -> None:
"document using MinerU."
),
responses={
413: {"model": ErrorResponse, "description": "Payload Too Large"},
415: {"model": ErrorResponse, "description": "Unsupported Media Type"},
502: {"model": ErrorResponse, "description": "Bad Gateway"},
503: {"model": ErrorResponse, "description": "Service Unavailable"},
413: {"description": "Payload Too Large"},
415: {"description": "Unsupported Media Type"},
502: {"description": "Bad Gateway"},
503: {"description": "Service Unavailable"},
},
tags=["Parser"],
)
Expand Down
13 changes: 8 additions & 5 deletions src/newsdom_api/mineru_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@

import json
import os
import re
import shutil
import subprocess
import tempfile
Expand All @@ -14,14 +13,14 @@

from .errors import MineruIncompleteOutputError, MineruRuntimeUnavailableError

_MINERU_COMMAND_ARG_PATTERN = re.compile(r"[\w /\\.:~()+\-@=,\[\]!']+")
_SHELL_CONTROL_CHARS = frozenset("&;|`$<>")


def _mineru_command_arg(value: str | Path, *, label: str) -> str:
"""Validate a path or executable string before passing it to MinerU argv."""

value_str = str(value)
if not _MINERU_COMMAND_ARG_PATTERN.fullmatch(value_str):
if "\0" in value_str or any(char in value_str for char in _SHELL_CONTROL_CHARS):
raise ValueError(f"Unsafe {label} for MinerU command")
if value_str.startswith("-"):
raise ValueError(f"Unsafe {label} for MinerU command")
Expand Down Expand Up @@ -114,9 +113,13 @@ def _read_mineru_json(path: Path, *, artifact: str) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise MineruIncompleteOutputError(f"{artifact} JSON was malformed") from exc
raise MineruIncompleteOutputError(
f"{artifact} JSON was malformed"
) from exc
except (OSError, UnicodeDecodeError) as exc:
raise MineruIncompleteOutputError(f"{artifact} JSON could not be read") from exc
raise MineruIncompleteOutputError(
f"{artifact} JSON could not be read"
) from exc


def _parse_mineru_output(
Expand Down
74 changes: 0 additions & 74 deletions src/newsdom_api/schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,53 +6,6 @@

from pydantic import BaseModel, Field

BOUNDING_BOX_EXAMPLE = {"x0": 120.0, "y0": 320.0, "x1": 420.0, "y1": 560.0}
CAPTION_EXAMPLE = {
"text": "Photo caption text extracted from the newspaper page.",
"bbox": {"x0": 120.0, "y0": 565.0, "x1": 420.0, "y1": 620.0},
}
IMAGE_EXAMPLE = {
"path": "extracted_images/page1_img1.jpg",
"media_type": "image",
"bbox": BOUNDING_BOX_EXAMPLE,
"captions": [CAPTION_EXAMPLE],
"footnotes": [],
}
ARTICLE_EXAMPLE = {
"article_id": "page-1-article-1",
"headline": "Morning edition headline",
"bbox": {"x0": 10.0, "y0": 20.0, "x1": 760.0, "y1": 900.0},
"body_blocks": [
"First paragraph of the article body.",
"Second paragraph continues the story.",
],
"images": [IMAGE_EXAMPLE],
"captions": [],
"footnotes": [],
}
PAGE_EXAMPLE = {
"page_number": 1,
"width": 800.5,
"height": 1200.0,
"articles": [ARTICLE_EXAMPLE],
"ads": ["Advertisement block text"],
"headers": ["2026-04-09 morning edition"],
"footers": ["Published by NewsDOM sample publisher"],
"page_numbers": ["1"],
}
PARSE_QUALITY_EXAMPLE = {
"status": "success",
"parser": "mineru",
"warnings": ["Page 1: overlapping layout blocks were normalized."],
}
PARSE_RESPONSE_EXAMPLE = {
"document_id": "sample-newspaper",
"pages": [PAGE_EXAMPLE],
"quality": PARSE_QUALITY_EXAMPLE,
}
HEALTH_RESPONSE_EXAMPLE = {"status": "ok"}
ERROR_RESPONSE_EXAMPLE = {"detail": "Unsupported Media Type"}


class BoundingBox(BaseModel):
"""Axis-aligned bounding box expressed in page coordinates."""
Expand All @@ -62,8 +15,6 @@ class BoundingBox(BaseModel):
x1: float = Field(..., description="Rightmost X coordinate of the bounding box.")
y1: float = Field(..., description="Bottommost Y coordinate of the bounding box.")

model_config = {"json_schema_extra": {"example": BOUNDING_BOX_EXAMPLE}}


class CaptionNode(BaseModel):
"""Caption text associated with an image or figure."""
Expand All @@ -74,8 +25,6 @@ class CaptionNode(BaseModel):
description="Bounding box of the caption when parser coordinates are available.",
)

model_config = {"json_schema_extra": {"example": CAPTION_EXAMPLE}}


class ImageNode(BaseModel):
"""Image metadata preserved in the canonical page structure."""
Expand All @@ -98,8 +47,6 @@ class ImageNode(BaseModel):
description="Footnotes associated with this image.",
)

model_config = {"json_schema_extra": {"example": IMAGE_EXAMPLE}}


class ArticleNode(BaseModel):
"""Article-level grouping of headline, body blocks, and related media."""
Expand Down Expand Up @@ -129,8 +76,6 @@ class ArticleNode(BaseModel):
description="Footnotes associated with the article.",
)

model_config = {"json_schema_extra": {"example": ARTICLE_EXAMPLE}}


class PageNode(BaseModel):
"""Single parsed page including article, ad, and header groupings."""
Expand Down Expand Up @@ -167,8 +112,6 @@ class PageNode(BaseModel):
description="Visible page-number text blocks extracted from this page.",
)

model_config = {"json_schema_extra": {"example": PAGE_EXAMPLE}}


class ParseQuality(BaseModel):
"""Quality metadata describing parser provenance and warnings."""
Expand All @@ -184,8 +127,6 @@ class ParseQuality(BaseModel):
description="Non-fatal warnings encountered during the parsing process.",
)

model_config = {"json_schema_extra": {"example": PARSE_QUALITY_EXAMPLE}}


class ParseResponse(BaseModel):
"""Top-level API response for a parsed document."""
Expand All @@ -202,8 +143,6 @@ class ParseResponse(BaseModel):
description="Metadata detailing parsing provenance and encountered warnings.",
)

model_config = {"json_schema_extra": {"example": PARSE_RESPONSE_EXAMPLE}}


class HealthResponse(BaseModel):
"""Liveness response model for deployment health checks."""
Expand All @@ -212,16 +151,3 @@ class HealthResponse(BaseModel):
default="ok",
description="Current operational status of the service.",
)

model_config = {"json_schema_extra": {"example": HEALTH_RESPONSE_EXAMPLE}}


class ErrorResponse(BaseModel):
"""Standardized error response model for HTTP exceptions."""

detail: str = Field(
...,
description="A clear and specific error message describing what went wrong.",
)

model_config = {"json_schema_extra": {"example": ERROR_RESPONSE_EXAMPLE}}
8 changes: 2 additions & 6 deletions tests/test_benchmark_ocr.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,7 @@ def test_benchmark_ocr_harness_json(mock_pdf_dir: Path, tmp_path: Path) -> None:
assert results["summary"]["total_runs"] == 6


def test_benchmark_ocr_recursive_and_csv(
mock_pdf_dir: Path, tmp_path: Path
) -> None:
def test_benchmark_ocr_recursive_and_csv(mock_pdf_dir: Path, tmp_path: Path) -> None:
output_path = tmp_path / "results.csv"

mock_engine = MagicMock(return_value={"status": "success", "page_count": 2})
Expand Down Expand Up @@ -118,9 +116,7 @@ def test_benchmark_ocr_no_pdfs(tmp_path: Path) -> None:
)


def test_benchmark_ocr_unknown_engine(
mock_pdf_dir: Path, tmp_path: Path
) -> None:
def test_benchmark_ocr_unknown_engine(mock_pdf_dir: Path, tmp_path: Path) -> None:
"""If an unknown engine is specified, ValueError is raised."""
with pytest.raises(ValueError, match="Unknown engine: fake_engine"):
benchmark_ocr.main(
Expand Down
Loading
Loading