Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 38 additions & 15 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,53 @@ on:

permissions:
contents: read
packages: write

jobs:
pytest:
build-test-image:
name: Build Test Image
runs-on: ubuntu-latest
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
outputs:
image_name: ${{ steps.image_name.outputs.name }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
uses: actions/checkout@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

액션들이 SHA로 핀되어 있지 않아 quality-gate 파이프라인이 실패합니다.

리포지토리 정책상 워크플로우의 모든 액션은 SHA로 핀되어야 합니다(test_workflow_actions_are_pinned_by_sha 실패). 현재 actions/checkout@v4, docker/setup-buildx-action@v3, docker/login-action@v3, docker/build-push-action@v5, 그리고 하단의 actions/checkout@v4 모두 태그로만 참조되고 있어 이 중 첫 번째가 보고되어 있고 나머지도 순차적으로 실패할 것입니다. 각 액션을 공식 릴리스의 커밋 SHA로 교체하고, 가독성을 위해 태그를 주석으로 남겨 주세요.

🔒 예시 (버전 주석 유지)
- uses: actions/checkout@<full-sha>  # v4.x.y
- uses: docker/setup-buildx-action@<full-sha>  # v3.x.y
- uses: docker/login-action@<full-sha>  # v3.x.y
- uses: docker/build-push-action@<full-sha>  # v5.x.y

Also applies to: 23-23, 26-26, 37-37, 56-56

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml at line 20, Replace every workflow action
reference that's using a tag (e.g., actions/checkout@v4,
docker/setup-buildx-action@v3, docker/login-action@v3,
docker/build-push-action@v5) with the corresponding official release commit SHA
and keep the tag as an inline comment for readability (for example replace
"actions/checkout@v4" with "actions/checkout@<full-sha>  # v4.x.y"); update all
occurrences noted (the first actions/checkout@v4 and the other instances at the
indicated lines) so every "uses:" entry is pinned by SHA rather than by tag.


- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Setup Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405
- name: Log in to GitHub Container Registry
uses: docker/login-action@v3
with:
python-version: '3.10'
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Setup uv
uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57
- name: Define image name and tag
id: image_name
run: echo "name=ghcr.io/${{ github.repository_owner,, }}/newsdom-api/test:${{ github.sha }}" | tee -a $GITHUB_OUTPUT
Comment on lines +32 to +34

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

Does the GitHub Actions expression language support the ,,Bash-style lowercase parameter expansion (e.g.,${{ github.repository_owner,, }})?

💡 Result:

No. GitHub Actions expressions do not support Bash-style lowercase parameter expansion syntax like ${{ github.repository_owner,, }}. GitHub’s expression language supports a defined set of literals, operators, and built-in functions (documented in the Expressions reference). It does not document shell-style parameter expansion operators ,, as part of the expression syntax, and the docs describe expressions as evaluating context references plus supported functions/operators rather than performing Bash parameter expansions. If you need lowercase, do it in a runner step using the shell (for example ${VAR,,} in Bash) or use an action designed for string case conversion (e.g., a JS-based action calling toLowerCase), then pass the result via outputs/env.

Citations:


${{ github.repository_owner,, }}는 유효한 GitHub Actions 표현식이 아니므로 이미지 이름이 잘못 생성되어 푸시가 실패합니다.

,,는 Bash 파라미터 확장 문법이며, GitHub Actions 표현식 언어는 이를 지원하지 않습니다. github.repository_owner가 대문자를 포함하면 GHCR은 소문자 이름만 허용하므로 푸시가 실패합니다. Bash 스텝에서 소문자로 변환하고 결과를 출력으로 전달해야 합니다.

수정안
-      - name: Define image name and tag
-        id: image_name
-        run: echo "name=ghcr.io/${{ github.repository_owner,, }}/newsdom-api/test:${{ github.sha }}" | tee -a $GITHUB_OUTPUT
+      - name: Define image name and tag
+        id: image_name
+        env:
+          OWNER: ${{ github.repository_owner }}
+          SHA: ${{ github.sha }}
+        run: |
+          owner_lc="${OWNER,,}"
+          echo "name=ghcr.io/${owner_lc}/newsdom-api/test:${SHA}" | tee -a "$GITHUB_OUTPUT"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Define image name and tag
id: image_name
run: echo "name=ghcr.io/${{ github.repository_owner,, }}/newsdom-api/test:${{ github.sha }}" | tee -a $GITHUB_OUTPUT
- name: Define image name and tag
id: image_name
env:
OWNER: ${{ github.repository_owner }}
SHA: ${{ github.sha }}
run: |
owner_lc="${OWNER,,}"
echo "name=ghcr.io/${owner_lc}/newsdom-api/test:${SHA}" | tee -a "$GITHUB_OUTPUT"
🧰 Tools
🪛 actionlint (1.7.12)

[error] 34-34: parser did not reach end of input after parsing the expression. 2 remaining token(s) in the input: ",", ","

(expression)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml around lines 32 - 34, The image name step (id:
image_name) currently uses the Actions expression `${{ github.repository_owner,,
}}` which is invalid; change the run script to capture `${{
github.repository_owner }}` into a shell variable, lowercase it using Bash
parameter expansion (e.g., owner_low="${owner,,}"), then echo the image name
using that lowercased variable to $GITHUB_OUTPUT so GHCR receives a valid
lowercase owner; update the run body in the image_name step accordingly.


- name: Build and push test image
uses: docker/build-push-action@v5
with:
version: '0.11.3'
context: .
file: ./Dockerfile.test
push: true
tags: ${{ steps.image_name.outputs.name }}
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Install dependencies
run: uv sync --frozen --all-extras
pytest:
name: Run Pytest in Container
runs-on: ubuntu-latest
needs: build-test-image
container:
image: ${{ needs.build-test-image.outputs.image_name }}
env:
PYTHONWARNINGS: error
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Run tests with warnings as errors
env:
PYTHONWARNINGS: error
- name: Run tests
run: uv run pytest
Comment on lines +46 to 59

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

astral-sh/setup-uv 핀 부재로 quality-gate의 test_tests_workflow_pins_uv_version이 실패합니다.

이 PR은 uv 설치를 컨테이너 이미지 빌드 시점으로 옮겼기 때문에 워크플로우에서 astral-sh/setup-uv 스텝이 사라졌지만, 품질 게이트 테스트는 여전히 해당 액션의 SHA 핀 존재를 요구합니다. 둘 중 하나를 선택해야 합니다:

  • 품질 게이트 테스트를 컨테이너 내 uv 핀(예: Dockerfile.test의 pip install uv==<ver>)을 검증하도록 업데이트, 또는
  • pytest 잡에 다시 astral-sh/setup-uv@<sha>를 추가(컨테이너 내부 uv를 선호한다면 중복이 됩니다).

컨테이너 기반 전략을 유지하는 방향이라면 첫 번째 옵션이 자연스럽습니다.

품질 게이트 테스트가 어느 파일에 있는지 알려주시면 컨테이너 내부 uv 핀을 검사하도록 수정하는 패치를 제안해 드릴 수 있습니다.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml around lines 46 - 59, The tests workflow
(.github/workflows/tests.yml) removed the astral-sh/setup-uv step causing the
quality-gate test test_tests_workflow_pins_uv_version to fail because it expects
an action SHA pin; either restore the setup step with a pinned reference (add
astral-sh/setup-uv@<sha> back into the pytest job) or modify the quality-gate
check to validate the uv version is pinned inside the container build (e.g.,
check Dockerfile.test or the test image build step for a pip install uv==<ver>);
pick the container-first approach if you prefer, then update the quality-gate
test to look for a concrete uv version in Dockerfile.test (or wherever uv is
installed during build) instead of requiring astral-sh/setup-uv@<sha>.

Comment on lines +58 to 59

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

How does uv runbehave whenUV_PROJECT_ENVIRONMENTis unset and the current working directory differs from the directory whereuv sync was originally run? Does it create a new .venv?

💡 Result:

When UV_PROJECT_ENVIRONMENT is unset, uv run discovers the project root by searching upwards from the current working directory (CWD) for the nearest ancestor directory containing a pyproject.toml file (with a [project] table). It then uses the .venv in that discovered project root directory. If the CWD is within the project directory where uv sync was run (i.e., the project root or a subdirectory), uv run will find the same project root and use the existing .venv there (creating it if missing). If the CWD differs from that directory—specifically, if the CWD is outside the project tree (no pyproject.toml found upwards from CWD)—then uv run will not discover any project. In that case, it falls back to non-project behavior: it does not use or create a project .venv, but may create/use other environments depending on context (e.g., no persistent .venv creation tied to a project). uv run does not create a new .venv in the CWD or elsewhere unless using flags like --isolated or specific script modes. It only manages the project .venv when a project is discovered.

Citations:


🏁 Script executed:

# Find and examine the Dockerfile.test to see the actual setup
find . -name "Dockerfile.test" -o -name "Dockerfile" -o -name "*.Dockerfile" | head -20

Repository: Seongho-Bae/newsdom-api

Length of output: 127


🏁 Script executed:

# Look at the complete GitHub Actions workflow configuration
cat -n .github/workflows/tests.yml | head -80

Repository: Seongho-Bae/newsdom-api

Length of output: 1932


🏁 Script executed:

# Check if pyproject.toml exists and where it's located
find . -name "pyproject.toml" -type f

Repository: Seongho-Bae/newsdom-api

Length of output: 83


🏁 Script executed:

cat -n ./Dockerfile.test

Repository: Seongho-Bae/newsdom-api

Length of output: 1382


🏁 Script executed:

cat -n ./pyproject.toml | head -30

Repository: Seongho-Bae/newsdom-api

Length of output: 959


GHA 컨테이너 작업 단계에서 Dockerfile에 포함된 .venv 재사용이 불가능합니다.

actions/checkout@v4는 컨테이너 내에서 저장소를 기본적으로 /github/workspace에 체크아웃합니다. uv run pytest를 /github/workspace에서 실행하면, uv는 현재 디렉터리에서 pyproject.toml을 찾아 프로젝트 루트를 /github/workspace로 해석합니다. Dockerfile에서 /app에 구축한 .venv는 이 프로젝트 트리 내에 존재하지 않으므로 발견되지 않습니다. 이미지 빌드 단계의 노력이 활용되지 않으며, 대신 새 가상 환경이 필요하게 됩니다.

다음 중 하나로 해결할 수 있습니다:

  • GHA 단계에서 working-directory: /app를 설정하여 올바른 프로젝트 루트에서 실행
  • run: /app/.venv/bin/pytest로 직접 호출
  • env: UV_PROJECT_ENVIRONMENT=/app/.venv를 설정하여 명시적으로 가상 환경 경로 지정
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tests.yml around lines 58 - 59, The CI step running "uv
run pytest" is executed from /github/workspace so it cannot reuse the Docker
image's /app/.venv; update the workflow to run pytest from the correct project
root by either setting the job step's working-directory to /app, changing the
run command to call the venv directly with "run: /app/.venv/bin/pytest", or
setting the environment variable UV_PROJECT_ENVIRONMENT=/app/.venv so that "uv
run pytest" picks up the existing virtualenv; modify the step that currently
contains "run: uv run pytest" accordingly.

34 changes: 34 additions & 0 deletions Dockerfile.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Docker image for running the test suite in a controlled and reliable environment
FROM python:3.10-slim

# Set timezone to avoid interactive prompts during package installations
ENV TZ=Asia/Tokyo
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

WORKDIR /app

# Install system dependencies, including Git, Tesseract for future use, and Japanese fonts
RUN apt-get update && apt-get install -y --no-install-recommends \
git \
tesseract-ocr \
tesseract-ocr-jpn \
fonts-ipafont-gothic \
&& rm -rf /var/lib/apt/lists/*

# Copy only necessary files for dependency installation first to leverage caching
COPY pyproject.toml uv.lock ./

# Install uv via pip
RUN pip install uv==0.1.41

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

🧩 Analysis chain

🌐 Web query:

What is the current recommended stable version of the Python uv package manager in 2026, and are there any known issues with uv 0.1.41?

💡 Result:

The current recommended stable version of the Python uv package manager is 0.11.7, released on 2026-04-15. uv is described as stable and ready for production use. uv 0.1.41 was an early release from 2024-05-08 with a single documented bug fix: "Remove unconstrained version error from requirements (#3443)". No major or ongoing known issues are reported for it in recent sources; it appears to be a minor historical version superseded by many stable updates.

Citations:


uv==0.1.41 버전 핀이 매우 오래된 버전입니다

uv 0.1.41은 2024년 5월 릴리스된 버전으로, 현재 권장 안정 버전인 0.11.7(2026년 4월 릴리스)와 비교했을 때 상당한 기능 및 동작 차이가 있습니다. 특히 uv sync의 프로젝트 설치 동작이 변경되었을 가능성이 높습니다. CI 재현성을 유지하면서도 버그 픽스와 성능 개선을 반영할 수 있도록 최신 안정 버전(0.11.x)으로 핀을 업데이트하는 것을 권장합니다.

추가로, 공식 문서에서 권장하는 설치 방식은 독립 설치 스크립트 또는 GitHub Actions에서는 astral-sh/setup-uv를 사용하는 것입니다. 컨테이너 환경에서는 pip install uv==<version> 유지는 가능하지만, 버전을 최신 안정 버전으로 업그레이드하는 것이 좋습니다.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@Dockerfile.test` at line 22, The Dockerfile currently pins uv to an old
release via the RUN pip install uv==0.1.41 instruction; update the pin to the
current stable 0.11.x series (e.g., 0.11.7) to pick up bug fixes and behavioral
changes in uv sync, or switch to the recommended installer (use the independent
install script or GitHub Action like astral-sh/setup-uv) for container builds;
change the RUN pip install line (RUN pip install uv==0.1.41) to either install
uv==0.11.7 or invoke the recommended installer step so CI uses the modern stable
version.


# Install all project dependencies into a virtual environment
RUN uv sync --frozen --all-extras

# Copy the rest of the application source code
COPY . .

# Add the virtual environment's bin to the PATH to make scripts like 'mineru' available
ENV PATH="/app/.venv/bin:$PATH"

# Verify that the 'mineru' executable is now on the PATH
RUN which mineru
77 changes: 77 additions & 0 deletions tests/test_benchmark_ocr.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
from __future__ import annotations

import json
import subprocess
from pathlib import Path
from unittest.mock import MagicMock, patch

import pytest
from tools import benchmark_ocr


@pytest.fixture
def mock_pdf_dir(tmp_path: Path) -> Path:
pdf_dir = tmp_path / "pdfs"
pdf_dir.mkdir()
(pdf_dir / "doc1.pdf").write_text("dummy pdf 1")
(pdf_dir / "doc2.pdf").write_text("dummy pdf 2")
(pdf_dir / "doc3.pdf").write_text("dummy pdf 3")
return pdf_dir


def test_benchmark_ocr_harness(mock_pdf_dir: Path, tmp_path: Path):
"""
The benchmark harness should run a set of engines against a corpus of PDFs
and correctly aggregate success, failure, and timeout results.
"""
output_path = tmp_path / "results.json"

# Mock engine runners
mock_engine1 = MagicMock(return_value={"status": "success", "page_count": 2})
mock_engine2 = MagicMock()
mock_engine2.side_effect = [
subprocess.TimeoutExpired(cmd="engine2", timeout=10),
RuntimeError("OCR process failed"),
{"status": "success", "page_count": 1},
]

engines = {
"engine1": mock_engine1,
"engine2": mock_engine2,
}

with patch.dict(benchmark_ocr.OCR_ENGINES, engines):
benchmark_ocr.main(
[
"--fixtures-dir",
str(mock_pdf_dir),
"--output",
str(output_path),
"--engines",
"engine1",
"engine2",
]
)

assert output_path.exists()
results = json.loads(output_path.read_text())

# Assert engine1 (always succeeds)
assert results["engine1"]["success"] == 3
assert results["engine1"]["failed"] == 0
assert results["engine1"]["timed_out"] == 0
assert "doc1.pdf" in results["engine1"]["results"]
assert results["engine1"]["results"]["doc1.pdf"]["status"] == "success"

# Assert engine2 (fails, times out, then succeeds)
assert results["engine2"]["success"] == 1
assert results["engine2"]["failed"] == 1
assert results["engine2"]["timed_out"] == 1
assert results["engine2"]["results"]["doc1.pdf"]["status"] == "timed_out"
assert results["engine2"]["results"]["doc2.pdf"]["status"] == "failed"
assert "OCR process failed" in results["engine2"]["results"]["doc2.pdf"]["error"]
assert results["engine2"]["results"]["doc3.pdf"]["status"] == "success"

# Check summary
assert results["summary"]["total_files"] == 3
assert results["summary"]["total_runs"] == 6
154 changes: 154 additions & 0 deletions tools/benchmark_ocr.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
from __future__ import annotations

import argparse
import json
import subprocess
import time
from pathlib import Path
from typing import Any, Callable

from newsdom_api.service import parse_pdf_bytes


def run_mineru_engine(pdf_path: Path) -> dict[str, Any]:
"""
Run the 'mineru' OCR engine on a single PDF file.

This function acts as a wrapper around the existing 'parse_pdf_bytes' service,
adapting its output for the benchmark harness.

Args:
pdf_path: The path to the PDF file to process.

Returns:
A dictionary containing the processing status and key metrics.
"""
response = parse_pdf_bytes(pdf_path.read_bytes(), filename=pdf_path.name)
return {
"status": "success",
"page_count": len(response.pages),
"article_count": sum(len(p.articles) for p in response.pages),
}


OCR_ENGINES: dict[str, Callable[[Path], dict[str, Any]]] = {
"mineru": run_mineru_engine,
}


def run_benchmark(
fixtures_dir: Path, output_path: Path, engines: list[str]
) -> None:
"""
Run the OCR benchmark against a directory of PDFs for a list of engines.

It iterates through each PDF and runs each specified engine, capturing
success, failure, or timeout, along with performance metrics. The
aggregated results are written to a JSON file.

Args:
fixtures_dir: The directory containing the PDF files to benchmark.
output_path: The path to write the final JSON results file.
engines: A list of engine names to run.
"""
pdf_paths = sorted(list(fixtures_dir.glob("*.pdf")))
if not pdf_paths:
raise FileNotFoundError(f"No PDF files found in {fixtures_dir}")

all_results: dict[str, Any] = {
"summary": {
"total_files": len(pdf_paths),
"total_runs": len(pdf_paths) * len(engines),
}
}

for engine_name in engines:
if engine_name not in OCR_ENGINES:
raise ValueError(f"Unknown engine: {engine_name}")

print(f"--- Running benchmark for engine: {engine_name} ---")
engine_runner = OCR_ENGINES[engine_name]
engine_results: dict[str, Any] = {
"success": 0,
"failed": 0,
"timed_out": 0,
"results": {},
}

for pdf_path in pdf_paths:
start_time = time.monotonic()
try:
result = engine_runner(pdf_path)
duration = time.monotonic() - start_time
engine_results["success"] += 1
engine_results["results"][pdf_path.name] = {
"status": "success",
"duration": round(duration, 2),
**result,
}
print(f" [SUCCESS] {pdf_path.name} in {duration:.2f}s")

except (RuntimeError, subprocess.CalledProcessError) as e:
duration = time.monotonic() - start_time
engine_results["failed"] += 1
engine_results["results"][pdf_path.name] = {
"status": "failed",
"duration": round(duration, 2),
"error": str(e),
}
print(f" [FAILED] {pdf_path.name} in {duration:.2f}s")

except subprocess.TimeoutExpired:
duration = time.monotonic() - start_time
engine_results["timed_out"] += 1
engine_results["results"][pdf_path.name] = {
"status": "timed_out",
"duration": round(duration, 2),
}
print(f" [TIMED OUT] {pdf_path.name} after {duration:.2f}s")
Comment on lines +80 to +108

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# run_mineru 내부에서 어떤 예외 타입이 실제로 발생하는지 재확인
rg -nP --type=py -C2 'raise (HTTPException|FileNotFoundError|RuntimeError|subprocess\.)' src/newsdom_api/mineru_runner.py

Repository: Seongho-Bae/newsdom-api

Length of output: 1457


🏁 Script executed:

cat -n tools/benchmark_ocr.py | head -150

Repository: Seongho-Bae/newsdom-api

Length of output: 5983


mineru 엔진의 실제 예외가 여기서 포착되지 않아 벤치마크가 중단될 수 있습니다.

parse_pdf_bytes → run_mineru의 구현(src/newsdom_api/mineru_runner.py)은 subprocess.TimeoutExpired와 subprocess.CalledProcessError를 fastapi.HTTPException으로 래핑해서 다시 던집니다(상태 코드 504, 500). 또한 mineru 실행 파일을 찾지 못하거나 필수 JSON 파일이 없으면 FileNotFoundError를 발생시킵니다.

현재 핸들러 (RuntimeError, subprocess.CalledProcessError)(91행)와 subprocess.TimeoutExpired(101행)는 이들 예외 중 어느 것도 매칭되지 않습니다. 따라서 mineru이 타임아웃하거나 실패할 때마다 예외가 루프 밖으로 전파되어 벤치마크 실행 자체가 중단되며, "failed"/"timed_out" 카운터가 증가하지 않고 부분 결과도 JSON으로 남지 않습니다.

가장 견고한 해법은 run_mineru_engine이 내부에서 HTTPException/FileNotFoundError를 의미 있는 예외로 변환하거나, 또는 run_benchmark의 except 절에 포괄적인 Exception 폴백을 추가해 예기치 않은 예외도 "failed"로 기록하는 것입니다.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tools/benchmark_ocr.py` around lines 80 - 108, The benchmark loop currently
only catches RuntimeError, subprocess.CalledProcessError and
subprocess.TimeoutExpired but mineru_runner wraps errors in
fastapi.HTTPException (504 for timeout, 500 for failures) and can raise
FileNotFoundError, so update the except handling in the try block that calls
engine_runner(pdf_path) (the run_benchmark / loop that contains engine_runner)
to also catch fastapi.HTTPException and FileNotFoundError (and optionally a
final broad Exception fallback); when catching fastapi.HTTPException inspect
e.status_code and treat 504 as "timed_out" and other codes as "failed"
(increment engine_results["timed_out"] or ["failed"], set
engine_results["results"][pdf_path.name] with status, duration and error), treat
FileNotFoundError as "failed" with the error message, and ensure a final except
Exception records unexpected errors as "failed" so the benchmark continues and
partial results are saved.


all_results[engine_name] = engine_results

output_path.write_text(
json.dumps(all_results, ensure_ascii=False, indent=2), encoding="utf-8"
)
print(f"\nBenchmark results written to {output_path}")


def main(argv: list[str] | None = None) -> None:
"""
Main entry point for the OCR benchmark script.

Parses command-line arguments and initiates the benchmark run.

Args:
argv: Optional list of command-line arguments for testing.
"""
parser = argparse.ArgumentParser(
description="Run OCR benchmark against a directory of PDFs."
)
parser.add_argument(
"--fixtures-dir",
type=Path,
required=True,
help="Directory containing PDF files.",
)
parser.add_argument(
"--output",
type=Path,
required=True,
help="Path to write the JSON results file.",
)
parser.add_argument(
"--engines",
nargs="+",
default=["mineru"],
help=f"Space-separated list of engines to benchmark. Available: {', '.join(OCR_ENGINES.keys())}",
)
args = parser.parse_args(argv)

run_benchmark(args.fixtures_dir, args.output, args.engines)


if __name__ == "__main__":
main()
Loading