Repository navigation
feat(ci): Finalize Docker-based CI and OCR benchmark harness #80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -7,30 +7,53 @@ on: | |||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| permissions: | ||||||||||||||||||||||||
| contents: read | ||||||||||||||||||||||||
| packages: write | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| jobs: | ||||||||||||||||||||||||
| pytest: | ||||||||||||||||||||||||
| build-test-image: | ||||||||||||||||||||||||
| name: Build Test Image | ||||||||||||||||||||||||
| runs-on: ubuntu-latest | ||||||||||||||||||||||||
| env: | ||||||||||||||||||||||||
| FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true | ||||||||||||||||||||||||
| outputs: | ||||||||||||||||||||||||
| image_name: ${{ steps.image_name.outputs.name }} | ||||||||||||||||||||||||
| steps: | ||||||||||||||||||||||||
| - name: Checkout | ||||||||||||||||||||||||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd | ||||||||||||||||||||||||
| uses: actions/checkout@v4 | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| - name: Set up Docker Buildx | ||||||||||||||||||||||||
| uses: docker/setup-buildx-action@v3 | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| - name: Setup Python | ||||||||||||||||||||||||
| uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 | ||||||||||||||||||||||||
| - name: Log in to GitHub Container Registry | ||||||||||||||||||||||||
| uses: docker/login-action@v3 | ||||||||||||||||||||||||
| with: | ||||||||||||||||||||||||
| python-version: '3.10' | ||||||||||||||||||||||||
| registry: ghcr.io | ||||||||||||||||||||||||
| username: ${{ github.actor }} | ||||||||||||||||||||||||
| password: ${{ secrets.GITHUB_TOKEN }} | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| - name: Setup uv | ||||||||||||||||||||||||
| uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 | ||||||||||||||||||||||||
| - name: Define image name and tag | ||||||||||||||||||||||||
| id: image_name | ||||||||||||||||||||||||
| run: echo "name=ghcr.io/${{ github.repository_owner,, }}/newsdom-api/test:${{ github.sha }}" | tee -a $GITHUB_OUTPUT | ||||||||||||||||||||||||
|
Comment on lines
+32
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🌐 Web query:
💡 Result: No. GitHub Actions expressions do not support Bash-style lowercase parameter expansion syntax like ${{ github.repository_owner,, }}. GitHub’s expression language supports a defined set of literals, operators, and built-in functions (documented in the Expressions reference). It does not document shell-style parameter expansion operators ,, as part of the expression syntax, and the docs describe expressions as evaluating context references plus supported functions/operators rather than performing Bash parameter expansions. If you need lowercase, do it in a runner step using the shell (for example ${VAR,,} in Bash) or use an action designed for string case conversion (e.g., a JS-based action calling toLowerCase), then pass the result via outputs/env. Citations:
수정안- - name: Define image name and tag
- id: image_name
- run: echo "name=ghcr.io/${{ github.repository_owner,, }}/newsdom-api/test:${{ github.sha }}" | tee -a $GITHUB_OUTPUT
+ - name: Define image name and tag
+ id: image_name
+ env:
+ OWNER: ${{ github.repository_owner }}
+ SHA: ${{ github.sha }}
+ run: |
+ owner_lc="${OWNER,,}"
+ echo "name=ghcr.io/${owner_lc}/newsdom-api/test:${SHA}" | tee -a "$GITHUB_OUTPUT"📝 Committable suggestion
Suggested change
🧰 Tools🪛 actionlint (1.7.12)[error] 34-34: parser did not reach end of input after parsing the expression. 2 remaining token(s) in the input: ",", "," (expression) 🤖 Prompt for AI Agents |
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| - name: Build and push test image | ||||||||||||||||||||||||
| uses: docker/build-push-action@v5 | ||||||||||||||||||||||||
| with: | ||||||||||||||||||||||||
| version: '0.11.3' | ||||||||||||||||||||||||
| context: . | ||||||||||||||||||||||||
| file: ./Dockerfile.test | ||||||||||||||||||||||||
| push: true | ||||||||||||||||||||||||
| tags: ${{ steps.image_name.outputs.name }} | ||||||||||||||||||||||||
| cache-from: type=gha | ||||||||||||||||||||||||
| cache-to: type=gha,mode=max | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| - name: Install dependencies | ||||||||||||||||||||||||
| run: uv sync --frozen --all-extras | ||||||||||||||||||||||||
| pytest: | ||||||||||||||||||||||||
| name: Run Pytest in Container | ||||||||||||||||||||||||
| runs-on: ubuntu-latest | ||||||||||||||||||||||||
| needs: build-test-image | ||||||||||||||||||||||||
| container: | ||||||||||||||||||||||||
| image: ${{ needs.build-test-image.outputs.image_name }} | ||||||||||||||||||||||||
| env: | ||||||||||||||||||||||||
| PYTHONWARNINGS: error | ||||||||||||||||||||||||
| steps: | ||||||||||||||||||||||||
| - name: Checkout | ||||||||||||||||||||||||
| uses: actions/checkout@v4 | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| - name: Run tests with warnings as errors | ||||||||||||||||||||||||
| env: | ||||||||||||||||||||||||
| PYTHONWARNINGS: error | ||||||||||||||||||||||||
| - name: Run tests | ||||||||||||||||||||||||
| run: uv run pytest | ||||||||||||||||||||||||
|
Comment on lines
+46
to
59
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
이 PR은
컨테이너 기반 전략을 유지하는 방향이라면 첫 번째 옵션이 자연스럽습니다. 품질 게이트 테스트가 어느 파일에 있는지 알려주시면 컨테이너 내부 uv 핀을 검사하도록 수정하는 패치를 제안해 드릴 수 있습니다. 🤖 Prompt for AI Agents
Comment on lines
+58
to
59
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🌐 Web query:
💡 Result: When UV_PROJECT_ENVIRONMENT is unset, uv run discovers the project root by searching upwards from the current working directory (CWD) for the nearest ancestor directory containing a pyproject.toml file (with a [project] table). It then uses the .venv in that discovered project root directory. If the CWD is within the project directory where uv sync was run (i.e., the project root or a subdirectory), uv run will find the same project root and use the existing .venv there (creating it if missing). If the CWD differs from that directory—specifically, if the CWD is outside the project tree (no pyproject.toml found upwards from CWD)—then uv run will not discover any project. In that case, it falls back to non-project behavior: it does not use or create a project .venv, but may create/use other environments depending on context (e.g., no persistent .venv creation tied to a project). uv run does not create a new .venv in the CWD or elsewhere unless using flags like --isolated or specific script modes. It only manages the project .venv when a project is discovered. Citations:
🏁 Script executed: # Find and examine the Dockerfile.test to see the actual setup
find . -name "Dockerfile.test" -o -name "Dockerfile" -o -name "*.Dockerfile" | head -20Repository: Seongho-Bae/newsdom-api Length of output: 127 🏁 Script executed: # Look at the complete GitHub Actions workflow configuration
cat -n .github/workflows/tests.yml | head -80Repository: Seongho-Bae/newsdom-api Length of output: 1932 🏁 Script executed: # Check if pyproject.toml exists and where it's located
find . -name "pyproject.toml" -type fRepository: Seongho-Bae/newsdom-api Length of output: 83 🏁 Script executed: cat -n ./Dockerfile.testRepository: Seongho-Bae/newsdom-api Length of output: 1382 🏁 Script executed: cat -n ./pyproject.toml | head -30Repository: Seongho-Bae/newsdom-api Length of output: 959 GHA 컨테이너 작업 단계에서 Dockerfile에 포함된
다음 중 하나로 해결할 수 있습니다:
🤖 Prompt for AI Agents |
||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # Docker image for running the test suite in a controlled and reliable environment | ||
| FROM python:3.10-slim | ||
|
|
||
| # Set timezone to avoid interactive prompts during package installations | ||
| ENV TZ=Asia/Tokyo | ||
| RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone | ||
|
|
||
| WORKDIR /app | ||
|
|
||
| # Install system dependencies, including Git, Tesseract for future use, and Japanese fonts | ||
| RUN apt-get update && apt-get install -y --no-install-recommends \ | ||
| git \ | ||
| tesseract-ocr \ | ||
| tesseract-ocr-jpn \ | ||
| fonts-ipafont-gothic \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| # Copy only necessary files for dependency installation first to leverage caching | ||
| COPY pyproject.toml uv.lock ./ | ||
|
|
||
| # Install uv via pip | ||
| RUN pip install uv==0.1.41 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion | 🟠 Major 🧩 Analysis chain🌐 Web query:
💡 Result: The current recommended stable version of the Python uv package manager is 0.11.7, released on 2026-04-15. uv is described as stable and ready for production use. uv 0.1.41 was an early release from 2024-05-08 with a single documented bug fix: "Remove unconstrained version error from requirements ( Citations:
추가로, 공식 문서에서 권장하는 설치 방식은 독립 설치 스크립트 또는 GitHub Actions에서는 🤖 Prompt for AI Agents |
||
|
|
||
| # Install all project dependencies into a virtual environment | ||
| RUN uv sync --frozen --all-extras | ||
|
|
||
| # Copy the rest of the application source code | ||
| COPY . . | ||
|
|
||
| # Add the virtual environment's bin to the PATH to make scripts like 'mineru' available | ||
| ENV PATH="/app/.venv/bin:$PATH" | ||
|
|
||
| # Verify that the 'mineru' executable is now on the PATH | ||
| RUN which mineru | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import subprocess | ||
| from pathlib import Path | ||
| from unittest.mock import MagicMock, patch | ||
|
|
||
| import pytest | ||
| from tools import benchmark_ocr | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def mock_pdf_dir(tmp_path: Path) -> Path: | ||
| pdf_dir = tmp_path / "pdfs" | ||
| pdf_dir.mkdir() | ||
| (pdf_dir / "doc1.pdf").write_text("dummy pdf 1") | ||
| (pdf_dir / "doc2.pdf").write_text("dummy pdf 2") | ||
| (pdf_dir / "doc3.pdf").write_text("dummy pdf 3") | ||
| return pdf_dir | ||
|
|
||
|
|
||
| def test_benchmark_ocr_harness(mock_pdf_dir: Path, tmp_path: Path): | ||
| """ | ||
| The benchmark harness should run a set of engines against a corpus of PDFs | ||
| and correctly aggregate success, failure, and timeout results. | ||
| """ | ||
| output_path = tmp_path / "results.json" | ||
|
|
||
| # Mock engine runners | ||
| mock_engine1 = MagicMock(return_value={"status": "success", "page_count": 2}) | ||
| mock_engine2 = MagicMock() | ||
| mock_engine2.side_effect = [ | ||
| subprocess.TimeoutExpired(cmd="engine2", timeout=10), | ||
| RuntimeError("OCR process failed"), | ||
| {"status": "success", "page_count": 1}, | ||
| ] | ||
|
|
||
| engines = { | ||
| "engine1": mock_engine1, | ||
| "engine2": mock_engine2, | ||
| } | ||
|
|
||
| with patch.dict(benchmark_ocr.OCR_ENGINES, engines): | ||
| benchmark_ocr.main( | ||
| [ | ||
| "--fixtures-dir", | ||
| str(mock_pdf_dir), | ||
| "--output", | ||
| str(output_path), | ||
| "--engines", | ||
| "engine1", | ||
| "engine2", | ||
| ] | ||
| ) | ||
|
|
||
| assert output_path.exists() | ||
| results = json.loads(output_path.read_text()) | ||
|
|
||
| # Assert engine1 (always succeeds) | ||
| assert results["engine1"]["success"] == 3 | ||
| assert results["engine1"]["failed"] == 0 | ||
| assert results["engine1"]["timed_out"] == 0 | ||
| assert "doc1.pdf" in results["engine1"]["results"] | ||
| assert results["engine1"]["results"]["doc1.pdf"]["status"] == "success" | ||
|
|
||
| # Assert engine2 (fails, times out, then succeeds) | ||
| assert results["engine2"]["success"] == 1 | ||
| assert results["engine2"]["failed"] == 1 | ||
| assert results["engine2"]["timed_out"] == 1 | ||
| assert results["engine2"]["results"]["doc1.pdf"]["status"] == "timed_out" | ||
| assert results["engine2"]["results"]["doc2.pdf"]["status"] == "failed" | ||
| assert "OCR process failed" in results["engine2"]["results"]["doc2.pdf"]["error"] | ||
| assert results["engine2"]["results"]["doc3.pdf"]["status"] == "success" | ||
|
|
||
| # Check summary | ||
| assert results["summary"]["total_files"] == 3 | ||
| assert results["summary"]["total_runs"] == 6 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| import subprocess | ||
| import time | ||
| from pathlib import Path | ||
| from typing import Any, Callable | ||
|
|
||
| from newsdom_api.service import parse_pdf_bytes | ||
|
|
||
|
|
||
| def run_mineru_engine(pdf_path: Path) -> dict[str, Any]: | ||
| """ | ||
| Run the 'mineru' OCR engine on a single PDF file. | ||
|
|
||
| This function acts as a wrapper around the existing 'parse_pdf_bytes' service, | ||
| adapting its output for the benchmark harness. | ||
|
|
||
| Args: | ||
| pdf_path: The path to the PDF file to process. | ||
|
|
||
| Returns: | ||
| A dictionary containing the processing status and key metrics. | ||
| """ | ||
| response = parse_pdf_bytes(pdf_path.read_bytes(), filename=pdf_path.name) | ||
| return { | ||
| "status": "success", | ||
| "page_count": len(response.pages), | ||
| "article_count": sum(len(p.articles) for p in response.pages), | ||
| } | ||
|
|
||
|
|
||
| OCR_ENGINES: dict[str, Callable[[Path], dict[str, Any]]] = { | ||
| "mineru": run_mineru_engine, | ||
| } | ||
|
|
||
|
|
||
| def run_benchmark( | ||
| fixtures_dir: Path, output_path: Path, engines: list[str] | ||
| ) -> None: | ||
| """ | ||
| Run the OCR benchmark against a directory of PDFs for a list of engines. | ||
|
|
||
| It iterates through each PDF and runs each specified engine, capturing | ||
| success, failure, or timeout, along with performance metrics. The | ||
| aggregated results are written to a JSON file. | ||
|
|
||
| Args: | ||
| fixtures_dir: The directory containing the PDF files to benchmark. | ||
| output_path: The path to write the final JSON results file. | ||
| engines: A list of engine names to run. | ||
| """ | ||
| pdf_paths = sorted(list(fixtures_dir.glob("*.pdf"))) | ||
| if not pdf_paths: | ||
| raise FileNotFoundError(f"No PDF files found in {fixtures_dir}") | ||
|
|
||
| all_results: dict[str, Any] = { | ||
| "summary": { | ||
| "total_files": len(pdf_paths), | ||
| "total_runs": len(pdf_paths) * len(engines), | ||
| } | ||
| } | ||
|
|
||
| for engine_name in engines: | ||
| if engine_name not in OCR_ENGINES: | ||
| raise ValueError(f"Unknown engine: {engine_name}") | ||
|
|
||
| print(f"--- Running benchmark for engine: {engine_name} ---") | ||
| engine_runner = OCR_ENGINES[engine_name] | ||
| engine_results: dict[str, Any] = { | ||
| "success": 0, | ||
| "failed": 0, | ||
| "timed_out": 0, | ||
| "results": {}, | ||
| } | ||
|
|
||
| for pdf_path in pdf_paths: | ||
| start_time = time.monotonic() | ||
| try: | ||
| result = engine_runner(pdf_path) | ||
| duration = time.monotonic() - start_time | ||
| engine_results["success"] += 1 | ||
| engine_results["results"][pdf_path.name] = { | ||
| "status": "success", | ||
| "duration": round(duration, 2), | ||
| **result, | ||
| } | ||
| print(f" [SUCCESS] {pdf_path.name} in {duration:.2f}s") | ||
|
|
||
| except (RuntimeError, subprocess.CalledProcessError) as e: | ||
| duration = time.monotonic() - start_time | ||
| engine_results["failed"] += 1 | ||
| engine_results["results"][pdf_path.name] = { | ||
| "status": "failed", | ||
| "duration": round(duration, 2), | ||
| "error": str(e), | ||
| } | ||
| print(f" [FAILED] {pdf_path.name} in {duration:.2f}s") | ||
|
|
||
| except subprocess.TimeoutExpired: | ||
| duration = time.monotonic() - start_time | ||
| engine_results["timed_out"] += 1 | ||
| engine_results["results"][pdf_path.name] = { | ||
| "status": "timed_out", | ||
| "duration": round(duration, 2), | ||
| } | ||
| print(f" [TIMED OUT] {pdf_path.name} after {duration:.2f}s") | ||
|
Comment on lines
+80
to
+108
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# run_mineru 내부에서 어떤 예외 타입이 실제로 발생하는지 재확인
rg -nP --type=py -C2 'raise (HTTPException|FileNotFoundError|RuntimeError|subprocess\.)' src/newsdom_api/mineru_runner.pyRepository: Seongho-Bae/newsdom-api Length of output: 1457 🏁 Script executed: cat -n tools/benchmark_ocr.py | head -150Repository: Seongho-Bae/newsdom-api Length of output: 5983 mineru 엔진의 실제 예외가 여기서 포착되지 않아 벤치마크가 중단될 수 있습니다.
현재 핸들러 가장 견고한 해법은 🤖 Prompt for AI Agents |
||
|
|
||
| all_results[engine_name] = engine_results | ||
|
|
||
| output_path.write_text( | ||
| json.dumps(all_results, ensure_ascii=False, indent=2), encoding="utf-8" | ||
| ) | ||
| print(f"\nBenchmark results written to {output_path}") | ||
|
|
||
|
|
||
| def main(argv: list[str] | None = None) -> None: | ||
| """ | ||
| Main entry point for the OCR benchmark script. | ||
|
|
||
| Parses command-line arguments and initiates the benchmark run. | ||
|
|
||
| Args: | ||
| argv: Optional list of command-line arguments for testing. | ||
| """ | ||
| parser = argparse.ArgumentParser( | ||
| description="Run OCR benchmark against a directory of PDFs." | ||
| ) | ||
| parser.add_argument( | ||
| "--fixtures-dir", | ||
| type=Path, | ||
| required=True, | ||
| help="Directory containing PDF files.", | ||
| ) | ||
| parser.add_argument( | ||
| "--output", | ||
| type=Path, | ||
| required=True, | ||
| help="Path to write the JSON results file.", | ||
| ) | ||
| parser.add_argument( | ||
| "--engines", | ||
| nargs="+", | ||
| default=["mineru"], | ||
| help=f"Space-separated list of engines to benchmark. Available: {', '.join(OCR_ENGINES.keys())}", | ||
| ) | ||
| args = parser.parse_args(argv) | ||
|
|
||
| run_benchmark(args.fixtures_dir, args.output, args.engines) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
액션들이 SHA로 핀되어 있지 않아
quality-gate파이프라인이 실패합니다.리포지토리 정책상 워크플로우의 모든 액션은 SHA로 핀되어야 합니다(
test_workflow_actions_are_pinned_by_sha실패). 현재actions/checkout@v4,docker/setup-buildx-action@v3,docker/login-action@v3,docker/build-push-action@v5, 그리고 하단의actions/checkout@v4모두 태그로만 참조되고 있어 이 중 첫 번째가 보고되어 있고 나머지도 순차적으로 실패할 것입니다. 각 액션을 공식 릴리스의 커밋 SHA로 교체하고, 가독성을 위해 태그를 주석으로 남겨 주세요.🔒 예시 (버전 주석 유지)
Also applies to: 23-23, 26-26, 37-37, 56-56
🤖 Prompt for AI Agents