Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .cursor/environment.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"name": "Codec Carver",
"install": "bash .cursor/install.sh",
"start": "bash .cursor/start.sh",
"ports": [
{ "name": "web", "port": 8000 }
]
}
41 changes: 41 additions & 0 deletions .cursor/install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# Idempotent Cloud Agent setup for codec-carver.
# Runs after each checkout. Always resolve the repository root from this file
# so a cached or non-root working directory cannot pip/cargo the wrong tree.
set -euo pipefail
cd "$(dirname "$0")/.."
export PATH="${HOME}/.local/bin:${PATH}"

# ffmpeg/ffprobe are required at runtime for probing and conversion.
# The default Cloud Agent base image already ships them; install only if missing.
if ! command -v ffmpeg >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
fi

# rustfmt backs `cargo fmt` (rust-toolchain.toml pins a minimal profile without it).
# Fail closed: CI and local agents both run `cargo fmt --check`.
if ! command -v rustup >/dev/null 2>&1; then
echo "rustup is required so rustfmt can be added for cargo fmt checks." >&2
exit 1
fi
rustup component add rustfmt

# Match CI exactly: install the authenticated hash lock, then link the checked-out
# editable tree without dependency resolution or index access. requirements-dev.txt
# remains an opt-in local fuzzing surface until it has its own reviewed hash lock;
# unattended Cloud Agent bootstrap must not resolve those unhashed requirements.
python3 -m pip install --user --disable-pip-version-check \
--require-hashes -r requirements-lock.txt
python3 -m pip install --user --disable-pip-version-check \
--no-index --no-deps --no-build-isolation -e .

# Rust core binary (codec-carver-core) used by the audio-library CLI.
# A fresh checkout wipes rust-core/target, so (re)build it here.
cargo build --release --manifest-path rust-core/Cargo.toml

# Expose pip --user console scripts (codec-carver, codec-carver-library, ...) in
# non-login shells too. Login shells already pick up ~/.local/bin via ~/.profile.
if ! grep -q '.local/bin' "$HOME/.bashrc" 2>/dev/null; then
printf '\nif [ -d "$HOME/.local/bin" ]; then PATH="$HOME/.local/bin:$PATH"; fi\n' >> "$HOME/.bashrc"
fi
48 changes: 48 additions & 0 deletions .cursor/start.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# Per-boot reconciliation. install is not re-run when a pod boots from a
# prebuilt environment, so anything that must exist on every boot lives here.
# `start` must reach a clear ready-or-fail state: a green exit with a dead
# uvicorn leaves the published :8000 port pointing at nothing.
set -euo pipefail
cd "$(dirname "$0")/.."
export PATH="${HOME}/.local/bin:${PATH}"

WEB_HEALTH_URL="${CODEC_CARVER_WEB_HEALTH_URL:-http://127.0.0.1:8000/health}"
WEB_READY_ATTEMPTS="${CODEC_CARVER_WEB_READY_ATTEMPTS:-40}"
WEB_LOG="${CODEC_CARVER_WEB_LOG:-/tmp/codec-carver-web.log}"

# 1) Ensure the Rust backend binary exists (a fresh checkout wipes rust-core/target).
if [ ! -x rust-core/target/release/codec-carver-core ]; then
cargo build --release --manifest-path rust-core/Cargo.toml
fi

_web_is_ready() {
curl -sf -o /dev/null "${WEB_HEALTH_URL}"
}

# 2) Start the FastAPI SaaS web service (saas_web:app) if it is not already up.
# Backgrounded so `start` returns after readiness; the probe keeps it a single
# instance on every boot mode (just-in-time or prebuilt build/snapshot).
if _web_is_ready; then
exit 0
fi

nohup python3 -m uvicorn saas_web:app --host 0.0.0.0 --port 8000 \
> "${WEB_LOG}" 2>&1 &
web_pid=$!

attempt=0
while [ "${attempt}" -lt "${WEB_READY_ATTEMPTS}" ]; do
if _web_is_ready; then
exit 0
fi
if ! kill -0 "${web_pid}" 2>/dev/null; then
echo "uvicorn exited before becoming ready; see ${WEB_LOG}" >&2
exit 1
fi
attempt=$((attempt + 1))
sleep 0.25
done

echo "uvicorn did not become ready after ${WEB_READY_ATTEMPTS} probes; see ${WEB_LOG}" >&2
exit 1
11 changes: 11 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,17 @@ repo.
or feature-extraction path.
<!-- END cwl-agent-guidance -->

## Cloud Agent environment

`.cursor/environment.json` is repository-managed and overrides dashboard
environments. `install` (`bash .cursor/install.sh`) must stay idempotent,
hash-lock Python deps, and fail if `rustfmt` cannot be added. `start`
(`bash .cursor/start.sh`) must wait on `GET /health` before exiting 0; do not
`nohup` uvicorn and return. Do not add `$schema` to the JSON. Decision record:
[`docs/doctoring/cloud-agent-environment.md`](docs/doctoring/cloud-agent-environment.md).
The module map, job-store ERD, and connector order live in
[`ARCHITECTURE.md`](ARCHITECTURE.md).

## Code-owner review gates — disabled (on hold)

As of 2026-08-04, code-owner review requirements (`require_code_owner_reviews` in branch
Expand Down
134 changes: 134 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Architecture

Codec Carver is the speech/video conversion module for STT and omni-modal LLM
input. It must run **standalone** (CLI, FastAPI upload service, MCP tool) and
as a **git submodule** consumed by naruon and sibling ContextualWisdomLab
services.

## Runtime surfaces

```mermaid
flowchart LR
subgraph standalone [Standalone]
CLI["codec-carver CLI"]
WEB["saas_web FastAPI"]
MCP["mcp_driver shrink_media"]
end
subgraph core [Conversion core]
MS["media_shrinker.convert_file"]
RS["rust-core codec-carver-core"]
end
CLI --> MS
WEB --> MS
MCP --> MS
CLI --> RS
WEB --> JOB["job_store SQLite WAL"]
MS --> FF["ffmpeg / ffprobe"]
```

| Surface | Module | Buyer action |
| --- | --- | --- |
| CLI | `media_shrinker.py` | Carve a folder of recordings into size-capped FLAC/Opus. |
| Library CLI | `audio_library.py` → Rust | Inventory, hash, TMK, and GPU transcription batches. |
| Web | `saas_web.py` | Upload one file or a batch; poll `/jobs/{id}` for long work. |
| MCP | `mcp_driver.py` | Call `shrink_media` from an agent runtime. |
| Jobs | `job_store.py` | Survive process restart; do not keep results only in RAM. |

Sources are never overwritten. Generated names keep the original filename plus
a new suffix (`meeting.wav.flac`).

## Conversion pipeline

```mermaid
flowchart TD
A[find_candidates] --> B[probe_media / ffprobe]
B --> C{duration over cap?}
C -->|yes| D[silencedetect + build_segments]
C -->|no| E[single segment]
D --> F[build_audio_plan FLAC]
E --> F
F --> G{_execute_plan}
G -->|FLAC over target| H[build_opus_plan]
H --> G
G --> I[preserve_file_attributes]
I --> J[write_report JSON]
```

Silence splits prefer long quiet intervals; a hard split just under the
duration cap is the fallback. Parsers of ffmpeg/ffprobe text
(`parse_silencedetect_intervals`, `_parse_probe_payload`) treat that text as
untrusted input and raise `MediaShrinkerError` rather than unexpected types.

## Cloud Agent environment

`.cursor/environment.json` is the highest-precedence environment source for
Cloud Agents. Do not add `$schema` (the public schema rejects undeclared
fields).

| Phase | Script | Must do |
| --- | --- | --- |
| `install` | `.cursor/install.sh` | `cd` to repo root; hash-locked pip; fail-closed rustfmt; release-build Rust. |
| `start` | `.cursor/start.sh` | Rebuild Rust only if the binary is missing; start uvicorn; **wait on `GET /health`**. |
| ports | `8000` / `web` | Bind `0.0.0.0:8000` so the published port reaches the SaaS UI. |

Decision record: [`docs/doctoring/cloud-agent-environment.md`](docs/doctoring/cloud-agent-environment.md).

## Job-store data (current)

`job_store.py` persists async conversions. Table `jobs` is a single-word name
and is **known debt**: the org naming rule requires two-or-more-word
snake_case objects (`conversion_jobs`) plus a 3NF-preserving rename migration.
Do not add more single-word tables. Columns already use multi-word names where
they store paths (`output_path`, `output_name`, `temp_dir`).

```mermaid
erDiagram
JOBS ||--o| CONVERSION_OUTPUT : produces
JOBS {
text id PK
text status
text created_at
text updated_at
text output_path
text output_name
text error
text temp_dir
}
```

Callers pass `now` explicitly. The store never calls `datetime.now()`.

## Security and operability baseline

- ffmpeg/ffprobe: `-nostdin` and `-protocol_whitelist file,crypto,data`.
- Uploaded names: sanitize to a safe basename; normalize Windows separators.
- Permissions copied onto outputs drop setuid/setgid/sticky bits.
- Opt-in API keys (`CODEC_CARVER_API_KEYS`) are a **known deviation**. Runtime
reads must move to a SQLite/KV credential registry; env is bootstrap
transport only (issues #329, #373).
- `GET /` and `GET /health` stay reachable without a key so a buyer can open
the form and a probe can confirm liveness.
- PII in recordings is the product. Do not mask audio content; protect it with
access control, retention, and encryption at rest instead.

## Ecosystem connectors (leverage order)

1. **naruon** — consume carved FLAC/Opus + JSON reports as speech/DOM input.
2. **contextual-orchestrator** — route transcription/LLM describe calls; do not
read provider keys from raw env at runtime.
3. **wardnet** — place the SaaS UI behind the org WAF/APIM when exposed.
4. **keyverse** — replace shared API keys with passwordless service identity.
5. **clearfolio / newsdom-api** — attach transcripts and reports to documents.

Each connector must keep this repo independently runnable.

## Test and research baseline

- Unit/integration: `python3 -m unittest discover -s tests -v` (100% coverage
over the configured sources; 100% interrogate on production modules).
- Reality checks: WAV → FLAC lossless; tiny `--target-bytes` → Opus; silence
splits on long fixtures; Rust inventory SHA-256.
- Fuzz: Atheris harnesses in `fuzz/` plus Hypothesis mirrors in
`tests/test_fuzz_properties.py`.
- Speech/codec changes attach APA 7th citations under `docs/doctoring/` and
`docs/papers/` (PDF only when redistribution is permitted).
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## [Unreleased]
### Added
- Cloud Agent `install`/`start` 스크립트가 해시 잠금 pip, fail-closed rustfmt, `GET /health` 준비 대기를 강제합니다. 근거는 [`docs/doctoring/cloud-agent-environment.md`](docs/doctoring/cloud-agent-environment.md)에 있습니다.
- `GET /health` 활성 프로브를 추가해 업로드 페이지와 같이 API 키 없이 프로세스 생존을 확인할 수 있습니다.
- 다중 파일 업로드 선택 시 즉각적인 파일 개수 피드백 및 제한 초과 경고 메시지 추가
- 일괄 업로드 폼에 대상 바이트 프리셋 버튼과 총 파일 크기 미리보기를 추가하여 사용성을 개선했습니다.
- 클라이언트 측 폼 검증 시 하드코딩된 '5 GiB' 텍스트를 동적으로 변환되도록 수정하고 일괄 업로드 폼에 최대 크기(MAX_UPLOAD_BYTES) 검증 피드백을 추가했습니다.
Expand Down
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,10 @@ CI installs pinned dependencies with `pip install --require-hashes -r requiremen
Four flat top-level modules (declared as `py-modules` in `pyproject.toml`; there is no package directory):

- **`media_shrinker.py`** — the core engine and CLI, deliberately stdlib-only (external work happens in `ffmpeg`/`ffprobe` subprocesses). The console script `codec-carver` maps to `media_shrinker:main`. Pipeline for a batch run: `find_candidates` scans the root (pruned `os.walk`, excludes the output dir and `--exclude-dir-prefix` dirs) → per file, `convert_file` probes with ffprobe (`probe_media` / `_parse_probe_payload`), detects silence and builds a split plan for long sources (`detect_silence_intervals`, `parse_silencedetect_intervals`, `build_segments`) → each segment gets a `ConversionPlan` (`build_audio_plan` prefers FLAC; `build_opus_plan` is the fallback when a FLAC output exceeds the target size) → `_execute_plan` runs ffmpeg and `preserve_file_attributes` restores permissions/timestamps/xattrs best-effort → `write_report` emits a JSON report. `convert_file(source, root=..., output_dir=..., target_bytes=...)` is the programmatic API that the web and MCP layers call.
- **`saas_web.py`** — single-file FastAPI upload UI (the `[web]` extra; what the Docker image serves). Streams one upload into a temp workspace, calls `media_shrinker.convert_file`, and returns the first generated output as a download. Middleware enforces a 5 GiB upload cap and security headers. Processing is synchronous per request.
- **`saas_web.py`** — single-file FastAPI upload UI (the `[web]` extra; what the Docker image serves). Streams one upload into a temp workspace, calls `media_shrinker.convert_file`, and returns the first generated output as a download. `GET /health` is the liveness probe used by Cloud Agent `start` and load balancers. Long work uses `/jobs` backed by `job_store`. Middleware enforces a 5 GiB upload cap, optional API keys, and security headers.
- **`mcp_driver.py`** — FastMCP server (the `[mcp]` extra) exposing a single `shrink_media` tool that wraps `convert_file`.
- **`job_store.py`** — stdlib-only SQLite (WAL) durable job store intended for async/worker job tracking. It is tested but not yet wired into `saas_web.py`. Callers pass `now` explicitly; the store never calls `datetime.now()` itself.
- **`job_store.py`** — stdlib-only SQLite (WAL) durable job store used by `saas_web.py` async jobs. Callers pass `now` explicitly; the store never calls `datetime.now()` itself.
- **`.cursor/`** — repository-managed Cloud Agent environment. `install.sh` hash-locks pip and fail-closes rustfmt; `start.sh` waits on `GET /health`. See `ARCHITECTURE.md` and `docs/doctoring/cloud-agent-environment.md`.

Supporting directories: `fuzz/` holds Atheris harnesses plus seed corpora for the three untrusted-input parsing surfaces (`parse_silencedetect_intervals`, `_parse_probe_payload`, `build_segments`); the same invariants run as Hypothesis property tests in `tests/test_fuzz_properties.py` so they execute in the normal suite. `docs/papers/` holds the fuzzing survey the harness design references.

Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ USER appuser

EXPOSE 8000

# Liveness: the FastAPI app serves the upload UI at "/".
# Liveness: use the same cheap JSON readiness surface as Cloud Agent startup.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/', timeout=4).status==200 else 1)"
CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4).status==200 else 1)"

CMD ["uvicorn", "saas_web:app", "--host", "0.0.0.0", "--port", "8000"]
68 changes: 68 additions & 0 deletions docs/doctoring/cloud-agent-environment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Cloud Agent environment bootstrap

## Decision

Codec Carver commits a repository-managed Cloud Agent environment at
`.cursor/environment.json`. `install` installs hash-locked Python dependencies,
adds `rustfmt` to the pinned minimal Rust toolchain, and builds
`codec-carver-core`. `start` rebuilds that binary only when a fresh checkout
wiped `rust-core/target`, then brings up `saas_web:app` on `0.0.0.0:8000` and
**waits** until `GET /health` succeeds.

Agents should treat a green `start` as “the upload UI is listening.” A
`nohup` without a readiness wait is not an environment start.

## Technical basis

Cursor resolves environment configuration from the checked-out
`.cursor/environment.json` before any dashboard-managed personal or team
environment (Cursor, n.d.). The public schema rejects undeclared fields,
including `$schema`, so this repository does not add one.

Python package integrity follows pip hash-checking mode: install the lock
file with `--require-hashes`, then link the editable tree with `--no-index
--no-deps` so the index cannot substitute a different wheel (Python Software
Foundation, n.d.). That pairing is the same contract CI uses and is the
minimum SLSA-aligned control this repository can apply without a hermetic
build service (SLSA, 2023).

Fail-closed toolchain setup follows NIST SSDF PW.4 / PW.8: produce a
repeatable build environment and do not continue when a required verification
tool (`rustfmt`) is missing (Souppaya et al., 2022).

`GET /health` is a liveness probe, not a substitute for job-store or ffmpeg
readiness. Load balancers and Cloud Agent `start` share that URL so a crashed
uvicorn cannot look like a successful boot.

## Verification and rollback

- `bash -n .cursor/install.sh .cursor/start.sh` must succeed.
- `tests/test_cloud_agent_environment.py` locks the JSON fields, the hash-locked
pip lines, rustfmt fail-closed install, and the `/health` wait.
- `GET /health` stays auth-exempt and returns
`{"status": "ok", "service": "codec-carver"}` whether or not API keys are
configured.
- Roll back by restoring the previous scripts; do not reintroduce `|| true` on
rustfmt or a `nohup` that returns before the probe succeeds.

## Next action

After this lands, mark the Cloud Agent environment draft ready for review and
close the earlier environment-only PR that lacked the readiness contract.

## References

Cursor. (n.d.). *Cloud Agents environment schema*. Retrieved August 16, 2026,
from https://cursor.com/schemas/environment.schema.json

Python Software Foundation. (n.d.). *Secure installs: Hash-checking mode*
(pip documentation). Retrieved August 16, 2026, from
https://pip.pypa.io/en/stable/topics/secure-installs/

SLSA. (2023). *Supply-chain Levels for Software Artifacts (SLSA) v1.0*.
https://slsa.dev/spec/v1.0/

Souppaya, M., Scarfone, K., & Dodson, D. (2022). *Secure Software Development
Framework (SSDF) Version 1.1: Recommendations for mitigating the risk of
software vulnerabilities* (NIST Special Publication 800-218). National
Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-218
Loading
Loading