test(fuzz): add coverage-guided + property-based fuzzing - #47
Conversation
Add fuzzing across the four untrusted-input surfaces CodeGraph surfaced: the HTTP request-body parser/validators (server._coerce_json, _validate_mode/_validate_messages/_reject_unknown_keys), the agent-pool config parser (ModelAgent.from_dict), secret/PII redaction (redact_text/redact_value), and end-to-end orchestration on arbitrary prompt text against mock:// providers (TaskOrchestrator.run + SSE framing). Tooling is permissive-licensed only: - Hypothesis (MPL-2.0) property tests in tests/fuzz/ run in the normal pytest suite on every platform, no native toolchain required. - Atheris (Apache-2.0) coverage-guided libFuzzer harnesses in fuzz/, wired into .github/workflows/fuzz.yml with a bounded per-target budget (60s on PR/push, 300s on the weekly schedule) to keep CI cost low. Both drivers call the same invariant checks in fuzz/targets.py, so a bug found by either reproduces under the other. Seed corpora live in fuzz/corpus/. All targets run fully offline (no network, no secrets). Existing suite stays green (140 -> 147 tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…action seed Resolves 4 GHAS code-scanning alerts on the fuzzing PR: - Alerts 23/24/25 (OpenSSF Scorecard Pinned-Dependencies): the three `pip install` invocations in .github/workflows/fuzz.yml were unpinned. Replace with `pip install --require-hashes -r` against fully hash-locked requirements (fuzz/requirements-property.txt for the py3.12 property-test job, fuzz/requirements-atheris.txt for the py3.11 Atheris job), generated via `uv pip compile --generate-hashes`. Source .in files kept for reproducible recompiles. pip itself is now pinned by hash too. - Alert 22 (Trivy AWS Access Key ID, CRITICAL): the redaction fuzz seed fuzz/corpus/redaction/multi_secret.txt held an AKIA-format synthetic key. The redactor keys off the `token=`/`password:` prefix (12+ chars), not the AWS format, so the AKIA value added zero coverage. Swap it for a non-AWS placeholder that matches the exact same SECRET_PATTERNS regex path — full coverage preserved, no secret-shaped string committed. tests/fuzz property suite: 7 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head4ab294091cafd5159f8b3ec55c16102e7489658f. -
Head SHA:
4ab294091cafd5159f8b3ec55c16102e7489658f -
Workflow run: 28981783767
-
Workflow attempt: 1
Coverage evidence
Coverage Evidence
- Head SHA:
4ab294091cafd5159f8b3ec55c16102e7489658f - Required test evidence: supported repository test suites must pass.
- Required docstring evidence: repository-owned docstring gates must pass when configured; otherwise docstring coverage is advisory.
Python project dependencies (.)
Using CPython 3.12.3 interpreter at: /usr/bin/python3
Creating virtual environment at: .venv
Resolved 28 packages in 336ms
Checked in 0.00ms
- Result: PASS
Python coverage with missing-line report (.)
Downloading pygments (1.2MiB)
Downloaded pygments
Installed 6 packages in 12ms
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0
rootdir: /home/runner/work/contextual-orchestrator/contextual-orchestrator/pr-head
configfile: pyproject.toml
collected 149 items / 1 error
==================================== ERRORS ====================================
_____________ ERROR collecting tests/fuzz/test_fuzz_properties.py ______________
ImportError while importing test module '/home/runner/work/contextual-orchestrator/contextual-orchestrator/pr-head/tests/fuzz/test_fuzz_properties.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/fuzz/test_fuzz_properties.py:17: in <module>
from hypothesis import given, settings, strategies as st
E ModuleNotFoundError: No module named 'hypothesis'
=========================== short test summary info ============================
ERROR tests/fuzz/test_fuzz_properties.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.88s ===============================
- Result: FAIL (exit 2)
Python docstring coverage advisory
RESULT: PASSED (minimum: 80.0%, actual: 88.7%)
- Result: PASS
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: fuzz.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: fuzz.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file (33 files)"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file (33 files)"]
R2 --> V2["required checks"]
Evidence --> S3["Docs (2 files)"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs (2 files)"]
R3 --> V3["docs review"]
Evidence --> S4["Test: test_fuzz_properties.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_fuzz_properties.py"]
R4 --> V4["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage Evidence
Python project dependencies (.)
Python coverage with missing-line report (.)
Python docstring coverage advisory
Coverage Decision
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: fuzz.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: fuzz.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file (33 files)"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file (33 files)"]
R2 --> V2["required checks"]
Evidence --> S3["Docs (2 files)"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs (2 files)"]
R3 --> V3["docs review"]
Evidence --> S4["Test: test_fuzz_properties.py"]
S4 --> I4["regression suite"]
I4 --> R4["Review risk: Test: test_fuzz_properties.py"]
R4 --> V4["targeted test run"]
|
|
Closing as superseded by the fuzz/security work already merged to main via later PRs. The remaining failures on this stale head are real but obsolete: coverage-evidence could not collect tests because Hypothesis was not installed for the evidence run, and Atheris failed because this branch pins unavailable atheris==3.1.0. Current main carries the corrected fuzz workflow and corpus, uses atheris==3.0.0, keeps upload-artifact on v5, includes the unhashable-role regression coverage, and passed the full local suite during the queue refresh. Keeping this stale PR open would keep failing checks around and risks reverting newer main work if merged. |
2 participants
What
Adds fuzzing to the highest-value untrusted-input surfaces, located with CodeGraph (
codegraph explore "parse decode deserialize request config validate untrusted input JSON payload", pluscodegraph explore "load_agents parse agent config"and the redaction/orchestration follow-ups):server._coerce_json/_reject_unknown_keys/_validate_mode/_validate_messagesorchestrator.ModelAgent.from_dictorchestrator.redact_text/redact_value(regex + recursion; idempotence invariant)orchestrator.TaskOrchestrator.runon arbitrary prompt text againstmock://providers, incl. SSE framing round-tripEach target asserts no unhandled crash / only documented exceptions / structural invariants on arbitrary input.
Tools (permissive licenses only)
tests/fuzz/), run in the normal pytest suite on every platformfuzz/), bounded CI job on Python 3.11Both drivers call the same invariant checks in
fuzz/targets.py, so a bug found by either reproduces under the other.CI
.github/workflows/fuzz.yml:Actions pinned by SHA to match repo convention.
Notes
os.getenv), no DB.fuzz/corpus/<target>/(valid + malformed + Unicode/Korean cases).pytest -q).docs/papers/fuzzing-art-science-engineering-manes-2019.pdf(open-access arXiv survey).docs/fuzzing.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P