fix(ci): stop find | head -1 | grep -q from silently losing matches under pipefail - #1464
fix(ci): stop find | head -1 | grep -q from silently losing matches under pipefail#1464seonghobae wants to merge 5 commits into
Conversation
… pipefail Under set -o pipefail, `find ... | head -1 | grep -q .` races: if find still has buffered matches to write when head -1 reads its one line and closes the pipe, find's next write() fails with SIGPIPE and it exits non-zero. head and grep still exit zero, but pipefail reports the pipeline's exit status as the last non-zero one in pipeline order -- find's -- so the surrounding `if` silently evaluates false even though a match existed, whenever there's enough matching output to overflow the pipe buffer before head closes it (readily reproducible with a few thousand matches). This hit all 8 occurrences across codeql-pr.yml, python-security.yml, and scheduled-security-scan.yml: CodeQL's PR/scheduled-scan language matrix could silently drop Python (or JS/TS, or Java/Kotlin), and Bandit plus the pip-audit project-manifest check could be skipped entirely, on any Python-heavy repository in the org, with no error surfaced anywhere. Replaced every occurrence with `find ... -print -quit | grep -q .`: find stops itself after the first match (or none), so nothing external ever cuts off its output mid-write. This is already this repo's own established idiom for the same check (see test_opencode_agent_contract.py's `find "$destination" -type l -print -quit`). New test extracts the live python-security.yml step body and runs it against a directory with 5,000 matching files -- confirmed failing (has_python=false/has_manifest=false) against the pre-fix workflow content, passing against the fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6UJHYbfbGdHfYPjgbVhAr
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Contextual-Orchestrator와 관계한 것들을 같이 손보든 어쩌든 해결하세요. Bypass merge 필요하면 가능 (chicken and eggs 상황이라면) + NVIDIA NIM 만 쓰는 건 허용하지 않아요. Contextual-Orchestrator를 쓰세요. Timeout은 적어도 3시간으로 잡으세요. 120초 같은 건 당황스럽군요. Strix가 6시간 이상 동작해서 취약점 잡는 것도 본 일이 있습니다. Opencode와 Noema 는 Coderabbitai 및 Devin 수준으로 실제로 리뷰를 하게 하시오. Strix도 보안 리뷰를 꼼꼼하게 하도록 하시오. 특히 보안 리뷰는 전체 코드로 수행하는 것입니다. Contextual-Orchestrator는 실시간으로 빠르면서 능력이 좋은 모델에 요청을 보내어 시간을 당기시오. |
|
Fresh protected-main revalidation at |
Preserve the four-file #1464 repair while incorporating protected main without rebase or force-push. Protected main has not modified these four paths since the branch merge base, so the merge tree is main plus the exact reviewed branch blobs.
|
Current-main convergence completed without rebase or force-push. Exact head is now |
|
Fresh protected-main compare after #1546 is still cleanly scoped: three central workflow detectors plus the dedicated 5,000-file regression, 5 ahead / 1 behind, and GitHub reports mergeable. Protected main still contains the unsafe |
2 participants
Summary
Found while investigating why
Bandit (Python SAST)was showingskippedon.github#1347/.github#1438with no explainable root cause — the job log showedfind: 'standard output': Broken pipe/find: write error, which is a signature ofSIGPIPEkillingfindmid-write.Under
set -o pipefail,find ... | head -1 | grep -q .races: iffindstill has buffered matches to write whenhead -1reads its one line and closes the pipe,find's nextwrite()fails withSIGPIPEand it exits non-zero.headandgrepstill exit zero, butpipefailreports the pipeline's exit status as the last non-zero one in pipeline order —find's — so the surroundingifsilently evaluates false even though a match existed, whenever there's enough matching output to overflow the pipe buffer beforeheadcloses it. Reproduced directly with 5,000 dummy.pyfiles locally:has_pythoncomes backfalse.This hit all 8 occurrences across three central, org-wide workflows:
codeql-pr.yml(3×: JS/TS, Python, Java/Kotlin language-matrix detection)python-security.yml(3×: Python detection, manifest detection, pip-audit project-manifest detection)scheduled-security-scan.yml(2×: JS/TS, Python language-matrix detection)Concretely: CodeQL's PR/scheduled-scan language matrix could silently drop Python (or JS/TS, or Java/Kotlin) from analysis, and Bandit plus the pip-audit project-manifest check could be skipped entirely, on any Python-heavy repository in the org — with no error surfaced anywhere.
Fix
Replaced every occurrence with
find ... -print -quit | grep -q .:findstops itself after the first match (or none), so nothing external ever cuts off its output mid-write — no race. This is already this repo's own established idiom for the same "does at least one match exist" check (seetest_opencode_agent_contract.py'sfind "$destination" -type l -print -quit).Verification
tests/test_workflow_file_detection_pipefail_regression.pyextracts the livepython-security.ymlstep body (not a hand-copied duplicate) and runs it against a directory with 5,000.py/requirements*.txtfiles — confirmed failing (has_python=false/has_manifest=false) against the pre-fix workflow content viagit stash, and passing against the fix.PYTHONPATH=. coverage run -m pytest tests -q→ 2029 passed, 1 skipped, 21 subtests, coverage 100%.python -m interrogate→ 100%.Related
.github#1438/.github#1347's CI state — unrelated to those PRs' own diffs, so filed as its own dedicated, focused fix rather than bundled into either.Generated by Claude Code
Exact-head operational evidence
Exact head:
97f2cbdddb6e01015a6067056b3257fe537a1f69, based on protectedmain@2c69e31373607c3dc9b2d7b4b948a154c47aca65.The dependency repair from #1467 is now integrated on protected main. Independent verification of this exact tree:
74 passed2,098 passed · 1 skipped · 21 subtests10,455/10,4554,164/4,164100%git diff --check: passHosted Python Security now proves the complete causal chain on this exact head:
Detect Python: successBandit (Python SAST): executed and successpip-audit: executed and successExact-head Security Scan, SAST, CodeQL, OSV, SBOM, Secret Scan, Scorecard, and Python Security are terminal-success. Formal exact-current-head review verdicts and required review/security gates remain non-passing until terminal; predecessor evidence does not transfer. Do not merge from source tests or generated security jobs alone.