feat: butler/drill 影子双写 + 三源统一查询（W1-B2 后半，IR-0006） #430
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| #!/usr/bin/env bash | ||
| # evidence-query.sh —— 三源统一证据查询（IR-0006 W1-B2 / BEH-03 / ADR-0103，AC-4a） | ||
| # | ||
| # 一条命令跨三源拉取 schema v1 影子账本、逐源验链（fail-closed：链断=红）、 | ||
| # 按时间归并输出统一 JSONL（stdout）+ 分源统计（stderr）： | ||
| # 源 1 metering Cloudbird-Software/CI-Workflows @ metering-ledger shadow-evidence-*.jsonl（根） | ||
| # 源 2 drill Cloudbird-Software/.github @ drill-ledger governance/drill/shadow-evidence.jsonl | ||
| # 源 3 butler Cloudbird-Software/.github @ butler-ledger governance/butler/shadow-evidence.jsonl | ||
| # | ||
| # 用法: | ||
| # bash governance/evidence-query.sh [--card owner/repo#n] [--json] # --json=汇总行也走 stdout | ||
| # env: | ||
| # GH_TOKEN 必填（读私有仓 contents；org token 或对两仓可读的 PAT） | ||
|
There was a problem hiding this comment. 1. evidence-query accepts arbitrary pats The new query script requires an externally supplied GH_TOKEN and explicitly permits an org token or PAT before making direct gh api calls. This bypasses the repository-standard GitHub App identity and credential helper required for GitHub operations. Agent Prompt
|
||
| # 退出码: 0=查询成功（输出统一 JSONL）| 2=参数/环境 | 3=任一源链断（fail-closed， | ||
| # 不可信数据不出结果——宁红勿假） | ||
| set -uo pipefail | ||
| DIR="$(cd "$(dirname "$0")" && pwd)" | ||
| GH="${GH:-gh}" | ||
|
|
||
| CARD_FILTER=""; JSON_ONLY=0 | ||
| while [[ $# -gt 0 ]]; do | ||
| case "$1" in | ||
| --card) CARD_FILTER="${2:?}"; shift 2 ;; | ||
| --json) JSON_ONLY=1; shift ;; | ||
| *) echo "未知参数 $1（用法见文件头）" >&2; exit 2 ;; | ||
| esac | ||
| done | ||
| [[ -n "${GH_TOKEN:-}" ]] || { echo "GH_TOKEN 未设置（需对两仓 contents 读权限）" >&2; exit 2; } | ||
| command -v "$GH" >/dev/null 2>&1 || { echo "gh 不可用" >&2; exit 2; } | ||
|
|
||
| TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT | ||
| # ---- 拉源（404=该源尚无影子记录：过渡期合法，跳过留痕；其余 API 错=fail-closed） ---- | ||
| fetch_file() { # fetch_file <repo> <branch> <path> <out>；rc=1=源缺席（404，非红） | ||
| local repo="$1" branch="$2" path="$3" out="$4" | ||
| if "$GH" api "repos/$repo/contents/$path?ref=$branch" >"$TMP/api.json" 2>"$TMP/api.err"; then | ||
| python3 - "$TMP/api.json" "$out" <<'PYEOF' | ||
| import base64, json, sys | ||
| d = json.load(open(sys.argv[1], encoding="utf-8")) | ||
| open(sys.argv[2], "w", encoding="utf-8", newline="\n").write( | ||
| base64.b64decode(d["content"]).decode("utf-8")) | ||
| PYEOF | ||
| return 0 | ||
| fi | ||
| if grep -qi 'not found' "$TMP/api.err" 2>/dev/null; then | ||
| return 1 # 源缺席（尚无影子记录）——过渡期合法，非红 | ||
| fi | ||
| echo "FATAL: $repo@$branch $path 拉取失败（非 404）：" >&2; cat "$TMP/api.err" >&2; exit 2 | ||
| } | ||
|
|
||
| SRC_METER="$TMP/metering"; mkdir -p "$SRC_METER" | ||
| if "$GH" api "repos/Cloudbird-Software/CI-Workflows/contents?ref=metering-ledger" >"$TMP/list.json" 2>"$TMP/api.err"; then | ||
| python3 - "$TMP/list.json" "$SRC_METER" <<'PYEOF' | ||
| import base64, json, sys | ||
| for ent in json.load(open(sys.argv[1], encoding="utf-8")): | ||
| if ent["type"] == "file" and ent["name"].startswith("shadow-evidence-") and ent["name"].endswith(".jsonl"): | ||
| open(f"{sys.argv[2]}/{ent['name']}", "w", encoding="utf-8", newline="\n").write( | ||
| base64.b64decode(ent["content"]).decode("utf-8")) | ||
|
Comment on lines
+55
to
+57
There was a problem hiding this comment. 3. Metering source is never fetched The metering directory response is treated as though each entry contains base64 content, but GitHub directory entries only carry file metadata; the decoder raises KeyError and, because its status is unchecked, the query continues successfully with metering silently omitted. The test fixture masks this by adding a non-existent content field to directory entries. Agent Prompt
Comment on lines
+56
to
+57
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🌐 Web query:
💡 Result: 在 GitHub REST API 的 "Get repository content" 接口中，当你获取目录内容时，响应返回的是一个包含目录中所有项（文件和子目录）的数组对象（或在特定媒体类型下为包含 entries 数组的对象）[1][2]。 在该目录列表响应中，每一个子项的描述对象不包含 base64 编码的 Citations:
🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- governance/evidence-query.sh ---'
sed -n '1,130p' governance/evidence-query.sh
printf '%s\n' '--- relevant test fixture ---'
sed -n '100,155p' governance/tests/test-evidence-shadow.sh
printf '%s\n' '--- scoped repository conventions ---'
find /tmp/coderabbit-repo-knowledge/cloudbird-software-github-0dfcc7f5 -maxdepth 2 -type f -name '*.md' -printRepository: Cloudbird-Software/.github Length of output: 9066 必须逐文件获取 metering 分片内容并对解码失败 fail-closed。 GitHub Contents API 的目录项只包含元数据，不包含 🤖 Prompt for AI Agents |
||
| PYEOF | ||
| else | ||
| if ! grep -qi 'not found' "$TMP/api.err" 2>/dev/null; then | ||
| echo "FATAL: metering-ledger 目录拉取失败：" >&2; cat "$TMP/api.err" >&2; exit 2 | ||
| fi | ||
| fi | ||
| DRILL_OK=0; fetch_file "Cloudbird-Software/.github" "drill-ledger" "governance/drill/shadow-evidence.jsonl" "$TMP/drill.jsonl" && DRILL_OK=1 || [[ $? -eq 1 ]] || exit 2 | ||
| BUTLER_OK=0; fetch_file "Cloudbird-Software/.github" "butler-ledger" "governance/butler/shadow-evidence.jsonl" "$TMP/butler.jsonl" && BUTLER_OK=1 || [[ $? -eq 1 ]] || exit 2 | ||
|
|
||
| # ---- 逐源验链 + 归并输出（链断=exit 3：不可信数据不出结果） ---- | ||
| export CARD_FILTER JSON_ONLY DRILL_OK BUTLER_OK | ||
| python3 - "$DIR/evidence_shadow.py" "$SRC_METER" "$TMP/drill.jsonl" "$TMP/butler.jsonl" "$TMP" <<'PYEOF' | ||
| import glob, json, os, sys | ||
|
|
||
| sys.path.insert(0, os.path.dirname(os.path.abspath(sys.argv[1]))) | ||
| import evidence_shadow # noqa: E402 验链与 CI-Workflows 侧同源语义 | ||
|
|
||
| metering_dir, drill_f, butler_f, tmp = sys.argv[2:6] | ||
| sources = {"metering": sorted(glob.glob(os.path.join(metering_dir, "shadow-evidence-*.jsonl"))), | ||
| "drill": [drill_f] if os.environ.get("DRILL_OK") == "1" else [], | ||
| "butler": [butler_f] if os.environ.get("BUTLER_OK") == "1" else []} | ||
| errs, recs = [], [] | ||
| for src, files in sources.items(): | ||
| for f in files: | ||
| if not os.path.isfile(f) or os.path.getsize(f) == 0: | ||
| continue | ||
| errs.extend(evidence_shadow.verify_file(f)) | ||
| for ln in evidence_shadow.read_lines(f): | ||
| recs.append({"source": src, **json.loads(ln)}) | ||
|
Comment on lines
+84
to
+86
There was a problem hiding this comment. 7. Malformed chains bypass exit three The query records JSON parse errors through verify_file but then immediately parses every line again before checking errs, so malformed JSON raises an uncaught exception and exits 1 instead of the promised chain-failure exit 3. Consumers that distinguish trust failures by exit code will not receive the documented fail-closed signal. Agent Prompt
|
||
| if errs: | ||
| for e in errs: | ||
| print(f"CHAIN {e}", file=sys.stderr) | ||
| print("FATAL: 任一源链断——统一查询拒绝出结果（fail-closed，宁红勿假）", file=sys.stderr) | ||
| sys.exit(3) | ||
|
|
||
| card = os.environ.get("CARD_FILTER") or None | ||
| recs.sort(key=lambda r: (r.get("ts", ""), r.get("source"))) | ||
| out = [r for r in recs if not card or r.get("subject", {}).get("card") == card] | ||
| for r in out: | ||
| print(json.dumps(r, ensure_ascii=False, sort_keys=True, separators=(",", ":"))) | ||
|
|
||
| summary = { | ||
| "total": len(out), | ||
| "by_source": {s: sum(1 for r in out if r["source"] == s) for s in ("metering", "drill", "butler")}, | ||
| "by_tenant": {}, | ||
| "by_card_top": {}, | ||
| } | ||
| for r in out: | ||
| t = r.get("subject", {}).get("tenant", "?") | ||
| c = r.get("subject", {}).get("card", "?") | ||
| summary["by_tenant"][t] = summary["by_tenant"].get(t, 0) + 1 | ||
| summary["by_card_top"][c] = summary["by_card_top"].get(c, 0) + 1 | ||
| summary["by_card_top"] = dict(sorted(summary["by_card_top"].items(), key=lambda kv: -kv[1])[:10]) | ||
| line = json.dumps(summary, ensure_ascii=False, sort_keys=True) | ||
| if os.environ.get("JSON_ONLY") == "1": | ||
| print(line) | ||
| else: | ||
| print(f"SUMMARY {line}", file=sys.stderr) | ||
| PYEOF | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2. governance_token writes ledger
📘 Rule violation⛨ SecurityAgent Prompt
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools