feat(governance): cost-check LLM 预算通道（W2-C3 .github#216，ADR-0062） - #231
Merged
Conversation
…github#216，ADR-0062） - policy llm_tokens：data_source pending→ciw-metering + metering 定位三件套 （repo/branch/code_path，复用现有结构不新建配置面；pending 保留为回滚值） - cost-check.sh：新增 llm_channel_account 通道（标记对可提取单测）——拉 metering-ledger 分支 tarball → CI-Workflows metering.py aggregate 按角色档 归账（先验链后归账）；DATA/ZERO/INFRA 三态：链断=INFRA fail-closed（不可信 数据不入账不盲熔断）、分支未建/空账本=ZERO 零用量声明；告警/硬停档 issue 正文带当月归账与角色档分布 - 顺带修复（同路径潜在缺陷，e2e 实测暴露）：org 变量读取成功且 value=false （人工复位后常态）曾被判'非 404 读取失败'落 INFRA——复位确认路径每小时误红； 现按退出码判读。告警档 ACT 行此前只报 Actions 百分比（token 单独触发时误导） - cost-check.yml：sparse checkout CI-Workflows pipeline/metering（归账引擎， 该仓公开且自身 CI+org-gate 把关；破坏=显式 INFRA 红） - governance/tests/test-cost-llm-channel.sh：标记对提取 + 桩 metering/桩 gh 六 形态单测（gate.yml 自动纳入）；本地 e2e：真实账本 165 token 走通 66%绿/ 82.5%告警/100%硬停三档
|
Warning Review limit reached
Next review available in: 4 minutes Limit details: You’ve used all 10 included reviews currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Comment |
PR Summary by Qodofeat(governance): account LLM token budget via CIW metering ledger
AI Description
Diagram
High-Level Assessment
Files changed (4)
|
Code Review by Qodo
1. API失败被判ZERO
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+52
to
+55
| # 归账引擎 sparse checkout（W2-C3 .github#216，ADR-0062）：CI-Workflows 是公开仓且 | ||
| # 自身有 CI+org-gate（C1），跟 main 消费；升级若破坏 aggregate 会让本检查显式 | ||
| # INFRA 红（fail-closed 可见），不会被静默吞掉 | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 |
There was a problem hiding this comment.
1. Missing card: metadata line 📘 Rule violation § Compliance
The PR description body does not include the required single Card: <owner>/<repo>#<n> line, so downstream tooling cannot reliably associate this change with its tracking item.
Agent Prompt
## Issue description
PR description is missing the required card metadata line in the exact format `Card: <owner>/<repo>#<n>`.
## Issue Context
This PR already references `.github#216` / `W2-C3 (.github#216)` in text, but it must be provided as a dedicated `Card:` line (e.g., `Card: Cloudbird-Software/.github#216`).
## Fix Focus Areas
- .github/workflows/cost-check.yml[52-55]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+155
to
+158
| if ! "$GH" api "repos/$LT_M_REPO/tarball/$LT_M_BRANCH" >"$led.tar.gz" 2>/dev/null; then | ||
| if "$GH" api "repos/$LT_M_REPO/branches/$LT_M_BRANCH" >/dev/null 2>&1; then | ||
| printf 'INFRA\tmetering 账本分支存在但 tarball 拉取失败（%s@%s）\n' "$LT_M_REPO" "$LT_M_BRANCH" | ||
| else |
There was a problem hiding this comment.
2. Api失败被判zero 🐞 Bug ☼ Reliability
llm_channel_account() 在 tarball 拉取失败后，用 gh api .../branches/... 的成功与否来判断分支是否存在，但对 branches 查询的非 404 失败（限流/网络/权限）会被当作“分支未建”并输出 ZERO，导致用量被静默归零。该路径违背本 PR 声明的 fail-closed（不可信数据不入账），会造成预算检查漏报/假绿。
Agent Prompt
### Issue description
`llm_channel_account()` 在 tarball 拉取失败后，用 `gh api repos/.../branches/...` 作为“分支是否存在”的判据，但当前写法把 branches 查询的任何失败都当成“分支不存在”，从而输出 `ZERO`。
这会在 GitHub API 限流/网络抖动/权限异常等场景下把未知状态 fail-open 成 0 用量，导致 LLM 通道预算检查漏报。
### Issue Context
- 该通道在 PR 描述中强调：tarball 失败/链断/引擎缺失应为 INFRA（exit 2 fail-closed）。
- 但当前分支存在性探测没有区分 404 vs 其他错误。
### Fix Focus Areas
- governance/cost-check.sh[155-160]
### Suggested change
- 捕获 branches 查询的 stdout/stderr 与退出码：
- rc==0 => 分支存在（tarball 失败应为 INFRA）
- rc!=0 且输出包含 Not Found/404 => 分支不存在（ZERO）
- 其他任何失败 => INFRA（“无法确认分支是否存在/不可判定”，fail-closed）
- 同时建议不要吞掉 tarball 拉取失败的错误信息（至少截断输出用于诊断）。
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+163
to
+166
| if ! tar -xzf "$led.tar.gz" -C "$led" --strip-components=1 --wildcards \ | ||
| "*/$LT_M_CODE/records-*.jsonl" 2>/dev/null; then | ||
| printf 'INFRA\tmetering 账本 tar 解包失败（strip-components=1 + %s/records-*.jsonl）\n' "$LT_M_CODE" | ||
| return 0 |
There was a problem hiding this comment.
3. 空账本被判infra 🐞 Bug ≡ Correctness
llm_channel_account() 使用 tar 的 wildcard 解包 records-*.jsonl，当分支已建但尚无任何 records 文件时，tar 会因“无匹配成员/Not found in archive”失败并被当作 INFRA。该场景在 PR 设计中应归类为 ZERO（零用量声明），否则会导致通道上线初期或月初窗口持续误报基础设施故障。
Agent Prompt
### Issue description
当前 tar 解包逻辑要求归档中必须存在 `*/$LT_M_CODE/records-*.jsonl`，否则直接返回 INFRA。若 metering-ledger 分支已创建但尚未产生任何 records 文件（空账本），这会把“零用量”误判为 INFRA。
### Issue Context
脚本后续已把 `metering.py aggregate` 的 rc=2 视为 ZERO（无周片/零用量）。但在真实数据源路径下，tar 阶段就可能因无 records 文件而提前退出，导致永远到不了 aggregate 的 rc=2 分支。
### Fix Focus Areas
- governance/cost-check.sh[163-167]
### Suggested change
- 在 tar 前先 `tar -tzf` 列出并检查是否存在匹配 records：
- 若无匹配：不要报 INFRA，直接让 ledger 目录保持为空并继续执行 aggregate，让其以 rc=2 输出 ZERO；或直接输出 ZERO（带说明“分支已建但无 records 文件”）。
- 若有匹配但解包失败：仍按 INFRA 处理。
- 或者捕获 tar 的错误输出，仅当错误为“Not found in archive/无匹配”时转 ZERO，其余错误转 INFRA。
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+56
to
+60
| with: | ||
| repository: Cloudbird-Software/CI-Workflows | ||
| sparse-checkout: pipeline/metering | ||
| path: ciw-metering | ||
| persist-credentials: false |
There was a problem hiding this comment.
4. 外部仓checkout未固定ref 🐞 Bug ⛨ Security
workflow 直接 checkout Cloudbird-Software/CI-Workflows 默认分支来获取 metering.py，未固定到特定 commit/tag，导致成本熔断行为随外部仓 HEAD 变化而漂移，存在供应链与可重复性风险。若外部仓被误推/破坏，预算检查可能被绕过或频繁误报。
Agent Prompt
### Issue description
`actions/checkout` 拉取外部仓 `Cloudbird-Software/CI-Workflows` 时未指定 `ref`，会默认跟随外部仓默认分支最新提交。预算归账/验链的执行逻辑因此不可复现且易受外部变更影响。
### Issue Context
该 PR 将 metering.py 作为“预算通道”的关键执行组件，且 cost-check 拥有治理 token（GH_TOKEN）。即使外部仓有 CI，也仍建议对关键治理逻辑固定版本并通过显式升级流程更新。
### Fix Focus Areas
- .github/workflows/cost-check.yml[55-60]
### Suggested change
- 在 checkout CI-Workflows 步骤增加 `ref: <pinned sha or tag>`（推荐 sha），并在需要升级时通过 PR 明确更新该 ref。
- 若确实要跟随 main，也建议至少在 policy/脚本中记录期望版本并加 drift-check 校验（否则治理行为不可追溯）。
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+319
to
320
| W_TITLE="成本告警（$YEAR-$MONTH）：Actions 分钟 ${PCT_MIN}% / LLM token ${PCT_TOK:--}%（阈值 ${AM_WARN}%）" | ||
| W_EXISTING=$(gov_open_issues cost-budget-warning | grep -m1 "$YEAR-$MONTH" | cut -f1) |
There was a problem hiding this comment.
5. 告警阈值字段混用 🐞 Bug ≡ Correctness
告警档标题与日志同时展示 Actions 与 LLM token，但括号内阈值仍使用 AM_WARN（Actions 阈值）而非 LT_WARN（LLM 阈值），当两者配置不一致时会误导值守与审计。该问题会导致告警信息描述与实际判定条件不一致。
Agent Prompt
### Issue description
`W_TITLE` / `act` 文案在同时展示 Actions 与 LLM 的百分比时，仍把“阈值”字段写成 `${AM_WARN}%`。如果未来 Actions 与 LLM 的 warn_pct 不同，会出现：判定用 LT_WARN，但文案展示 AM_WARN 的不一致。
### Issue Context
脚本已分别计算 `WARN_MIN` vs `WARN_TOK`（LT_WARN），因此文案应与之对齐。
### Fix Focus Areas
- governance/cost-check.sh[318-333]
### Suggested change
- 标题/日志中明确区分阈值：例如 `（阈值 Actions ${AM_WARN}% / LLM ${LT_WARN}%）`。
- 或者将“阈值”文字改成更中性的描述，并在正文里分别列出两套阈值（当前正文已部分包含）。
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
There was a problem hiding this comment.
Pull request overview
This PR implements the “LLM token budget channel” consumption side for governance/cost-check.sh by reading CI-Workflows metering ledger data (ciw-metering) and integrating it into the existing warn/hardstop budget enforcement flow, plus adds an offline unit test harness for the channel contract.
Changes:
- Switch
cost.llm_tokens.data_sourcefrompendingtociw-meteringand add metering locator config (repo/branch/code_path). - Add
llm_channel_account()to pull and aggregate metering-ledger data (DATA/ZERO/INFRA tri-state) and wire results into warning/hardstop issue content. - Update the workflow to sparse-checkout the metering engine and add a focused shell test for the channel contract.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| governance/tests/test-cost-llm-channel.sh | Adds offline unit tests that extract and validate llm_channel_account() contract via marker-pair. |
| governance/policy/automation-limits.yaml | Enables ciw-metering as the LLM token data source and declares metering locator fields. |
| governance/cost-check.sh | Implements the LLM metering consumption path and integrates it with warn/hardstop flows. |
| .github/workflows/cost-check.yml | Adds sparse checkout of CI-Workflows metering engine and wires COST_LLM_METERING_PY. |
Suppressed comments (1)
governance/cost-check.sh:166
- 这里把 tar 解包失败一律当作 INFRA，但空账本/首月无 records-.jsonl 时，按当前的
tar ... "*/$LT_M_CODE/records-*.jsonl"写法会因为通配无匹配而失败，从而把“空账本应 ZERO（零用量）”误判为 INFRA。建议仅在 tarball 不可读或确实包含 records-.jsonl 但解包失败时才 INFRA；若 tarball 可读且不含 records 文件，则继续让 aggregate 返回 rc=2 → ZERO。
if ! tar -xzf "$led.tar.gz" -C "$led" --strip-components=1 --wildcards \
"*/$LT_M_CODE/records-*.jsonl" 2>/dev/null; then
printf 'INFRA\tmetering 账本 tar 解包失败（strip-components=1 + %s/records-*.jsonl）\n' "$LT_M_CODE"
return 0
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+155
to
+162
| if ! "$GH" api "repos/$LT_M_REPO/tarball/$LT_M_BRANCH" >"$led.tar.gz" 2>/dev/null; then | ||
| if "$GH" api "repos/$LT_M_REPO/branches/$LT_M_BRANCH" >/dev/null 2>&1; then | ||
| printf 'INFRA\tmetering 账本分支存在但 tarball 拉取失败（%s@%s）\n' "$LT_M_REPO" "$LT_M_BRANCH" | ||
| else | ||
| printf 'ZERO\t计量账本分支 %s@%s 未建（尚无经 wrapper 的 LLM 调用落账）——当月用量记 0\n' "$LT_M_REPO" "$LT_M_BRANCH" | ||
| fi | ||
| return 0 | ||
| fi |
Comment on lines
+319
to
+323
| W_TITLE="成本告警（$YEAR-$MONTH）：Actions 分钟 ${PCT_MIN}% / LLM token ${PCT_TOK:--}%（阈值 ${AM_WARN}%）" | ||
| W_EXISTING=$(gov_open_issues cost-budget-warning | grep -m1 "$YEAR-$MONTH" | cut -f1) | ||
| W_BODY="额度告警（ADR-0040，$(date -u +%FT%TZ)）：Actions 分钟（$YEAR-$MONTH）$USED_MIN / $AM_QUOTA = ${PCT_MIN}%，达 ${AM_WARN}% 告警档——未硬停；达 ${AM_STOP}% 将置 \`$CB_VARIABLE\` 熔断并撤全部 auto-merge。" | ||
| W_BODY="额度告警（ADR-0040，$(date -u +%FT%TZ)）：Actions 分钟（$YEAR-$MONTH）$USED_MIN / $AM_QUOTA = ${PCT_MIN}%，达 ${AM_WARN}% 告警档——未硬停；达 ${AM_STOP}% 将置 \`$CB_VARIABLE\` 熔断并撤全部 auto-merge。 | ||
| ${LLM_SUMMARY:+$LLM_SUMMARY | ||
| }（LLM 阈值 ${LT_WARN}%/${LT_STOP}%，ADR-0062 归账通道）" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
动机
ADR-0040 决策 6 把 LLM token 预算通道标为 pending（阈值已声明、数据源缺失）。W2-C3（.github#216，ADR-0062）落地数据消费端：cost-check 每小时读取 CI-Workflows 计量账本（metering-ledger 分支，按 invoke 聚合的 JSONL hash 链），经归账引擎按角色档（spec-author/probe/conductor…）汇总当月 token，进入既有 warn 80% / hardstop 100% 熔断分级——替代 pending 声明。数据生产端（wrapper 完整版 + ledger 同步管道）在配套 CI-Workflows PR #54。
变更清单
governance/policy/automation-limits.yaml：cost.llm_tokens.data_source: pending → ciw-metering+metering:定位三件套（repo/branch/code_path）——复用本段现有结构，不新建配置面；pending保留为回滚值（脚本仍识别）governance/cost-check.sh：llm_channel_account()（@w2c3-llm-channel标记对，供测试提取）：拉metering-ledger分支 tarball →metering.py aggregate --since <月初>按角色档归账（归账前先验链）。三态输出：DATA（当月 token + 角色档 json，进 PCT_TOK 与告警/硬停 issue 正文）、ZERO（账本分支未建/空账本=零用量声明，非 infra）、INFRA（tarball 失败/链断/引擎缺失——不可信数据不入账，exit 2 fail-closed，不盲熔断不静默归零）.github/workflows/cost-check.yml：sparse checkout CI-Workflowspipeline/metering（归账引擎；该仓公开、自身 CI+org-gate 把关，跟 main 消费——升级破坏会让本检查显式 INFRA 红而非静默）+COST_LLM_METERING_PYenvgovernance/tests/test-cost-llm-channel.sh：标记对提取被测函数 + 桩 metering.py（与真实 aggregate 退出码契约一致：0/2/3）+ 桩 gh，六形态断言；gate.yml 的 test-*.sh 通配自动纳入 CIAC 映射（.github#216 AC-4 消费端）
COST_DRY_RUN=1+ 真实 CI-Workflows metering.py）：三档（绿/告警/硬停）零 INFRA；角色档分布进告警 issue 正文（W_BODY/P0_BODY）。
单测（
bash governance/tests/test-cost-llm-channel.sh）：测试方法
bash governance/tests/test-cost-llm-channel.sh（零网络零真实 gh，gate 自动跑）COST_LLM_METERING_DIR/COST_LLM_METERING_PY注入 + 桩 gh（billing/变量/issue）→ 三档分级与角色档归账断言（输出见上）test-ir0002.sh在本地 Windows 环境存在 2 处预存失败（origin/main 干净工作树同样失败，与本 PR 无关——本 PR 未触碰 drift-check.sh；CI ubuntu 上为绿）风险与回滚
data_source改回pending即回到 ADR-0040 决策 6 声明形态（脚本分支保留）；摘除通道需同步摘 ADR-0062 数据源声明Card: #216