Skip to content

feat(automation): add governed NVIDIA NIM OpenCode commercial loop - #115

Merged
seonghobae merged 37 commits into
mainfrom
feat/ai-proposal-quality-evaluation
Aug 6, 2026
Merged

feat(automation): add governed NVIDIA NIM OpenCode commercial loop#115
seonghobae merged 37 commits into
mainfrom
feat/ai-proposal-quality-evaluation

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Buyer-visible outcome

LifeOS can continue safe commercial development every hour through OpenCode and NVIDIA NIM while preserving independent human, CodeRabbit, security, and GitHub Check gates.

Implemented contract

  • hourly and manual OpenCode workflow with a two-hour bounded runtime
  • NVIDIA_NIM_API_KEY as the only model credential; no COPILOT_GITHUB_TOKEN
  • existing review-agent workflows and credential names pinned byte-for-byte by an executable preservation contract
  • deterministic PR/issue queue validation and exact-head merge eligibility
  • Fugu, Conductor, and Trinity test-time-compute profiles with bounded stages, recursion, work decomposition, role-specific reasoning effort, and command access lists
  • executable ablation gates that promote deeper routing only for strict correctness or evidence gains over the Fugu baseline
  • prompt-injection-resistant immutable policy prompt
  • OpenCode receives no GitHub token and cannot commit, push, call GitHub write APIs, or merge
  • a trusted non-model wrapper prepares same-repository repair branches, creates bounded new branches, publishes only after repository verification, and performs --match-head-commit squash merges
  • optional shell-free, bounded contextual-orchestrator adapter with a credential-stripped process environment and standalone fallback
  • realistic fixtures for failed PostgreSQL checks, rate-limited CodeRabbit review, stale review SHA, exact-head readiness, buyer-gap selection, neutral/skipped checks, and malicious repository instructions
  • least-privilege GitHub permissions, immutable action pins, non-overlapping hourly concurrency, lockfile-pinned OpenCode, and post-agent full repository verification
  • AGENTS.md, CLAUDE.md, ARCHITECTURE.md, ADR, operations runbook, diagrams, current standards/research note, capability evidence, and CHANGELOG entry

Merge gate

Formatting, lint, type checking, exact package coverage, realistic fixture tests, OpenCode smoke evidence, build, Compose validation, AppGuardrail, Semgrep, Security Scan, Commercial Readiness, CodeRabbit, and every actionable human/security review thread must pass on the exact current head before squash merge.

Closes #114.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 45 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f7d786a7-0778-4332-8c1b-1832cb345e79

📥 Commits

Reviewing files that changed from the base of the PR and between 3ccce4f and a97104b.

📒 Files selected for processing (5)
  • apps/ai-service/src/proposal-quality-evaluation.ts
  • apps/ai-service/src/proposal-quality-review-regression.test.ts
  • docs/operations/ai-proposal-quality-evaluation.md
  • docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md
  • docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md
📝 Walkthrough

Walkthrough

AI 제안 품질 평가기를 추가했습니다. 고정 fixture를 ProposalService 경계에서 평가하고, 품질 지표와 오류를 불변 JSON 보고서로 집계합니다. 결정적 테스트, 정규화 검증, 운영 문서와 연구 기준도 추가했습니다.

Changes

AI 제안 품질 평가

Layer / File(s) Summary
Evaluator 계약과 보고서 집계
apps/ai-service/src/proposal-quality-evaluation.ts, apps/ai-service/src/proposal-quality-evaluation.test.ts, apps/ai-service/src/proposal-quality-normalization.test.ts
평가 입력과 결과 타입, fixture 검증, ProposalService 기반 순차 평가, 모델 오류 처리, 품질 지표 집계를 추가했습니다. 결과와 중첩 데이터를 깊게 동결하고 응답 본문과 오류 메시지를 보고서에서 제외합니다.
버전 fixture와 결정적 검증
apps/ai-service/src/proposal-quality-fixtures.ts, apps/ai-service/src/proposal-quality-fixtures.test.ts, apps/ai-service/src/proposal-quality-evaluation.test.ts
활성 작업, 완료 항목, 빈 context, 한국어 요청, 날짜 목표, 직접·간접 프롬프트 인젝션 fixture를 추가했습니다. 허용 작업, 대상 ID, 금지 문자열, 동결 상태와 입력 제한을 검증합니다.
설계와 운영 기준
docs/operations/ai-proposal-quality-evaluation.md, docs/research/2026-08-05-ai-proposal-quality-evaluation-standards.md, docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md, docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md, CHANGELOG.md, apps/ai-service/package.json
평가 지표, 보안 경계, live conformance 범위, 임계값, 회귀 대응, 연구 근거와 검증 명령을 문서화했습니다. lint의 Prettier 대상과 변경 로그를 갱신했습니다.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ProposalQualityEvaluator
  participant ProposalService
  participant ProposalModel
  ProposalQualityEvaluator->>ProposalService: 각 fixture 평가 요청
  ProposalService->>ProposalModel: 제안 생성 요청
  ProposalModel-->>ProposalService: 제안 또는 모델 오류
  ProposalService-->>ProposalQualityEvaluator: 검증 결과 또는 실패 상태
  ProposalQualityEvaluator-->>ProposalQualityEvaluator: 품질 counts/rates와 불변 보고서 생성
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 결정론적 AI 제안 품질 평가 기능 추가라는 주요 변경 사항을 정확하고 간결하게 설명합니다.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/ai-proposal-quality-evaluation

Comment @coderabbitai help to get the list of available commands.

@seonghobae
seonghobae marked this pull request as ready for review August 5, 2026 01:53

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Comment thread .github/workflows/proposal-quality-normalization-fix.yml Fixed
Comment thread .github/workflows/proposal-quality-normalization-fix.yml Fixed

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (4)
apps/ai-service/src/proposal-quality-evaluation.ts (3)

165-184: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

허용 종류 상한을 집합 크기에서 유도하십시오.

value.length > 3EVALUATION_OPERATION_KINDS 의 크기를 하드코딩합니다. 나중에 네 번째 inert 종류를 집합에 추가하면 상한이 3으로 남아 유효한 fixture가 거부됩니다. 상한을 집합에서 직접 유도하십시오.

♻️ 제안 수정
   if (
     !Array.isArray(value) ||
     value.length === 0 ||
-    value.length > 3
+    value.length > EVALUATION_OPERATION_KINDS.size
   ) {
     return invalid();
   }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/ai-service/src/proposal-quality-evaluation.ts` around lines 165 - 184,
Update requireOperationKinds to derive the maximum allowed input length from
EVALUATION_OPERATION_KINDS.size instead of the hardcoded value 3, while
preserving the existing validation for empty arrays, invalid kinds, and
duplicates.

477-487: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

채점 로직의 결함이 모델 실패로 위장됩니다.

try 블록이 successfulCase(fixture, proposal) 도 포함합니다. 따라서 채점 코드 내부의 프로그래밍 오류(예: TypeError)가 proposal_unavailable 로 보고됩니다. 운영자는 평가기 결함을 provider 실패로 잘못 해석합니다. 이는 docs/operations/ai-proposal-quality-evaluation.md 81번째 줄의 실패 분류 절차를 무력화합니다.

try 범위를 모델 호출로만 좁히십시오.

♻️ 제안 수정
     for (const fixture of fixtures) {
+      let proposal: AuditableProposal;
       try {
-        const proposal = await service.generateProposal(
+        proposal = await service.generateProposal(
           workspaceId,
           fixture.request,
         );
-        cases.push(successfulCase(fixture, proposal));
       } catch {
         cases.push(unavailableCase(fixture));
+        continue;
       }
+      cases.push(successfulCase(fixture, proposal));
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/ai-service/src/proposal-quality-evaluation.ts` around lines 477 - 487,
In the fixture loop, narrow the try/catch around service.generateProposal so
only provider/model failures produce unavailableCase(fixture); move
successfulCase(fixture, proposal) outside the catch-protected region so
evaluator errors propagate instead of being classified as proposal_unavailable.

296-316: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

운영 문서의 연산 적합성 정의를 구현 계약과 일치시키십시오.

validateOperationsoperations를 1–20개로 제한합니다. ProposalQualityEvaluator는 이 검증을 통과한 proposal만 평가합니다. 따라서 빈 배열이 benignUtilityPassed: true가 되는 실행 경로는 없습니다. 그러나 운영 문서는 연산 개수 조건을 생략하고 설계 문서와 계획 문서만 해당 조건을 명시합니다. 운영 문서에 동일한 조건을 추가하십시오.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/ai-service/src/proposal-quality-evaluation.ts` around lines 296 - 316,
운영 문서의 연산 적합성 정의에 proposal.operations 개수가 1–20개여야 한다는 조건을 추가하십시오.
apps/ai-service/src/proposal-quality-evaluation.ts 296-316의 validateOperations
관련 구현은 직접 변경하지 말고 계약의 근거로 유지하십시오.
docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md 45와
docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md 92의 기존 조건과
일관되도록 해당 문서 정의를 갱신하십시오.
apps/ai-service/package.json (1)

8-8: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

문서 경로 목록이 슬라이스마다 계속 늘어납니다.

lint 스크립트는 이제 18개의 명시적 문서 경로를 담습니다. 새 기능 슬라이스마다 네 개 경로를 추가해야 하며, 경로를 빠뜨리면 형식 검사가 조용히 누락됩니다. 이번 추가 경로 네 개는 이 PR의 새 문서와 정확히 일치하므로 기능상 문제는 없습니다.

후속 작업으로 이 목록을 저장소 루트의 형식 검사로 옮기거나 glob으로 대체하는 방안을 고려하십시오. glob 범위를 넓히면 현재 검사 대상이 아닌 문서가 포함될 수 있으므로 별도 변경으로 처리하십시오.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/ai-service/package.json` at line 8, Update the lint configuration around
the package.json “lint” script to avoid maintaining an ever-growing explicit
document-path list, preferably by delegating document formatting checks to the
repository-root formatter or an appropriate scoped glob. Preserve the existing
TypeScript and configuration-file checks, and ensure the replacement does not
unintentionally include currently out-of-scope documents.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@apps/ai-service/src/proposal-quality-evaluation.ts`:
- Around line 230-244: Normalize errors from validateProposalRequest within
requireFixture by catching ProposalValidationError and rethrowing it as
ProposalQualityEvaluationError. Preserve the existing validated request flow and
ensure invalid record.request inputs consistently satisfy the
ProposalQualityEvaluationError contract.

In `@docs/research/2026-08-05-ai-proposal-quality-evaluation-standards.md`:
- Line 71: In
docs/research/2026-08-05-ai-proposal-quality-evaluation-standards.md:71-71 and
docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md:120-120,
standardize the CyberSecEval 2 author list to include Song, D., Wan, S., and
Ahmad, F.; add Wan, S. and change Ahmad, S. to Ahmad, F. in the second document.

In `@docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md`:
- Line 92: Align the documented conformance condition with the implementation
around evaluateOperations and validateOperations: ensure the 1–20
operation-count requirement is enforced in code, or revise the documentation to
explicitly identify validateOperations as providing that guarantee if it already
does. Keep the remaining operation-kind, required-target, and context-evidence
conditions unchanged.

In `@docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md`:
- Line 45: Unify the operationConformanceRate definition across the design spec,
evaluateOperations, and the operations documentation: either implement
operation-count validation in evaluateOperations and document it consistently,
or remove “operation count” from the metric definition and related conditions.
Ensure all three references describe the same conformance criteria.

---

Nitpick comments:
In `@apps/ai-service/package.json`:
- Line 8: Update the lint configuration around the package.json “lint” script to
avoid maintaining an ever-growing explicit document-path list, preferably by
delegating document formatting checks to the repository-root formatter or an
appropriate scoped glob. Preserve the existing TypeScript and configuration-file
checks, and ensure the replacement does not unintentionally include currently
out-of-scope documents.

In `@apps/ai-service/src/proposal-quality-evaluation.ts`:
- Around line 165-184: Update requireOperationKinds to derive the maximum
allowed input length from EVALUATION_OPERATION_KINDS.size instead of the
hardcoded value 3, while preserving the existing validation for empty arrays,
invalid kinds, and duplicates.
- Around line 477-487: In the fixture loop, narrow the try/catch around
service.generateProposal so only provider/model failures produce
unavailableCase(fixture); move successfulCase(fixture, proposal) outside the
catch-protected region so evaluator errors propagate instead of being classified
as proposal_unavailable.
- Around line 296-316: 운영 문서의 연산 적합성 정의에 proposal.operations 개수가 1–20개여야 한다는 조건을
추가하십시오. apps/ai-service/src/proposal-quality-evaluation.ts 296-316의
validateOperations 관련 구현은 직접 변경하지 말고 계약의 근거로 유지하십시오.
docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md 45와
docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md 92의 기존 조건과
일관되도록 해당 문서 정의를 갱신하십시오.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 159b280d-f112-4df4-99ab-739038b8d63d

📥 Commits

Reviewing files that changed from the base of the PR and between dcd0a2e and 3ccce4f.

📒 Files selected for processing (11)
  • CHANGELOG.md
  • apps/ai-service/package.json
  • apps/ai-service/src/proposal-quality-evaluation.test.ts
  • apps/ai-service/src/proposal-quality-evaluation.ts
  • apps/ai-service/src/proposal-quality-fixtures.test.ts
  • apps/ai-service/src/proposal-quality-fixtures.ts
  • apps/ai-service/src/proposal-quality-normalization.test.ts
  • docs/operations/ai-proposal-quality-evaluation.md
  • docs/research/2026-08-05-ai-proposal-quality-evaluation-standards.md
  • docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md
  • docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md

Comment thread apps/ai-service/src/proposal-quality-evaluation.ts
Comment thread docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md Outdated
Comment thread docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md Outdated
Comment thread .github/workflows/pr115-format-review-regression.yml Fixed

Copy link
Copy Markdown
Contributor Author

@coderabbitai @opencode-agent @cwl-noema-review Please independently review exact current head a97104b3277f6ad711dd51f3fd6b47f936b062bd. The branch now normalizes only ProposalValidationError into ProposalQualityEvaluationError, preserves unexpected request-inspection failures, derives operation-kind cardinality from the authoritative set, isolates provider/model failures from semantic scoring defects, aligns the 1–20 production validator guarantee across operations/design/plan documentation, corrects the CyberSecEval 2 APA author list, removes the one-shot formatter workflow, and passes exact-head CI, AppGuardrail, Commercial Readiness, Security Scan, and Semgrep. Please resolve only the two remaining threads if their exact-current-head fixes are verified, and submit APPROVE only after confirming 100% statement/branch/function/line coverage and zero actionable threads.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='a97104b3277f6ad711dd51f3fd6b47f936b062bd'
actual="$(git rev-parse HEAD)"
printf 'HEAD=%s\nEXPECTED=%s\nHEAD_MATCH=%s\n' "$actual" "$expected" "$([ "$actual" = "$expected" ] && echo yes || echo no)"

printf '\n== Changed files at exact head ==\n'
git diff-tree --no-commit-id --name-status -r "$expected"

printf '\n== Relevant TypeScript structure ==\n'
ast-grep outline apps/ai-service/src/proposal-quality-evaluation.ts --items all --view expanded || true
ast-grep outline apps/ai-service/src/proposal-quality-evaluation.test.ts --items all --view expanded || true

printf '\n== Exception handling, cardinality, and scoring references ==\n'
rg -n -C 5 -e 'ProposalValidationError|ProposalQualityEvaluationError|operationKinds|operation.*kind|provider|model|semantic|failure' \
  apps/ai-service/src/proposal-quality-evaluation.ts \
  apps/ai-service/src/proposal-quality-evaluation.test.ts \
  apps/ai-service/src/proposal-quality-fixtures.ts

printf '\n== Production validator references ==\n'
rg -n -C 3 -e '1.{0,8}20|ProposalValidationError|validate' \
  docs/operations/ai-proposal-quality-evaluation.md \
  docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md \
  docs/superpowers/plans/2026-08-05-ai-proposal-quality-evaluation.md \
  apps/ai-service/src/proposal-quality-evaluation.ts

printf '\n== CyberSecEval references ==\n'
rg -n -C 3 -i 'CyberSecEval' docs/research/2026-08-05-ai-proposal-quality-evaluation-standards.md

printf '\n== Formatter workflow references ==\n'
rg -n -C 2 -i 'formatter|prettier|format' .github apps/ai-service/package.json 2>/dev/null || true

printf '\n== Coverage configuration and committed coverage evidence ==\n'
rg -n -C 3 -e 'coverage|100|threshold|branches|functions|lines|statements' \
  apps/ai-service/package.json \
  apps/ai-service/vitest.config.* \
  apps/ai-service/vite.config.* \
  package.json \
  2>/dev/null || true
fd -HI -t f -E .git -g '*coverage*' -g 'coverage-final.json' -g 'lcov.info' -g 'junit*.xml' -g '*test-results*' . | head -100

printf '\n== PR review threads and checks ==\n'
gh api graphql -F owner='ContextualWisdomLab' -F name='life-os' -F number=115 -f query='
query($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      headRefOid
      reviewThreads(first: 100) {
        nodes {
          id
          isResolved
          isOutdated
          path
          line
          originalLine
          comments(first: 20) { nodes { author { login } body url createdAt } }
        }
      }
      commits(last: 1) {
        nodes {
          commit {
            oid
            checkSuites(first: 100) {
              nodes {
                app { name }
                conclusion
                status
                workflowRun { workflow { name } }
              }
            }
          }
        }
      }
    }
  }
}' > /tmp/pr115.json
jq '{headRefOid: .data.repository.pullRequest.headRefOid,
     threads: [.data.repository.pullRequest.reviewThreads.nodes[] |
       {id,isResolved,isOutdated,path,line,originalLine,
        comments: [.comments.nodes[] | {author: .author.login, body, url}]}],
     checks: [.data.repository.pullRequest.commits.nodes[0].commit.checkSuites.nodes[] |
       {app: .app.name, conclusion, status, workflow: (.workflowRun.workflow.name // null)}]}' /tmp/pr115.json

Length of output: 96955


🏁 Script executed:

#!/bin/bash
set -euo pipefail

sha='a97104b3277f6ad711dd51f3fd6b47f936b062bd'
repo='ContextualWisdomLab/life-os'

printf '%s\n' '== Evaluator execution path =='
sed -n '280,520p' apps/ai-service/src/proposal-quality-evaluation.ts

printf '%s\n' '== Production proposal-service error and validation contract =='
ast-grep outline apps/ai-service/src/proposal-service.ts --items all --view expanded
rg -n -A100 -B10 -e 'export function validateProposalRequest|class ProposalValidationError|validateOperations|async create|async propose|generate' \
  apps/ai-service/src/proposal-service.ts

printf '%s\n' '== Tests for typed normalization and unexpected request-inspection failures =='
rg -n -A24 -B12 -e 'unexpected|ProposalValidationError|request-inspection|validateProposalRequest|throw.*Error|rejects.*error' \
  apps/ai-service/src/proposal-quality-evaluation.test.ts

printf '%s\n' '== Exact CyberSecEval entries in both documents =='
rg -n -C 1 'Bhatt, M\.' \
  docs/research/2026-08-05-ai-proposal-quality-evaluation-standards.md \
  docs/superpowers/specs/2026-08-05-ai-proposal-quality-evaluation-design.md

printf '%s\n' '== Exact-head CI workflow runs =='
gh run list --repo "$repo" --commit "$sha" --limit 100 \
  --json databaseId,name,status,conclusion,workflowName,event,createdAt,updatedAt,url

ci_run_id="$(gh run list --repo "$repo" --commit "$sha" --limit 100 \
  --json databaseId,workflowName,status,conclusion \
  --jq '.[] | select(.workflowName == "CI") | .databaseId' | head -n 1)"

if [ -z "$ci_run_id" ]; then
  echo 'CI_RUN_NOT_FOUND'
  exit 0
fi

printf '\n== CI jobs for run %s ==\n' "$ci_run_id"
gh run view "$ci_run_id" --repo "$repo" --json status,conclusion,jobs,url

printf '\n== CI coverage-related log lines ==\n'
gh run view "$ci_run_id" --repo "$repo" --log 2>&1 | \
  rg -n -i -C 3 'coverage|statements|branches|functions|lines|all files|100\.?0*' || true

printf '\n== CI artifacts ==\n'
gh api "repos/$repo/actions/runs/$ci_run_id/artifacts" --paginate \
  --jq '.artifacts[] | {id,name,expired,size_in_bytes,archive_download_url}' || true

Length of output: 66684


🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo_owner='ContextualWisdomLab'
repo_name='life-os'
pr_number=115

for thread_id in \
  'PRRT_kwDOTq3wqs6Whf5r' \
  'PRRT_kwDOTq3wqs6Whf5w'
do
  gh api graphql -F threadId="$thread_id" -f query='
mutation($threadId: ID!) {
  resolveReviewThread(input: {threadId: $threadId}) {
    thread { id isResolved isOutdated path line }
  }
}' --jq '.data.resolveReviewThread.thread'
done

printf '\n== Current unresolved review threads ==\n'
gh api graphql -F owner="$repo_owner" -F name="$repo_name" -F number="$pr_number" -f query='
query($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      reviewThreads(first: 100) {
        nodes {
          id isResolved isOutdated path line
          comments(first: 1) { nodes { author { login } body url } }
        }
      }
    }
  }
}' --jq '.data.repository.pullRequest.reviewThreads.nodes
  | map(select(.isResolved == false))
  | map({id,isOutdated,path,line,author: .comments.nodes[0].author.login,url: .comments.nodes[0].url})'

printf '\n== Current exact-head workflow states ==\n'
gh run list --repo "$repo_owner/$repo_name" \
  --commit 'a97104b3277f6ad711dd51f3fd6b47f936b062bd' \
  --limit 100 \
  --json name,status,conclusion,url \
  --jq '.[] | {name,status,conclusion,url}'

Length of output: 3760


@seonghobae I verified and resolved the two requested threads.

  • ProposalValidationError normalization is correct. Non-validation request-inspection failures remain unmodified.
  • Both CyberSecEval 2 entries use the corrected author list.
  • No unresolved review threads remain.
  • Exact-head CI passed with 100% statement, branch, function, and line coverage for apps/ai-service, including proposal-quality-evaluation.ts.
  • AppGuardrail, Commercial Readiness, Security Scan, SAST Semgrep, and the required reviews passed.

I cannot approve yet. Strix Security Scan is still in progress for the exact head.

You are interacting with an AI system.

@seonghobae
seonghobae merged commit d91f176 into main Aug 6, 2026
34 checks passed
@seonghobae seonghobae changed the title feat(ai): add proposal quality evaluation feat(automation): add governed NVIDIA NIM OpenCode commercial loop Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review
@cwl-noema-review
@opencode-agent

Please review the exact live head only. Verify proposal-quality evidence separates validity, semantic operation conformance, grounding, benign utility, forbidden-text leakage, and prompt-injection resistance; denominators are explicit and null-safe; English, Korean, temporal, completed-item, empty-context, and direct/indirect injection fixtures exercise the real ProposalService boundary; provider failures are credential-redacted; reports are deterministic and credential-free; and AI-service production statement/branch/function/line coverage is 100%. Review only; do not update the branch, merge, release, or alter protections.

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='a97104b3277f6ad711dd51f3fd6b47f936b062bd'
repo='ContextualWisdomLab/life-os'
pr=115

printf '== Exact head ==\n'
actual="$(git rev-parse HEAD)"
printf 'checked_out=%s\nexpected=%s\nmatch=%s\n' "$actual" "$expected" "$([ "$actual" = "$expected" ] && echo yes || echo no)"
gh pr view "$pr" --repo "$repo" --json headRefOid,url \
  --jq '"github_head=\(.headRefOid)\nurl=\(.url)\nmatch_expected=\(.headRefOid == "'"$expected"'")"'

printf '\n== Changed files at requested head ==\n'
git diff-tree --no-commit-id --name-status -r "$expected"

printf '\n== Relevant file outlines ==\n'
ast-grep outline apps/ai-service/src/proposal-quality-evaluation.ts --items all --view expanded
ast-grep outline apps/ai-service/src/proposal-quality-fixtures.ts --items all --view expanded
ast-grep outline apps/ai-service/src/proposal-service.ts --items all --view expanded
ast-grep outline apps/ai-service/src/proposal-quality-evaluation.test.ts --items all --view expanded

printf '\n== Evaluator implementation ==\n'
sed -n '1,560p' apps/ai-service/src/proposal-quality-evaluation.ts

printf '\n== Fixture definitions ==\n'
sed -n '1,280p' apps/ai-service/src/proposal-quality-fixtures.ts

printf '\n== ProposalService validation and generation boundary ==\n'
rg -n -C 10 -e 'ProposalValidationError|validateProposalRequest|class ProposalService|async .*propos|async .*creat|model\.' \
  apps/ai-service/src/proposal-service.ts

printf '\n== Evaluator test evidence ==\n'
rg -n -C 5 -i -e 'validity|semantic|ground|utility|forbidden|injection|denominator|rate|provider|credential|secret|determin|ProposalService|fixture|coverage' \
  apps/ai-service/src/proposal-quality-evaluation.test.ts \
  apps/ai-service/src/proposal-quality-fixtures.test.ts \
  apps/ai-service/src/proposal-quality-normalization.test.ts

printf '\n== CI runs and coverage logs for exact head ==\n'
gh run list --repo "$repo" --commit "$expected" --limit 100 \
  --json databaseId,name,workflowName,status,conclusion,url \
  --jq '.[] | {databaseId,name,workflowName,status,conclusion,url}'

ci_id="$(gh run list --repo "$repo" --commit "$expected" --limit 100 \
  --json databaseId,workflowName,status,conclusion \
  --jq '.[] | select(.workflowName == "CI" and .conclusion == "success") | .databaseId' | head -n1)"
if [ -n "$ci_id" ]; then
  printf '\nCI run: %s\n' "$ci_id"
  gh run view "$ci_id" --repo "$repo" --json status,conclusion,jobs,url
  gh run view "$ci_id" --repo "$repo" --log 2>&1 | \
    rg -n -i -C 4 'all files|statements|branches|functions|lines|coverage|100\.?0*' || true
else
  printf '\nNo successful CI run found for exact head.\n'
fi

printf '\n== Review threads ==\n'
gh api graphql -F owner='ContextualWisdomLab' -F name='life-os' -F number="$pr" -f query='
query($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      reviewThreads(first: 100) {
        nodes {
          id
          isResolved
          isOutdated
          path
          line
          comments(first: 1) { nodes { author { login } body url } }
        }
      }
    }
  }
}' --jq '.data.repository.pullRequest.reviewThreads.nodes
  | map(select(.isResolved == false))
  | map({id,isOutdated,path,line,author:.comments.nodes[0].author.login,url:.comments.nodes[0].url,body:.comments.nodes[0].body})'

Length of output: 143978


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Evaluator tests: failure, deterministic report, and validation behavior =='
sed -n '370,680p' apps/ai-service/src/proposal-quality-evaluation.test.ts

printf '%s\n' '== Regression tests for the evaluator =='
sed -n '1,320p' apps/ai-service/src/proposal-quality-review-regression.test.ts

printf '%s\n' '== AI-service coverage configuration and quality gate =='
sed -n '1,260p' apps/ai-service/vitest.config.ts
sed -n '1,320p' apps/ai-service/src/quality-coverage.test.ts
cat apps/ai-service/package.json

printf '%s\n' '== Static references to report serialization and model label handling =='
rg -n -C 4 -e 'modelLabel|JSON\.stringify\(report\)|toEqual\(report\)|proposal_unavailable|unexpected request|request-inspection|ProposalValidationError' \
  apps/ai-service/src/proposal-quality-evaluation.test.ts \
  apps/ai-service/src/proposal-quality-review-regression.test.ts \
  apps/ai-service/src/proposal-quality-evaluation.ts

printf '%s\n' '== Current working tree state =='
git status --short

Length of output: 37673


@seonghobae I completed a read-only review of exact head a97104b3277f6ad711dd51f3fd6b47f936b062bd.

I found no actionable issue in the requested scope.

  • The evaluator separates validity, operation conformance, target grounding, forbidden-text results, benign utility, and prompt-injection resistance.
  • Each rate has an explicit denominator. Zero denominators return null.
  • The fixture suite covers English, Korean, temporal, completed-item, empty-context, direct-injection, and indirect-injection cases.
  • Each fixture passes through ProposalService.generateProposal().
  • Provider and model failures produce only proposal_unavailable. The report does not retain the caught error text.
  • Reports and nested evidence are immutable. Deterministic clock and identifier seams support reproducible report metadata.
  • Exact-head CI passed with 100% statement, branch, function, and line coverage for apps/ai-service.
  • No unresolved review threads remain.
  • All listed exact-head checks, including security checks, completed successfully.

I did not update the branch, merge the PR, release artifacts, or change repository protections.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants