Skip to content

ci: route Strix through GitHub Models - #335

Merged
seonghobae merged 4 commits into
masterfrom
living-baroness
Jun 2, 2026
Merged

ci: route Strix through GitHub Models#335
seonghobae merged 4 commits into
masterfrom
living-baroness

Conversation

@seonghobae

@seonghobae seonghobae commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • default Strix PR/security scans to GitHub Models via models: read, github.token, openai/openai/gpt-4.1, and a trusted LLM_API_BASE_FILE
  • keep explicit Vertex and direct OpenAI provider paths fail-closed and provider-scoped
  • align release governance tests and provider documentation with the new route

Verification

  • bash -n scripts/ci/strix_quick_gate.sh && bash -n scripts/ci/test_strix_quick_gate.sh && git diff --check
  • actionlint .github/workflows/strix.yml
  • python3 -m pytest -p no:warnings backend/tests/test_release_governance.py -q
  • targeted_strix_github_models_contract harness: PASS

Summary by CodeRabbit

  • New Features

    • Default Strix scans now route via GitHub Models (token-based), with a default model selection and required API base file.
  • Documentation

    • Governance, architecture, README, and planning docs updated to reflect GitHub Models as the default and fail-closed routing; Vertex and direct OpenAI remain explicit-only.
  • Chores / Tests

    • CI workflow, gate logic, and tests adjusted to enforce provider-mode routing, token/API-base handling, permissions, masking, and fallback semantics.

@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@seonghobae, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 25 minutes and 35 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3c4dc6d8-a24b-4899-b1b1-f9e2aa19ccf0

📥 Commits

Reviewing files that changed from the base of the PR and between 280fb7e and b4f6ab1.

📒 Files selected for processing (9)
  • .github/workflows/strix.yml
  • AGENTS.md
  • ARCHITECTURE.md
  • README.md
  • backend/tests/test_release_governance.py
  • docs/plans/2026-05-27-strix-openai-direct-only.md
  • docs/plans/2026-05-29-strix-full-scan-operational-model.md
  • docs/plans/2026-05-29-strix-vertex-model-warning-filter.md
  • scripts/ci/test_strix_quick_gate.sh
📝 Walkthrough

Walkthrough

This PR changes Strix scanning to default to GitHub Models (openai/openai/gpt-5), add provider_mode=github_models wiring, require a trusted LLM_API_BASE_FILE and provider-scoped github.token, and updates the CI gate script, tests, and documentation to enforce fail-closed provider routing.

Changes

Strix GitHub Models Default Provider

Layer / File(s) Summary
Workflow GitHub Models provider configuration
.github/workflows/strix.yml
Defaults PR dispatch input to openai/openai/gpt-5, grants models: read, maps openai/openai/* to provider_mode=github_models, wires github.token as the provider-scoped key, writes LLM_API_BASE_FILE, and updates model validation and wiring.
CI gate script GitHub Models validation and routing
scripts/ci/strix_quick_gate.sh
Allows GitHub Models prefixes while requiring LLM_API_BASE_FILE for those models, enforces API-base/model alignment in resolver logic, expands provider-only error detection to include GitHub Models markers, and permits GitHub Models-prefixed fallback candidates.
CI test harness GitHub Models scenarios and expectations
scripts/ci/test_strix_quick_gate.sh
Asserts models: read, PR-default strix_llm uses `github.event.inputs.strix_llm
Governance and documentation for GitHub Models defaults
AGENTS.md, ARCHITECTURE.md, README.md, docs/plans/*
Document GitHub Models as the active default route (require models: read, github.token, trusted LLM_API_BASE), keep Vertex and direct OpenAI as explicit-only paths, and reinforce fail-closed rules against generic LLM_API_KEY, cross-provider fallback, and provider-infrastructure fallbacks.
Governance test validation of GitHub Models default
backend/tests/test_release_governance.py
Replace prior Vertex-default assertions with checks that the workflow uses GitHub Models mode by default and includes the GitHub Models inference/base URL and github.token requirement message while ensuring prior Vertex-secret fallback strings are absent.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • Seongho-Bae/naruon#230: Modifies Strix workflow and tests to enforce GitHub Models provider-mode routing with models: read, github.token, and base-file handling.
  • Seongho-Bae/naruon#238: Also adjusts .github/workflows/strix.yml provider selection/credential wiring; related at workflow/test level.
  • Seongho-Bae/naruon#304: Changes the Strix workflow model-selection contract and related governance/tests; directly connected to this PR's workflow/test edits.

Poem

🐰 A GitHub Models path I tread with care,

Tokens scoped and base-file placed with flair,
The Strix gate checks models, base, and key,
Fallbacks vetted, provider-bound and free,
Hop—secure PRs bloom in guarded air!

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'ci: route Strix through GitHub Models' directly summarizes the primary objective of the changeset: defaulting Strix PR/security scans to use GitHub Models instead of the previous routing.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch living-baroness

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.


Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

PR governance metadata gate is not ready for b4f6ab13b93480a02315d9e1997ff1583b1a3a0c:

  • Review decision is CHANGES_REQUESTED; address requested changes before merge.
  • Current-head CodeRabbit issue comment has blocking warning/failure evidence on b4f6ab1.
  • Current-head CodeRabbit review comment has blocking warning/failure evidence on b4f6ab1.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
scripts/ci/test_strix_quick_gate.sh (1)

384-398: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Handle the new GitHub Models success scenario in the fake dispatcher.

run_gate_case now invokes github-models-model-prefix-with-api-base-succeeds at Lines 6801-6810, but this case block never matches that scenario. The self-test will drop into unknown scenario ... and fail before it verifies the GitHub Models happy path.

Suggested fix
-	vertex-primary-notfound-fallback-success|github-models-fallback-success|github-models-fallback-requires-api-base)
+	vertex-primary-notfound-fallback-success|github-models-fallback-success|github-models-fallback-requires-api-base|github-models-model-prefix-with-api-base-succeeds)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/ci/test_strix_quick_gate.sh` around lines 384 - 398, The fake
dispatcher case block handling STRIX_LLM scenarios doesn't include the new
"github-models-model-prefix-with-api-base-succeeds" scenario so run_gate_case
falls to "unknown scenario"; update the case list (the pattern group starting
with
vertex-primary-notfound-fallback-success|github-models-fallback-success|...) to
add a branch for github-models-model-prefix-with-api-base-succeeds that echoes
the expected GitHub Models success message (matching the other GitHub success
branches) and exits 0, ensuring run_gate_case will hit the happy-path for that
scenario.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/ci/test_strix_quick_gate.sh`:
- Around line 6783-6817: The two failing cases
("github-models-model-prefix-requires-api-base" and
"github-models-fallback-requires-api-base") are still letting run_gate_case()
create a default LLM_API_BASE_FILE because they don't pass an explicit override;
update those run_gate_case calls to include an explicit empty string argument
for the LLM_API_BASE_FILE/override parameter (same position used in the other
calls that pass "", e.g., the 7th positional override) so run_gate_case doesn't
write the default API base file and the test exercises the
missing-LLM_API_BASE_FILE path; locate and edit the run_gate_case invocations by
name to add the empty-string override.

---

Outside diff comments:
In `@scripts/ci/test_strix_quick_gate.sh`:
- Around line 384-398: The fake dispatcher case block handling STRIX_LLM
scenarios doesn't include the new
"github-models-model-prefix-with-api-base-succeeds" scenario so run_gate_case
falls to "unknown scenario"; update the case list (the pattern group starting
with
vertex-primary-notfound-fallback-success|github-models-fallback-success|...) to
add a branch for github-models-model-prefix-with-api-base-succeeds that echoes
the expected GitHub Models success message (matching the other GitHub success
branches) and exits 0, ensuring run_gate_case will hit the happy-path for that
scenario.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1c9a2637-a7f8-41fd-9268-208ffed5cd04

📥 Commits

Reviewing files that changed from the base of the PR and between a0fb9f7 and 2eb1742.

📒 Files selected for processing (14)
  • .github/workflows/strix.yml
  • AGENTS.md
  • ARCHITECTURE.md
  • README.md
  • backend/tests/test_release_governance.py
  • docs/plans/2026-05-19-north-star-gap-closure.md
  • docs/plans/2026-05-27-apm-connector-operational-signals.md
  • docs/plans/2026-05-27-self-sent-webdav-materialization-intent.md
  • docs/plans/2026-05-27-strix-openai-direct-only.md
  • docs/plans/2026-05-29-strix-full-scan-operational-model.md
  • docs/plans/2026-05-29-strix-vertex-model-warning-filter.md
  • docs/plans/2026-05-29-strix-warning-filter-log-surface.md
  • scripts/ci/strix_quick_gate.sh
  • scripts/ci/test_strix_quick_gate.sh

Comment thread scripts/ci/test_strix_quick_gate.sh
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 2, 2026
coderabbitai[bot]
coderabbitai Bot previously requested changes Jun 2, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ARCHITECTURE.md`:
- Around line 161-165: The ARCHITECTURE.md snippet incorrectly references the
environment variable LLM_API_BASE; update the text to use LLM_API_BASE_FILE
instead so it matches the workflow behavior and the new fail-closed
contract—replace mentions of "LLM_API_BASE" with "LLM_API_BASE_FILE" in the
paragraph describing defaults (including the line that lists
STRIX_LLM=openai/openai/gpt-5 and the GitHub Models endpoint) and ensure the
wording clarifies that the workflow writes the endpoint to a trusted input file
and forwards the LLM_API_BASE_FILE rather than injecting an env var.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 02b33b5f-fb36-4641-923f-5564daf3cd84

📥 Commits

Reviewing files that changed from the base of the PR and between ced551e and 280fb7e.

📒 Files selected for processing (9)
  • .github/workflows/strix.yml
  • AGENTS.md
  • ARCHITECTURE.md
  • README.md
  • backend/tests/test_release_governance.py
  • docs/plans/2026-05-27-strix-openai-direct-only.md
  • docs/plans/2026-05-29-strix-full-scan-operational-model.md
  • docs/plans/2026-05-29-strix-vertex-model-warning-filter.md
  • scripts/ci/test_strix_quick_gate.sh
🚧 Files skipped from review as they are similar to previous changes (2)
  • AGENTS.md
  • backend/tests/test_release_governance.py

Comment thread ARCHITECTURE.md Outdated
@seonghobae
seonghobae temporarily deployed to living-baroness - naruon PR #335 June 2, 2026 10:04 — with Render Destroyed
@seonghobae
seonghobae enabled auto-merge June 2, 2026 10:23
@seonghobae

Copy link
Copy Markdown
Contributor Author

Strix provider evidence for current head b4f6ab13b93480a02315d9e1997ff1583b1a3a0c:

  • Manual PR-scope Strix evidence passed with GitHub Models openai/openai/gpt-4.1: https://github.com/Seongho-Bae/naruon/actions/runs/26812822235
  • The automatic pull_request_target Strix failure is from the pre-merge base workflow path: it prepared Vertex credentials (STRIX_LLM_DEFAULT_PROVIDER=vertex_ai) and failed with Vertex BILLING_DISABLED, before this PR can replace the base workflow route.
  • GitHub Models higher-model probes for this repo/org token failed as unavailable: openai/openai/gpt-5 and openai/openai/o3; openai/openai/gpt-4.1 is the validated default.
  • CodeRabbit review threads are resolved; the current-head CodeRabbit status context is successful.

Local validation for the current patch:

  • bash -n scripts/ci/strix_quick_gate.sh && bash -n scripts/ci/test_strix_quick_gate.sh && actionlint .github/workflows/strix.yml && git diff --check
  • python3 -m pytest -p no:warnings backend/tests/test_release_governance.py -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant