Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 124 additions & 31 deletions .github/workflows/nightly-bfcl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,30 +22,34 @@ on:
- ".github/workflows/nightly-bfcl.yml"
workflow_dispatch:
inputs:
only:
description: "Run only this matrix leg (qwen3.6|gpt-oss|deepseek-v4|minimax-m2.7|kimi-k2.6); empty = all"
required: false
default: ""
model:
description: "HF model id (weights) served on both arms"
description: "Override HF model id (weights) for the selected leg; empty = matrix default. Override model and bfcl_model together — mismatched weights/handler pair if only one is overridden."
required: false
default: "Qwen/Qwen3.6-27B"
default: ""
bfcl_model:
description: "BFCL FC handler name (must end in -FC; auto-registered if new)"
description: "Override BFCL FC handler name (must end in -FC); empty = matrix default. Override model and bfcl_model together — mismatched weights/handler pair if only one is overridden."
required: false
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
default: "Qwen/Qwen3.6-27B-FC"
default: ""
categories:
description: "Comma-separated BFCL test categories"
required: false
default: "simple_python,simple_java,simple_javascript,multiple,parallel,parallel_multiple,irrelevance,live_simple,live_multiple,live_parallel,live_parallel_multiple,live_irrelevance,live_relevance,multi_turn_base,multi_turn_miss_func,multi_turn_miss_param,multi_turn_long_context"
vllm_tool_parser:
description: "pure-vLLM --tool-call-parser"
description: "Override pure-vLLM --tool-call-parser; empty = matrix default"
required: false
default: "qwen3_xml"
default: ""
smg_tool_parser:
description: "SMG --tool-call-parser"
description: "Override SMG --tool-call-parser; empty = matrix default (empty also = SMG auto-detect)"
required: false
default: "qwen_xml"
default: ""
reasoning_parser:
description: "reasoning parser for both arms (empty for non-thinking SKUs)"
description: "Override reasoning parser for BOTH arms; empty = matrix defaults"
required: false
default: "qwen3"
default: ""

concurrency:
group: nightly-bfcl-${{ github.event_name }}-${{ github.ref }}
Expand Down Expand Up @@ -91,20 +95,100 @@ jobs:
path: bindings/python/dist/*.whl
retention-days: 7

# Compute the model matrix as JSON so the dispatch `only` leg-filter can depend
# on inputs — the `matrix` context is not available in a job-level `if`.
setup:
# No needs: build-wheel — this job is pure CPU JSON generation with no wheel
# dependency. bfcl-ab still gates on build-wheel.result == 'success'.
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
matrix: ${{ steps.build.outputs.matrix }}
steps:
- id: build
env:
ONLY: ${{ github.event.inputs.only }}
run: |
python3 - <<'PY' >> "$GITHUB_OUTPUT"
import json, os
legs = [
# H100 legs (TP=2 per arm, both arms concurrent: GPUs 0,1 + 2,3).
{"name": "qwen3.6", "runner": "4-gpu-h100", "tp": 2,
"gpu_a": "0,1", "gpu_b": "2,3",
"model": "Qwen/Qwen3.6-27B", "bfcl_model": "Qwen/Qwen3.6-27B-FC",
"vllm_tool": "qwen3_xml", "vllm_reason": "qwen3",
"smg_tool": "qwen_xml", "smg_reason": "qwen3",
"vllm_extra": "", "gpu_mem": "0.85", "startup_timeout": "600"},
# gpt-oss SMG arm passes NO tool parser (smg_tool empty) -> harmony auto-detect.
{"name": "gpt-oss", "runner": "4-gpu-h100", "tp": 2,
"gpu_a": "0,1", "gpu_b": "2,3",
"model": "openai/gpt-oss-120b", "bfcl_model": "openai/gpt-oss-120b-FC",

ghost Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use matching BFCL handlers for non-Qwen legs

For new legs such as gpt-oss/DeepSeek/MiniMax/Kimi that bfcl-eval does not already ship, the existing registration step registers $MODEL through scripts/bfcl/register_bfcl_model.py with its default QwenFCHandler. In those scenarios BFCL formats and decodes Qwen-style XML instead of the native tool-call protocol these legs are meant to compare, so the run measures Qwen-prompt compatibility or invalid scores rather than SMG/vLLM parser parity; each non-Qwen leg needs a matching/generic FC handler or should avoid this fallback when unsupported.

Useful? React with 👍 / 👎.

"vllm_tool": "openai", "vllm_reason": "",
"smg_tool": "", "smg_reason": "",
"vllm_extra": "", "gpu_mem": "0.85", "startup_timeout": "600"},
# Blackwell legs (8xB300, TP=4 per arm, both arms concurrent: 0-3 + 4-7).
{"name": "deepseek-v4", "runner": "blackwell", "tp": 4,
"gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7",
"model": "deepseek-ai/DeepSeek-V4-Flash", "bfcl_model": "deepseek-ai/DeepSeek-V4-Flash-FC",
Comment on lines +145 to +147

ghost Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Download the matrix-selected checkpoint

For these new Blackwell matrix legs, the selected checkpoints are TP=4 models, but the later Stage model weights step is still hard-coded to scripts/ci_download_model.sh --gpu-tier 2; that script's tier resolver only downloads models with spec['tp'] <= tier (scripts/ci_download_model.sh:37-48). On a cold Blackwell runner these checkpoints are not pre-staged, so launch_arm.sh has only its 420s startup wait while vLLM tries to fetch hundreds of GB outside the download lock, which makes the new nightly legs time out before serving.

Useful? React with 👍 / 👎.

"vllm_tool": "deepseek_v4", "vllm_reason": "deepseek_v4",
"smg_tool": "deepseek_v4",
"smg_reason": "deepseek_v31", # TODO(confirm): no deepseek_v4 SMG reasoning parser yet
# --kv-cache-dtype fp8 + --block-size 256 are REQUIRED: V4 auto-quantizes to
# deepseek_v4_fp8 and its FlashMLA kernel asserts fp8 kv-cache (else EngineCore dies).
"vllm_extra": "--tokenizer-mode deepseek_v4 --trust-remote-code --kv-cache-dtype fp8 --block-size 256",
"gpu_mem": "0.90", "startup_timeout": "2400"},

ghost Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify evidence for remote-code execution + token presence in this workflow.
# Expected: Lines showing both `HF_TOKEN` and `--trust-remote-code` in the same job.
nl -ba .github/workflows/nightly-bfcl.yml | sed -n '130,160p;184,194p;218,230p'

Repository: lightseekorg/smg

Length of output: 101


🏁 Script executed:

# Read the workflow file to examine the specific lines and context
cat -n .github/workflows/nightly-bfcl.yml | sed -n '130,165p'

Repository: lightseekorg/smg

Length of output: 2889


🏁 Script executed:

# Check for HF_TOKEN environment setup
cat -n .github/workflows/nightly-bfcl.yml | sed -n '180,200p'

Repository: lightseekorg/smg

Length of output: 1427


🏁 Script executed:

# Check if there are model revision pins anywhere (e.g., using `@revision` or similar)
grep -n "revision\|@\|deepseek\|minimax\|kimi" .github/workflows/nightly-bfcl.yml | head -20

Repository: lightseekorg/smg

Length of output: 1522


🏁 Script executed:

# Find scripts that handle model loading/launching
fd -t f "launch" . | head -10

Repository: lightseekorg/smg

Length of output: 336


🏁 Script executed:

# Check if any revision pins exist elsewhere (e.g., in Python setup or launch scripts)
grep -r "revision\|@[a-f0-9]\|--revision" . --include="*.sh" --include="*.py" | grep -i "deepseek\|minimax\|kimi" | head -20

Repository: lightseekorg/smg

Length of output: 42


🏁 Script executed:

# Check how vLLM is invoked with the model
grep -n "python.*vllm\|vLLM\|serve\|MODEL" .github/workflows/nightly-bfcl.yml | grep -A2 -B2 "vllm"

Repository: lightseekorg/smg

Length of output: 483


🏁 Script executed:

# Check launch_arm.sh for how models are specified/loaded
cat scripts/bfcl/launch_arm.sh | head -100

Repository: lightseekorg/smg

Length of output: 4598


🏁 Script executed:

# Also check register_bfcl_model.py to see if it accepts or enforces revision specs
cat bindings/python/src/smg/launch_router.py | grep -A10 -B10 "model\|revision" | head -50

Repository: lightseekorg/smg

Length of output: 42


Pin immutable model revisions when enabling --trust-remote-code on the deepseek-v4, minimax-m2.7, and kimi-k2.6 legs.

These legs execute remote model repo Python during startup while HF_TOKEN is present in the job environment. Without pinned revisions, the models pull from untagged references (default branch), creating a supply-chain path to token/runner compromise if an upstream repo is retagged or compromised. Pin immutable revisions in the workflow matrix (e.g., deepseek-ai/DeepSeek-V4-Flash@<commit-sha>) and ensure both the weight-staging and serving paths consume the same pinned revision before relying on nightly automation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly-bfcl.yml around lines 139 - 140, Pin immutable
model revisions for the deepseek-v4, minimax-m2.7, and kimi-k2.6 legs in the
workflow matrix to prevent supply-chain risks when executing remote code with
the --trust-remote-code flag in vllm_extra. Modify the model references to use
the format model-name@commit-sha (e.g.,
deepseek-ai/DeepSeek-V4-Flash@<commit-sha>) in the matrix configuration, and
ensure that both the weight-staging and serving execution paths reference the
same pinned revision to maintain consistency throughout the nightly automation
workflow.

{"name": "minimax-m2.7", "runner": "blackwell", "tp": 4,
"gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7",
"model": "MiniMaxAI/MiniMax-M2.7", "bfcl_model": "MiniMaxAI/MiniMax-M2.7-FC",
"vllm_tool": "minimax_m2", "vllm_reason": "minimax_m2",
"smg_tool": "minimax_m2", "smg_reason": "minimax",
"vllm_extra": "--trust-remote-code", "gpu_mem": "0.90", "startup_timeout": "2400"},
# Kimi-K2.6 loads native int4 weights (~500GB) -> fits TP=4 half-node.
{"name": "kimi-k2.6", "runner": "blackwell", "tp": 4,
"gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7",
"model": "moonshotai/Kimi-K2.6", "bfcl_model": "moonshotai/Kimi-K2.6-FC",
"vllm_tool": "kimi_k2", "vllm_reason": "kimi_k2",
"smg_tool": "kimik2",
"smg_reason": "kimi_k25", # TODO(confirm): no kimi_k2 SMG reasoning parser yet
# Generous startup: 1T MoE at TP=4 spends minutes on load + torch.compile +
# weight quant; the default 420s timed out mid-compile.
"vllm_extra": "--trust-remote-code", "gpu_mem": "0.90", "startup_timeout": "2400"},
]
only = os.environ.get("ONLY", "")
valid_names = [l["name"] for l in legs]
if only:
legs = [l for l in legs if l["name"] == only]
# All legs (H100 + Blackwell) run on PRs as a sanity check; the run stays
# cheap because the PR CATEGORIES below are a tiny non-live subset
# (simple_python,irrelevance) for every leg regardless of runner.
if only and not legs:
raise SystemExit(f"unknown leg {only!r}; valid: {valid_names}")
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
print("matrix=" + json.dumps({"include": legs}))
PY

bfcl-ab:
needs: build-wheel
if: ${{ !cancelled() && needs.build-wheel.result == 'success' }}
runs-on: 4-gpu-h100 # TP=2 per arm, both arms concurrent (GPUs 0,1 + 2,3)
# Generous: the nightly set adds multi_turn (multi-turn state simulation),
# which is much slower than the single-turn AST/live categories.
# Concise leg label (default would list every matrix field): model + runner + TP.
name: bfcl-ab ${{ matrix.model }} (${{ matrix.runner }} TP${{ matrix.tp }})
needs: [build-wheel, setup]
if: ${{ !cancelled() && needs.build-wheel.result == 'success' && needs.setup.result == 'success' }}
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.setup.outputs.matrix) }}
runs-on: ${{ matrix.runner }}
# Generous: the nightly set adds multi_turn (state simulation), much slower
# than the single-turn AST/live categories.
timeout-minutes: 240
permissions:
contents: read
env:
ROUTER_LOCAL_MODEL_PATH: /models
HF_HOME: /models
MODEL: ${{ github.event.inputs.model || 'Qwen/Qwen3.6-27B' }}
BFCL_MODEL_FC: ${{ github.event.inputs.bfcl_model || 'Qwen/Qwen3.6-27B-FC' }}
MODEL: ${{ github.event.inputs.model || matrix.model }}
BFCL_MODEL_FC: ${{ github.event.inputs.bfcl_model || matrix.bfcl_model }}
Comment on lines +204 to +205

ghost Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require only before applying model overrides

For a workflow_dispatch run where model and bfcl_model are set but only is left at its default empty value, this job-level override is applied to every matrix row while the row-specific parsers, GPU split, runner, and extra vLLM flags still come from qwen/gpt-oss/deepseek/etc. That produces several expensive A/B jobs with mismatched parser configs and weights rather than overriding a single selected leg as the input text promises; fail the setup job unless overrides are paired with only, or apply them only to the intended row.

Useful? React with 👍 / 👎.

# Auth for gated/private HF repos + higher download rate limits (used by the
# weight-staging step and by vLLM/SMG at serve time).
HF_TOKEN: ${{ secrets.HF_TOKEN }}
# PR = quick non-live sanity subset; schedule/dispatch = non_live + live + multi_turn.
CATEGORIES: ${{ github.event.inputs.categories || (github.event_name == 'pull_request' && 'simple_python,irrelevance' || 'simple_python,simple_java,simple_javascript,multiple,parallel,parallel_multiple,irrelevance,live_simple,live_multiple,live_parallel,live_parallel_multiple,live_irrelevance,live_relevance,multi_turn_base,multi_turn_miss_func,multi_turn_miss_param,multi_turn_long_context') }}
steps:
Expand All @@ -123,27 +207,32 @@ jobs:
- name: Register model in bfcl (no-op if bfcl already ships a handler)
run: python scripts/bfcl/register_bfcl_model.py --model-id "$MODEL" || true
- name: Stage model weights
run: bash scripts/ci_download_model.sh --gpu-tier 2
# Only this leg's model — not the whole GPU tier (which pulls unrelated,
# sometimes-gated repos this A/B never serves).
run: bash scripts/ci_download_model.sh "$MODEL"
- name: Launch arms + run official BFCL A/B
env:
BFCL_MODEL: ${{ env.MODEL }}
VLLM_BIN: vllm
VLLM_PYTHON: python
SMG_LAUNCH: "python -m smg.launch_router"
BFCL_VLLM_TOOL_PARSER: ${{ github.event.inputs.vllm_tool_parser || 'qwen3_xml' }}
BFCL_SMG_TOOL_PARSER: ${{ github.event.inputs.smg_tool_parser || 'qwen_xml' }}
BFCL_VLLM_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || 'qwen3' }}
BFCL_SMG_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || 'qwen3' }}
BFCL_TP: "2"
BFCL_VLLM_TOOL_PARSER: ${{ github.event.inputs.vllm_tool_parser || matrix.vllm_tool }}
BFCL_SMG_TOOL_PARSER: ${{ github.event.inputs.smg_tool_parser || matrix.smg_tool }}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
BFCL_VLLM_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || matrix.vllm_reason }}
BFCL_SMG_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || matrix.smg_reason }}
BFCL_VLLM_EXTRA: ${{ matrix.vllm_extra }}
BFCL_TP: ${{ matrix.tp }}
BFCL_MAX_MODEL_LEN: "16384"
BFCL_GPU_MEM_UTIL: "0.85"
BFCL_GPU_MEM_UTIL: ${{ matrix.gpu_mem }}
# Per-leg health-wait budget: big MoEs need minutes for load + compile + quant.
BFCL_STARTUP_TIMEOUT: ${{ matrix.startup_timeout }}
run: |
set -euo pipefail
# TP=2 per arm: Arm A (pure vLLM) on GPUs 0,1; Arm B (SMG -> vLLM gRPC) on GPUs 2,3.
# Ports are left unset so launch_arm.sh picks free ones — lets two bfcl
# runners share a node (bin-packed, hostNetwork) without port collisions.
A_URL=$(BFCL_GPU=0,1 bash scripts/bfcl/launch_arm.sh a)
B_URL=$(BFCL_GPU=2,3 bash scripts/bfcl/launch_arm.sh b)
# Both arms concurrent: arm A (pure vLLM) on the first GPU half, arm B
# (SMG -> vLLM gRPC) on the second. Ports left unset so launch_arm.sh
# picks free ones (lets bin-packed runners share a node).
A_URL=$(BFCL_GPU=${{ matrix.gpu_a }} bash scripts/bfcl/launch_arm.sh a)
B_URL=$(BFCL_GPU=${{ matrix.gpu_b }} bash scripts/bfcl/launch_arm.sh b)
echo "arm A: $A_URL arm B: $B_URL"
python scripts/bfcl/run_ab.py \
--baseline "vllm=$A_URL" \
Expand All @@ -157,12 +246,16 @@ jobs:
--tolerance 0.02 || echo "BFCL_AB_REGRESSION=1" >> "$GITHUB_ENV"
- name: Render summary
if: always()
run: cat "$RUNNER_TEMP/bfcl_ab.md" >> "$GITHUB_STEP_SUMMARY" || echo "no report produced" >> "$GITHUB_STEP_SUMMARY"
run: |
{
echo "## BFCL A/B — ${{ matrix.name }} (${{ matrix.model }})"
cat "$RUNNER_TEMP/bfcl_ab.md"
} >> "$GITHUB_STEP_SUMMARY" || echo "no report produced for ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY"
- name: Upload comparison
if: always()
uses: actions/upload-artifact@v7

ghost Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Read-only check for mutable action refs in this workflow.
nl -ba .github/workflows/nightly-bfcl.yml | rg -n 'uses:\s*[^#]+@v[0-9]+'

Repository: lightseekorg/smg

Length of output: 101


🏁 Script executed:

# First, check if the file exists and read around line 247
if [ -f ".github/workflows/nightly-bfcl.yml" ]; then
  echo "=== File exists, reading around line 247 ==="
  sed -n '240,255p' .github/workflows/nightly-bfcl.yml | cat -n
else
  echo "File not found"
  git ls-files | grep -i nightly
fi

Repository: lightseekorg/smg

Length of output: 830


🏁 Script executed:

# Count lines and find all 'uses' directives with version tags
echo "=== Total lines and all 'uses' with version tags ==="
wc -l .github/workflows/nightly-bfcl.yml
echo ""
echo "=== All 'uses' directives with `@v` or `@tags` ==="
grep -n "uses:" .github/workflows/nightly-bfcl.yml | head -20

Repository: lightseekorg/smg

Length of output: 538


🌐 Web query:

GitHub Actions security best practice pin commit SHA vs version tag

💡 Result:

In GitHub Actions, pinning to a full-length commit SHA is considered the security gold standard, whereas pinning to a version tag is a matter of convenience that carries inherent risks [1][2]. The fundamental difference lies in immutability: 1. Commit SHA (Immutable): A full-length commit SHA is a unique, fixed identifier for a specific state of a repository [3][4]. Once pinned to a SHA, your workflow is guaranteed to execute the exact same code every time, making it immune to "tag retargeting" or "force-push" attacks where a malicious actor alters the code behind a seemingly legitimate tag [1][2][5]. 2. Version Tag (Mutable): Tags (e.g., v4, v4.1.0) are references that can be moved or deleted by repository maintainers [1][5]. Even if you trust an action's author, a compromised account or repository could allow an attacker to update a tag to point to malicious code, which your workflows would then automatically adopt [1][2][4]. Best Practices: - Always use the full-length commit SHA (e.g., actions/checkout@3df4ab1) instead of an abbreviated one [3][4]. - Maintain readability and auditability by including the version tag as a comment in your workflow file (e.g., uses: actions/checkout@3df4ab1 # v4.0.0) [4][6]. - Automate the maintenance of these pins. Tools like Renovate (using presets such as helpers:pinGitHubActionDigests) or Dependabot can automatically open pull requests to update your SHAs when new releases are published, allowing you to review changes before merging [7][4][5]. - If you are an administrator, consider enabling organization or repository-level policies that enforce commit SHA pinning, which will cause any non-pinned actions to fail [1][4][8]. While pinning to a tag is permissible if you strictly trust the creator, it remains a secondary choice compared to the immutability and security provided by commit SHAs [1][2].

Citations:


🌐 Web query:

actions/upload-artifact@v7 commit SHA pinning security

💡 Result:

Pinning actions/upload-artifact to a commit SHA is a recommended security best practice for GitHub Actions workflows [1][2][3]. As of June 2026, the latest version is v7.0.1 [4][5]. Security Context Using a tag like @v7 is a mutable reference; if the action's repository is compromised, an attacker could force-push a new commit to that tag, potentially injecting malicious code into your workflows [1][3]. Pinning to an immutable 40-character commit SHA ensures that your workflow always executes the exact same code version, protecting against supply chain attacks [6][1][3]. Implementation To use SHA pinning, replace the tag in your workflow file with the full commit hash. To maintain readability and allow automated tools to track updates, it is standard practice to include the version as a comment [6][7][3]: uses: actions/upload-artifact@bbbca2d # v7.0.1 Automated Maintenance You do not need to manually hunt for SHA updates. Tools like Dependabot and Renovate support SHA-pinned actions and will automatically open pull requests to update the SHA when a new release is published, keeping your workflows both secure and current [6][7][3]. Policy Enforcement Note that many organizations now enforce SHA pinning via GitHub organization-level policies [8][7]. If such a policy is active, workflows using tag-based references (e.g., @v7) will fail [7]. Furthermore, if you maintain actions that reference other actions internally, those sub-actions should also be SHA-pinned to ensure compatibility with these strict security policies [7].

Citations:


Pin actions/upload-artifact@v7 to a commit SHA.

Line 247 uses a mutable version tag (@v7), which is vulnerable to tag retargeting attacks. Pinning to a full-length commit SHA (e.g., actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.1) eliminates this supply-chain risk by making the action reference immutable. This aligns with GitHub's security best practices for Actions.

Note: This same pattern exists elsewhere in the workflow (e.g., lines 72, 75, 92, 190–191). Consider fixing all mutable action references.

🧰 Tools
🪛 zizmor (1.25.2)

[error] 247-247: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly-bfcl.yml at line 247, The
`actions/upload-artifact@v7` action reference uses a mutable version tag which
poses a supply-chain security risk. Replace the `@v7` tag with a full-length
commit SHA (format: `actions/upload-artifact@{COMMIT_SHA} # v7.0.1`) to pin the
action to an immutable reference. Apply this same fix to all other mutable
action references in the workflow file (also present at lines 72, 75, 92, and
190-191) to ensure consistent security practices across the entire workflow.

Source: Linters/SAST tools

with:
name: bfcl-ab-${{ github.run_id }}
name: bfcl-ab-${{ matrix.name }}-${{ github.run_id }}
path: |
${{ runner.temp }}/bfcl_ab.md
${{ runner.temp }}/bfcl_ab.json
Expand Down
55 changes: 42 additions & 13 deletions scripts/bfcl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,19 +58,48 @@ Key env knobs for `launch_arm.sh`: `BFCL_GPU` (CUDA_VISIBLE_DEVICES, e.g. `0,1`)

`run_ab.py` exits non-zero if the candidate's overall accuracy drops more than `--tolerance` (default 2pp) below the baseline.

## Per-model parser flags (vLLM ~0.22.x)

| model family | pure-vLLM `--tool-call-parser` / `--reasoning-parser` | SMG `--tool-call-parser` / `--reasoning-parser` |
|---|---|---|
| Qwen3 dense Instruct (e.g. Qwen3-4B-Instruct-2507) | `hermes` / — | `qwen` / — |
| Qwen3 thinking | `hermes` / `qwen3` | `qwen` / `qwen3` |
| Qwen3-Coder / 3.5 / 3.6 | `qwen3_coder` / `qwen3` | `qwen_xml` / `qwen3` |
| DeepSeek V3/R1 | `deepseek_v3` / `deepseek_r1` | `deepseek` / `deepseek_r1` |
| DeepSeek V3.1/V3.2/V4 | `deepseek_v31` / `deepseek_v3` | `deepseek31` / `deepseek_v4` (DSML) / … |
| Kimi K2 | `kimi_k2` / — | `kimik2` / `kimi` |
| MiniMax M2 | `minimax_m2` / `minimax_m2` | `minimax_m2` / `minimax` |

> The mid-2026 SKUs (DeepSeek V4, Kimi K2.6, Qwen3.6, MiniMax M2.7) may use newer parser names; confirm against the installed vLLM build: `vllm serve --help | grep -A40 tool-call-parser`.
## Per-model parser flags (the nightly matrix)

| model (matrix leg) | runner | TP/arm | pure-vLLM `--tool-call-parser` / `--reasoning-parser` | SMG `--tool-call-parser` / `--reasoning-parser` |
|---|---|---|---|---|
| Qwen3.6-27B (`qwen3.6`) | `4-gpu-h100` | 2 | `qwen3_xml` / `qwen3` | `qwen_xml` / `qwen3` |
| gpt-oss-120b (`gpt-oss`) | `4-gpu-h100` | 2 | `openai` / — | _(none — SMG auto-routes harmony)_ / — |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
| DeepSeek-V4-Flash (`deepseek-v4`) | `blackwell` | 4 | `deepseek_v4` / `deepseek_v4` (+`--tokenizer-mode deepseek_v4 --trust-remote-code`) | `deepseek_v4` / `deepseek_v31`† |
| MiniMax-M2.7 (`minimax-m2.7`) | `blackwell` | 4 | `minimax_m2` / `minimax_m2` (+`--trust-remote-code`) | `minimax_m2` / `minimax` |
| Kimi-K2.6 int4 (`kimi-k2.6`) | `blackwell` | 4 | `kimi_k2` / `kimi_k2` (+`--trust-remote-code`) | `kimik2` / `kimi_k25`† |

> **gpt-oss has no SMG tool-call-parser.** SMG handles gpt-oss through its harmony
> pipeline (`model_gateway/src/routers/grpc/harmony/`), auto-activated by
> `HarmonyDetector` on a `gpt-oss` model id / `GptOssForCausalLM` architecture. So
> the SMG arm passes **no** `--tool-call-parser` (`BFCL_SMG_TOOL_PARSER=""`).
>
> **† Reasoning-parser fallbacks.** SMG's reasoning registry has no `deepseek_v4` or
> `kimi_k2` entry yet; the closest existing parsers (`deepseek_v31`, `kimi_k25`) are
> used. Confirm on the first nightly; adding exact parsers to `crates/reasoning_parser`
> is a follow-up if outputs diverge.
>
> The mid-2026 SKU ids and a couple of vLLM parser names may shift; confirm against
> the installed vLLM build: `vllm serve --help | grep -A40 tool-call-parser`.

## Matrix & runners

The nightly (`.github/workflows/nightly-bfcl.yml`) runs the A/B as a GitHub Actions
matrix — one leg per model, `fail-fast: false`, each on its own runner:

- `4-gpu-h100` — Qwen3.6-27B and gpt-oss-120b, TP=2 per arm (GPUs 0,1 + 2,3).
- `blackwell` (8×B300) — DeepSeek-V4-Flash, MiniMax-M2.7, Kimi-K2.6, TP=4 per arm
(GPUs 0-3 + 4-7).

All legs run **both arms concurrently**; no sequential path is needed because the
"flash"/int4 checkpoints fit half a node. Per the A/B's premise, model size is
irrelevant — a smaller same-family checkpoint exercises the identical parser, so the
matrix uses DeepSeek-V4-Flash and int4 Kimi-K2.6 to validate the `deepseek_v4` /
`kimi_k2` parsers without paying for the full production weights.

`workflow_dispatch` can target one leg via the `only` input and override
`model`/`bfcl_model`/parsers per run. PRs touching this pipeline run **all** legs
(H100 + Blackwell) as an end-to-end sanity check, but cheaply — the PR category set
is a tiny non-live subset (`simple_python,irrelevance`) for every leg.

## Gotchas discovered while bringing this up (read before debugging)

Expand Down
17 changes: 16 additions & 1 deletion scripts/bfcl/launch_arm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,12 @@ start() {
echo "[launch_arm] started $name (pid $(cat "$RUN_DIR/$name.pid")) -> $log" >&2
}

# Tail a logfile to stderr for real-time startup visibility in CI. Stdout is
# reserved for the base_url (captured by the caller's $(...)), so logs must go to
# stderr — which still streams live to the CI console. Prints the tail pid so the
# caller can stop it once the server is healthy.
stream_log() { tail -n +1 -F "$1" >&2 & echo $!; }

wait_http() { # wait_http <url> <timeout_s>
local url="$1" timeout="${2:-300}" waited=0
until curl -sf -m 3 "$url" >/dev/null 2>&1; do
Expand Down Expand Up @@ -100,7 +106,9 @@ case "$ARM" in
# shellcheck disable=SC2206 # intentional word-split of optional extra flags
[ -n "$VLLM_EXTRA" ] && cmd+=($VLLM_EXTRA)
start arm_a "$RUN_DIR/arm_a_vllm.log" "${cmd[@]}"
log_tail=$(stream_log "$RUN_DIR/arm_a_vllm.log")
wait_http "http://127.0.0.1:$ARM_A_PORT/health" "${BFCL_STARTUP_TIMEOUT:-420}"
kill "$log_tail" 2>/dev/null || true
Comment on lines +118 to +120

ghost Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Tail processes may be orphaned if health checks fail.

If wait_http or wait_grpc returns non-zero (timeout or failure), the script exits immediately due to set -euo pipefail, leaving the background tail process running. In CI, the runner typically cleans up child processes when the job terminates, but on development machines these tail processes might persist.

Consider adding a trap handler to ensure cleanup on early exit.

🧹 Proposed trap-based cleanup

Add near the top of the script (after variable declarations):

+TAIL_PIDS=()
+cleanup_tails() {
+  for pid in "${TAIL_PIDS[@]}"; do
+    kill "$pid" 2>/dev/null || true
+  done
+}
+trap cleanup_tails EXIT

Then modify the pattern to track PIDs:

-    log_tail=$(stream_log "$RUN_DIR/arm_a_vllm.log")
+    log_tail=$(stream_log "$RUN_DIR/arm_a_vllm.log")
+    TAIL_PIDS+=("$log_tail")
     wait_http "http://127.0.0.1:$ARM_A_PORT/health" "${BFCL_STARTUP_TIMEOUT:-420}"
-    kill "$log_tail" 2>/dev/null || true
+    kill "$log_tail" 2>/dev/null || true

Also applies to: 128-130, 144-146

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/bfcl/launch_arm.sh` around lines 109 - 111, The script uses
stream_log to create background tail processes and stores their PIDs in
variables like log_tail, but if wait_http or wait_grpc fails due to set -euo
pipefail, the script exits immediately without cleaning up these background
processes. Add a trap handler near the top of the script after variable
declarations that will be triggered on EXIT to ensure all tracked background
tail process PIDs are killed. Modify the pattern wherever stream_log is called
to capture and track the returned PID in a variable, then reference these
variables in the trap handler to clean up all background processes when the
script exits, whether normally or due to an error. This applies to the
stream_log calls for log_tail variables at multiple locations throughout the
script.

echo "http://127.0.0.1:$ARM_A_PORT"
;;

Expand All @@ -117,18 +125,25 @@ case "$ARM" in
# shellcheck disable=SC2206 # intentional word-split of optional extra flags
[ -n "$VLLM_EXTRA" ] && wcmd+=($VLLM_EXTRA)
start arm_b_worker "$RUN_DIR/arm_b_worker.log" "${wcmd[@]}"
log_tail=$(stream_log "$RUN_DIR/arm_b_worker.log")
wait_grpc "$ARM_B_GRPC_PORT" "${BFCL_STARTUP_TIMEOUT:-420}"
kill "$log_tail" 2>/dev/null || true
# 2) SMG gateway in front, exposing the OpenAI API.
# shellcheck disable=SC2206 # intentional word-split of SMG_LAUNCH
declare -a smg_cmd=(
$SMG_LAUNCH
--model-path "$MODEL"
--worker-urls "grpc://127.0.0.1:$ARM_B_GRPC_PORT"
--tool-call-parser "$SMG_TOOL_PARSER"
--host 0.0.0.0 --port "$ARM_B_GW_PORT"
)
# Empty tool-call-parser => let SMG auto-detect (e.g. gpt-oss routes through
# the harmony pipeline via HarmonyDetector; passing a parser there is wrong).
[ -n "$SMG_TOOL_PARSER" ] && smg_cmd+=(--tool-call-parser "$SMG_TOOL_PARSER")

ghost Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There is a bug that prevents BFCL_SMG_TOOL_PARSER="" from actually omitting the --tool-call-parser flag.

On line 36, SMG_TOOL_PARSER is initialized using the :- operator:

SMG_TOOL_PARSER="${BFCL_SMG_TOOL_PARSER:-qwen}"

Because of :-, if BFCL_SMG_TOOL_PARSER is set to an empty string (""), it falls back to "qwen". Thus, SMG_TOOL_PARSER will never be empty, and --tool-call-parser qwen will always be appended, breaking the gpt-oss leg.

To fix this, change line 36 to use ${BFCL_SMG_TOOL_PARSER-qwen} (without the colon), which only falls back to "qwen" if the variable is completely unset, preserving the explicit empty string.

Additionally, for better compatibility with set -e (to avoid returning a non-zero exit code when the condition is false), refactor the one-liner && into a standard if block.

Suggested change
[ -n "$SMG_TOOL_PARSER" ] && smg_cmd+=(--tool-call-parser "$SMG_TOOL_PARSER")
if [ -n "$SMG_TOOL_PARSER" ]; then
smg_cmd+=(--tool-call-parser "$SMG_TOOL_PARSER")
fi
References
  1. In shell scripts with set -e enabled, avoid chaining commands with && if you expect the script to exit immediately upon the failure of the first command.

ghost Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the SMG parser truly unset for gpt-oss

When the gpt-oss matrix leg sets BFCL_SMG_TOOL_PARSER to the empty matrix.smg_tool, launch_arm.sh still initializes SMG_TOOL_PARSER with ${BFCL_SMG_TOOL_PARSER:-qwen}, so the value becomes qwen; this added condition is then true and the gateway is launched with --tool-call-parser qwen. That bypasses the Harmony auto-detect path the gpt-oss leg depends on, so this leg is tested with the wrong parser or can fail at startup. The default needs to apply only when the variable is unset, not when it is explicitly empty.

Useful? React with 👍 / 👎.

[ -n "$SMG_REASONING_PARSER" ] && smg_cmd+=(--reasoning-parser "$SMG_REASONING_PARSER")
start arm_b_gateway "$RUN_DIR/arm_b_gateway.log" "${smg_cmd[@]}"
log_tail=$(stream_log "$RUN_DIR/arm_b_gateway.log")
wait_http "http://127.0.0.1:$ARM_B_GW_PORT/health" "${BFCL_STARTUP_TIMEOUT:-420}"
kill "$log_tail" 2>/dev/null || true
echo "http://127.0.0.1:$ARM_B_GW_PORT"
;;

Expand Down
Loading