Skip to content
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 173 additions & 34 deletions .github/workflows/nightly-bfcl.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,30 +22,34 @@ on:
- ".github/workflows/nightly-bfcl.yml"
workflow_dispatch:
inputs:
only:
description: "Run only this matrix leg (qwen3.6|gpt-oss|deepseek-v4|minimax-m2.7|kimi-k2.6); empty = all"
required: false
default: ""
model:
description: "HF model id (weights) served on both arms"
description: "Override HF model id (weights) for the selected leg; empty = matrix default. Override model and bfcl_model together — mismatched weights/handler pair if only one is overridden."
required: false
default: "Qwen/Qwen3.6-27B"
default: ""
bfcl_model:
description: "BFCL FC handler name (must end in -FC; auto-registered if new)"
description: "Override BFCL FC handler name (must end in -FC); empty = matrix default. Override model and bfcl_model together — mismatched weights/handler pair if only one is overridden."
required: false
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
default: "Qwen/Qwen3.6-27B-FC"
default: ""
categories:
description: "Comma-separated BFCL test categories"
required: false
default: "simple_python,simple_java,simple_javascript,multiple,parallel,parallel_multiple,irrelevance,live_simple,live_multiple,live_parallel,live_parallel_multiple,live_irrelevance,live_relevance,multi_turn_base,multi_turn_miss_func,multi_turn_miss_param,multi_turn_long_context"
vllm_tool_parser:
description: "pure-vLLM --tool-call-parser"
description: "Override pure-vLLM --tool-call-parser; empty = matrix default"
required: false
default: "qwen3_xml"
default: ""
smg_tool_parser:
description: "SMG --tool-call-parser"
description: "Override SMG --tool-call-parser; empty = matrix default (empty also = SMG auto-detect)"
required: false
default: "qwen_xml"
default: ""
reasoning_parser:
description: "reasoning parser for both arms (empty for non-thinking SKUs)"
description: "Override reasoning parser for BOTH arms; empty = matrix defaults"
required: false
default: "qwen3"
default: ""

concurrency:
group: nightly-bfcl-${{ github.event_name }}-${{ github.ref }}
Expand Down Expand Up @@ -91,20 +95,116 @@ jobs:
path: bindings/python/dist/*.whl
retention-days: 7

# Compute the model matrix as JSON so the dispatch `only` leg-filter can depend
# on inputs — the `matrix` context is not available in a job-level `if`.
setup:
# No needs: build-wheel — this job is pure CPU JSON generation with no wheel
# dependency. bfcl-ab still gates on build-wheel.result == 'success'.
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
matrix: ${{ steps.build.outputs.matrix }}
steps:
- id: build
env:
ONLY: ${{ github.event.inputs.only }}
MODEL_OVERRIDE: ${{ github.event.inputs.model }}
BFCL_OVERRIDE: ${{ github.event.inputs.bfcl_model }}
run: |
set -euo pipefail
# Enforce the paired-override contract: `model` and `bfcl_model` must be
# given together or not at all (an overridden model with the default FC
# handler — or vice versa — is a mismatched weights/handler pair).
if { [ -n "$MODEL_OVERRIDE" ] && [ -z "$BFCL_OVERRIDE" ]; } || \
{ [ -z "$MODEL_OVERRIDE" ] && [ -n "$BFCL_OVERRIDE" ]; }; then
echo "::error::Set BOTH 'model' and 'bfcl_model' (or neither): model='$MODEL_OVERRIDE' bfcl_model='$BFCL_OVERRIDE'"
exit 1
fi
python3 - <<'PY' >> "$GITHUB_OUTPUT"
import json, os
legs = [
# H100 legs (TP=2 per arm, both arms concurrent: GPUs 0,1 + 2,3).
{"name": "qwen3.6", "runner": "4-gpu-h100", "tp": 2,
"gpu_a": "0,1", "gpu_b": "2,3",
"model": "Qwen/Qwen3.6-27B", "bfcl_model": "Qwen/Qwen3.6-27B-FC",
"vllm_tool": "qwen3_xml", "vllm_reason": "qwen3",
"smg_tool": "qwen_xml", "smg_reason": "qwen3",
"vllm_extra": "", "gpu_mem": "0.85", "startup_timeout": "600",
"model_cache": "/models"},
# gpt-oss SMG arm passes NO tool parser (smg_tool empty) -> harmony auto-detect.
{"name": "gpt-oss", "runner": "4-gpu-h100", "tp": 2,
"gpu_a": "0,1", "gpu_b": "2,3",
"model": "openai/gpt-oss-120b", "bfcl_model": "openai/gpt-oss-120b-FC",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use matching BFCL handlers for non-Qwen legs

For new legs such as gpt-oss/DeepSeek/MiniMax/Kimi that bfcl-eval does not already ship, the existing registration step registers $MODEL through scripts/bfcl/register_bfcl_model.py with its default QwenFCHandler. In those scenarios BFCL formats and decodes Qwen-style XML instead of the native tool-call protocol these legs are meant to compare, so the run measures Qwen-prompt compatibility or invalid scores rather than SMG/vLLM parser parity; each non-Qwen leg needs a matching/generic FC handler or should avoid this fallback when unsupported.

Useful? React with 👍 / 👎.

"vllm_tool": "openai", "vllm_reason": "",
"smg_tool": "", "smg_reason": "",
"vllm_extra": "", "gpu_mem": "0.85", "startup_timeout": "600",
"model_cache": "/models"},
# Blackwell legs (B200, TP=4 per arm, both arms concurrent: 0-3 + 4-7).
# Models are pre-staged on NVMe at /raid/models/<id> (no download).
{"name": "deepseek-v4", "runner": "blackwell", "tp": 4,
"gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7",
"model": "deepseek-ai/DeepSeek-V4-Flash", "bfcl_model": "deepseek-ai/DeepSeek-V4-Flash-FC",
Comment on lines +145 to +147

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Download the matrix-selected checkpoint

For these new Blackwell matrix legs, the selected checkpoints are TP=4 models, but the later Stage model weights step is still hard-coded to scripts/ci_download_model.sh --gpu-tier 2; that script's tier resolver only downloads models with spec['tp'] <= tier (scripts/ci_download_model.sh:37-48). On a cold Blackwell runner these checkpoints are not pre-staged, so launch_arm.sh has only its 420s startup wait while vLLM tries to fetch hundreds of GB outside the download lock, which makes the new nightly legs time out before serving.

Useful? React with 👍 / 👎.

"vllm_tool": "deepseek_v4", "vllm_reason": "deepseek_v4",
"smg_tool": "deepseek_v4",
"smg_reason": "deepseek_v31", # TODO(confirm): no deepseek_v4 SMG reasoning parser yet
# --kv-cache-dtype fp8 + --block-size 256 REQUIRED: V4's FlashMLA fp8
# kernel asserts fp8 kv-cache (else EngineCore dies at startup).
"vllm_extra": "--tokenizer-mode deepseek_v4 --trust-remote-code --kv-cache-dtype fp8 --block-size 256",
"gpu_mem": "0.90", "startup_timeout": "2400", "model_cache": "/raid/models"},
{"name": "minimax-m2.7", "runner": "blackwell", "tp": 4,
"gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7",
"model": "MiniMaxAI/MiniMax-M2.7", "bfcl_model": "MiniMaxAI/MiniMax-M2.7-FC",
"vllm_tool": "minimax_m2", "vllm_reason": "minimax_m2",
"smg_tool": "minimax_m2", "smg_reason": "minimax",
"vllm_extra": "--trust-remote-code", "gpu_mem": "0.90", "startup_timeout": "2400",
"model_cache": "/raid/models"},
# Kimi-K2.6 loads native int4 weights (~500GB) -> fits TP=4 half-node.
{"name": "kimi-k2.6", "runner": "blackwell", "tp": 4,
"gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7",
"model": "moonshotai/Kimi-K2.6", "bfcl_model": "moonshotai/Kimi-K2.6-FC",
"vllm_tool": "kimi_k2", "vllm_reason": "kimi_k2",
"smg_tool": "kimik2",
"smg_reason": "kimi_k25", # TODO(confirm): no kimi_k2 SMG reasoning parser yet
# Generous startup: 1T MoE at TP=4 needs minutes for load+compile+quant (420s timed out).
"vllm_extra": "--trust-remote-code", "gpu_mem": "0.90", "startup_timeout": "2400",
"model_cache": "/raid/models"},
]
only = os.environ.get("ONLY", "")
valid_names = [l["name"] for l in legs]
if only:
legs = [l for l in legs if l["name"] == only]
# All legs (incl. Blackwell) run on PRs; cheap because the PR CATEGORIES
# below are a tiny non-live subset for every leg.
if only and not legs:
raise SystemExit(f"unknown leg {only!r}; valid: {valid_names}")
print("matrix=" + json.dumps({"include": legs}))
PY

bfcl-ab:
needs: build-wheel
if: ${{ !cancelled() && needs.build-wheel.result == 'success' }}
runs-on: 4-gpu-h100 # TP=2 per arm, both arms concurrent (GPUs 0,1 + 2,3)
# Generous: the nightly set adds multi_turn (multi-turn state simulation),
# which is much slower than the single-turn AST/live categories.
# Concise leg label (default would list every matrix field): model + runner + TP.
name: bfcl-ab ${{ matrix.model }} (${{ matrix.runner }} TP${{ matrix.tp }})
needs: [build-wheel, setup]
if: ${{ !cancelled() && needs.build-wheel.result == 'success' && needs.setup.result == 'success' }}
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.setup.outputs.matrix) }}
runs-on: ${{ matrix.runner }}
# Generous: the nightly set adds multi_turn (state simulation), much slower
# than the single-turn AST/live categories.
timeout-minutes: 240
permissions:
contents: read
env:
ROUTER_LOCAL_MODEL_PATH: /models
HF_HOME: /models
MODEL: ${{ github.event.inputs.model || 'Qwen/Qwen3.6-27B' }}
BFCL_MODEL_FC: ${{ github.event.inputs.bfcl_model || 'Qwen/Qwen3.6-27B-FC' }}
# Per-runner model store: /models on h100, /raid/models (NVMe) on blackwell.
# Models pre-staged at <cache>/<id> load locally; anything missing downloads
# into the same dir's HF cache.
ROUTER_LOCAL_MODEL_PATH: ${{ matrix.model_cache }}
HF_HOME: ${{ matrix.model_cache }}
MODEL: ${{ github.event.inputs.model || matrix.model }}
BFCL_MODEL_FC: ${{ github.event.inputs.bfcl_model || matrix.bfcl_model }}
Comment on lines +204 to +205

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require only before applying model overrides

For a workflow_dispatch run where model and bfcl_model are set but only is left at its default empty value, this job-level override is applied to every matrix row while the row-specific parsers, GPU split, runner, and extra vLLM flags still come from qwen/gpt-oss/deepseek/etc. That produces several expensive A/B jobs with mismatched parser configs and weights rather than overriding a single selected leg as the input text promises; fail the setup job unless overrides are paired with only, or apply them only to the intended row.

Useful? React with 👍 / 👎.

# Auth for gated/private HF repos + higher download rate limits.
HF_TOKEN: ${{ secrets.HF_TOKEN }}
# PR = quick non-live sanity subset; schedule/dispatch = non_live + live + multi_turn.
CATEGORIES: ${{ github.event.inputs.categories || (github.event_name == 'pull_request' && 'simple_python,irrelevance' || 'simple_python,simple_java,simple_javascript,multiple,parallel,parallel_multiple,irrelevance,live_simple,live_multiple,live_parallel,live_parallel_multiple,live_irrelevance,live_relevance,multi_turn_base,multi_turn_miss_func,multi_turn_miss_param,multi_turn_long_context') }}
steps:
Expand All @@ -123,27 +223,40 @@ jobs:
- name: Register model in bfcl (no-op if bfcl already ships a handler)
run: python scripts/bfcl/register_bfcl_model.py --model-id "$MODEL" || true
- name: Stage model weights
run: bash scripts/ci_download_model.sh --gpu-tier 2
# Skip if pre-staged locally at <cache>/<id> (e.g. blackwell NVMe); else
# download just this leg's model (not the whole GPU tier, which pulls
# unrelated, sometimes-gated repos this A/B never serves).
run: |
if [ -d "$ROUTER_LOCAL_MODEL_PATH/$MODEL" ]; then
echo "model present at $ROUTER_LOCAL_MODEL_PATH/$MODEL — skipping download"
else
bash scripts/ci_download_model.sh "$MODEL"
fi
- name: Launch arms + run official BFCL A/B
env:
BFCL_MODEL: ${{ env.MODEL }}
VLLM_BIN: vllm
VLLM_PYTHON: python
SMG_LAUNCH: "python -m smg.launch_router"
BFCL_VLLM_TOOL_PARSER: ${{ github.event.inputs.vllm_tool_parser || 'qwen3_xml' }}
BFCL_SMG_TOOL_PARSER: ${{ github.event.inputs.smg_tool_parser || 'qwen_xml' }}
BFCL_VLLM_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || 'qwen3' }}
BFCL_SMG_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || 'qwen3' }}
BFCL_TP: "2"
BFCL_VLLM_TOOL_PARSER: ${{ github.event.inputs.vllm_tool_parser || matrix.vllm_tool }}
BFCL_SMG_TOOL_PARSER: ${{ github.event.inputs.smg_tool_parser || matrix.smg_tool }}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
BFCL_VLLM_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || matrix.vllm_reason }}
BFCL_SMG_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || matrix.smg_reason }}
BFCL_VLLM_EXTRA: ${{ matrix.vllm_extra }}
BFCL_TP: ${{ matrix.tp }}
BFCL_MAX_MODEL_LEN: "16384"
BFCL_GPU_MEM_UTIL: "0.85"
BFCL_GPU_MEM_UTIL: ${{ matrix.gpu_mem }}
# Per-leg health-wait budget: big MoEs need minutes for load + compile + quant.
BFCL_STARTUP_TIMEOUT: ${{ matrix.startup_timeout }}
# Arm logs + pidfiles under RUNNER_TEMP (per-job, uploadable) not /tmp.
BFCL_RUN_DIR: ${{ runner.temp }}/bfcl_run

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the launch run directory during teardown

When this CI step sets BFCL_RUN_DIR here, the pidfiles are written under ${{ runner.temp }}/bfcl_run, but step-level env does not carry over to the later Teardown arms step, which runs launch_arm.sh stop with the script's default /tmp/bfcl_ab. In any run that reaches arm launch, the intended pidfile/process-group cleanup therefore finds no pids and leaves cleanup to the GPU nuke backstop, including leaking the non-GPU SMG gateway until the job exits.

Useful? React with 👍 / 👎.

run: |
set -euo pipefail
# TP=2 per arm: Arm A (pure vLLM) on GPUs 0,1; Arm B (SMG -> vLLM gRPC) on GPUs 2,3.
# Ports are left unset so launch_arm.sh picks free ones — lets two bfcl
# runners share a node (bin-packed, hostNetwork) without port collisions.
A_URL=$(BFCL_GPU=0,1 bash scripts/bfcl/launch_arm.sh a)
B_URL=$(BFCL_GPU=2,3 bash scripts/bfcl/launch_arm.sh b)
# Both arms concurrent: arm A (pure vLLM) on the first GPU half, arm B
# (SMG -> vLLM gRPC) on the second. Ports left unset so launch_arm.sh
# picks free ones (lets bin-packed runners share a node).
A_URL=$(BFCL_GPU=${{ matrix.gpu_a }} bash scripts/bfcl/launch_arm.sh a)
B_URL=$(BFCL_GPU=${{ matrix.gpu_b }} bash scripts/bfcl/launch_arm.sh b)
echo "arm A: $A_URL arm B: $B_URL"
python scripts/bfcl/run_ab.py \
--baseline "vllm=$A_URL" \
Expand All @@ -157,16 +270,42 @@ jobs:
--tolerance 0.02 || echo "BFCL_AB_REGRESSION=1" >> "$GITHUB_ENV"
- name: Render summary
if: always()
run: cat "$RUNNER_TEMP/bfcl_ab.md" >> "$GITHUB_STEP_SUMMARY" || echo "no report produced" >> "$GITHUB_STEP_SUMMARY"
- name: Upload comparison
run: |
{
echo "## BFCL A/B — ${{ matrix.name }} (${{ matrix.model }})"
cat "$RUNNER_TEMP/bfcl_ab.md"
} >> "$GITHUB_STEP_SUMMARY" || echo "no report produced for ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY"
- name: Upload results (summary + per-case scores)
if: always()
uses: actions/upload-artifact@v7

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Read-only check for mutable action refs in this workflow.
nl -ba .github/workflows/nightly-bfcl.yml | rg -n 'uses:\s*[^#]+@v[0-9]+'

Repository: lightseekorg/smg

Length of output: 101


🏁 Script executed:

# First, check if the file exists and read around line 247
if [ -f ".github/workflows/nightly-bfcl.yml" ]; then
  echo "=== File exists, reading around line 247 ==="
  sed -n '240,255p' .github/workflows/nightly-bfcl.yml | cat -n
else
  echo "File not found"
  git ls-files | grep -i nightly
fi

Repository: lightseekorg/smg

Length of output: 830


🏁 Script executed:

# Count lines and find all 'uses' directives with version tags
echo "=== Total lines and all 'uses' with version tags ==="
wc -l .github/workflows/nightly-bfcl.yml
echo ""
echo "=== All 'uses' directives with `@v` or `@tags` ==="
grep -n "uses:" .github/workflows/nightly-bfcl.yml | head -20

Repository: lightseekorg/smg

Length of output: 538


🌐 Web query:

GitHub Actions security best practice pin commit SHA vs version tag

💡 Result:

In GitHub Actions, pinning to a full-length commit SHA is considered the security gold standard, whereas pinning to a version tag is a matter of convenience that carries inherent risks [1][2]. The fundamental difference lies in immutability: 1. Commit SHA (Immutable): A full-length commit SHA is a unique, fixed identifier for a specific state of a repository [3][4]. Once pinned to a SHA, your workflow is guaranteed to execute the exact same code every time, making it immune to "tag retargeting" or "force-push" attacks where a malicious actor alters the code behind a seemingly legitimate tag [1][2][5]. 2. Version Tag (Mutable): Tags (e.g., v4, v4.1.0) are references that can be moved or deleted by repository maintainers [1][5]. Even if you trust an action's author, a compromised account or repository could allow an attacker to update a tag to point to malicious code, which your workflows would then automatically adopt [1][2][4]. Best Practices: - Always use the full-length commit SHA (e.g., actions/checkout@3df4ab1) instead of an abbreviated one [3][4]. - Maintain readability and auditability by including the version tag as a comment in your workflow file (e.g., uses: actions/checkout@3df4ab1 # v4.0.0) [4][6]. - Automate the maintenance of these pins. Tools like Renovate (using presets such as helpers:pinGitHubActionDigests) or Dependabot can automatically open pull requests to update your SHAs when new releases are published, allowing you to review changes before merging [7][4][5]. - If you are an administrator, consider enabling organization or repository-level policies that enforce commit SHA pinning, which will cause any non-pinned actions to fail [1][4][8]. While pinning to a tag is permissible if you strictly trust the creator, it remains a secondary choice compared to the immutability and security provided by commit SHAs [1][2].

Citations:


🌐 Web query:

actions/upload-artifact@v7 commit SHA pinning security

💡 Result:

Pinning actions/upload-artifact to a commit SHA is a recommended security best practice for GitHub Actions workflows [1][2][3]. As of June 2026, the latest version is v7.0.1 [4][5]. Security Context Using a tag like @v7 is a mutable reference; if the action's repository is compromised, an attacker could force-push a new commit to that tag, potentially injecting malicious code into your workflows [1][3]. Pinning to an immutable 40-character commit SHA ensures that your workflow always executes the exact same code version, protecting against supply chain attacks [6][1][3]. Implementation To use SHA pinning, replace the tag in your workflow file with the full commit hash. To maintain readability and allow automated tools to track updates, it is standard practice to include the version as a comment [6][7][3]: uses: actions/upload-artifact@bbbca2d # v7.0.1 Automated Maintenance You do not need to manually hunt for SHA updates. Tools like Dependabot and Renovate support SHA-pinned actions and will automatically open pull requests to update the SHA when a new release is published, keeping your workflows both secure and current [6][7][3]. Policy Enforcement Note that many organizations now enforce SHA pinning via GitHub organization-level policies [8][7]. If such a policy is active, workflows using tag-based references (e.g., @v7) will fail [7]. Furthermore, if you maintain actions that reference other actions internally, those sub-actions should also be SHA-pinned to ensure compatibility with these strict security policies [7].

Citations:


Pin actions/upload-artifact@v7 to a commit SHA.

Line 247 uses a mutable version tag (@v7), which is vulnerable to tag retargeting attacks. Pinning to a full-length commit SHA (e.g., actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.1) eliminates this supply-chain risk by making the action reference immutable. This aligns with GitHub's security best practices for Actions.

Note: This same pattern exists elsewhere in the workflow (e.g., lines 72, 75, 92, 190–191). Consider fixing all mutable action references.

🧰 Tools
🪛 zizmor (1.25.2)

[error] 247-247: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly-bfcl.yml at line 247, The
`actions/upload-artifact@v7` action reference uses a mutable version tag which
poses a supply-chain security risk. Replace the `@v7` tag with a full-length
commit SHA (format: `actions/upload-artifact@{COMMIT_SHA} # v7.0.1`) to pin the
action to an immutable reference. Apply this same fix to all other mutable
action references in the workflow file (also present at lines 72, 75, 92, and
190-191) to ensure consistent security practices across the entire workflow.

Source: Linters/SAST tools

with:
name: bfcl-ab-${{ github.run_id }}
name: bfcl-ab-${{ matrix.name }}-${{ github.run_id }}
# Summary + per-category score files (which list each failed case for
# both arms). Raw generate dumps (bfcl_ab/<arm>/result/**) excluded as bulky.
path: |
${{ runner.temp }}/bfcl_ab.md
${{ runner.temp }}/bfcl_ab.json
${{ runner.temp }}/bfcl_ab/*/score/**
if-no-files-found: warn
retention-days: 30
- name: Upload arm logs (failure only)
if: failure()
uses: actions/upload-artifact@v7
with:
name: bfcl-ab-${{ matrix.name }}-logs-${{ github.run_id }}
# vLLM/SMG logs — only useful on crash/timeout; noise on success.
path: ${{ runner.temp }}/bfcl_run/*.log
if-no-files-found: ignore
retention-days: 14
- name: Teardown arms
if: always()
run: bash scripts/bfcl/launch_arm.sh stop || true
- name: Release GPUs (nuke + verify, same util as nightly-benchmark)
if: always()
# Backstop the pidfile teardown above: kill ANY process still holding the
# GPUs (by occupancy, not name) and assert the node is actually free. This
# catches root-owned, setsid-detached vLLM workers that survive a
# hard-killed launch step — exactly the leak that pinned ~250 GiB/GPU
# across jobs. Each bfcl leg uses the whole node (TP fills it), so no
# co-tenant job shares these GPUs.
run: bash scripts/ci_killall_sglang.sh nuke_gpus || true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Let GPU cleanup failures fail the job

When scripts/ci_killall_sglang.sh nuke_gpus cannot free the device or cannot query it, that helper emits ::error:: and exits nonzero after the dirty-GPU check, but this || true masks the failure. In the exact cleanup scenario this step is meant to catch—surviving vLLM/worker processes holding GPU memory—the workflow will still report success and return a dirty runner to the pool, which can contaminate subsequent GPU jobs.

Useful? React with 👍 / 👎.

57 changes: 44 additions & 13 deletions scripts/bfcl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,19 +58,50 @@ Key env knobs for `launch_arm.sh`: `BFCL_GPU` (CUDA_VISIBLE_DEVICES, e.g. `0,1`)

`run_ab.py` exits non-zero if the candidate's overall accuracy drops more than `--tolerance` (default 2pp) below the baseline.

## Per-model parser flags (vLLM ~0.22.x)

| model family | pure-vLLM `--tool-call-parser` / `--reasoning-parser` | SMG `--tool-call-parser` / `--reasoning-parser` |
|---|---|---|
| Qwen3 dense Instruct (e.g. Qwen3-4B-Instruct-2507) | `hermes` / — | `qwen` / — |
| Qwen3 thinking | `hermes` / `qwen3` | `qwen` / `qwen3` |
| Qwen3-Coder / 3.5 / 3.6 | `qwen3_coder` / `qwen3` | `qwen_xml` / `qwen3` |
| DeepSeek V3/R1 | `deepseek_v3` / `deepseek_r1` | `deepseek` / `deepseek_r1` |
| DeepSeek V3.1/V3.2/V4 | `deepseek_v31` / `deepseek_v3` | `deepseek31` / `deepseek_v4` (DSML) / … |
| Kimi K2 | `kimi_k2` / — | `kimik2` / `kimi` |
| MiniMax M2 | `minimax_m2` / `minimax_m2` | `minimax_m2` / `minimax` |

> The mid-2026 SKUs (DeepSeek V4, Kimi K2.6, Qwen3.6, MiniMax M2.7) may use newer parser names; confirm against the installed vLLM build: `vllm serve --help | grep -A40 tool-call-parser`.
## Per-model parser flags (the nightly matrix)

| model (matrix leg) | runner | TP/arm | pure-vLLM `--tool-call-parser` / `--reasoning-parser` | SMG `--tool-call-parser` / `--reasoning-parser` |
|---|---|---|---|---|
| Qwen3.6-27B (`qwen3.6`) | `4-gpu-h100` | 2 | `qwen3_xml` / `qwen3` | `qwen_xml` / `qwen3` |
| gpt-oss-120b (`gpt-oss`) | `4-gpu-h100` | 2 | `openai` / — | _(none — SMG auto-routes harmony)_ / — |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
| DeepSeek-V4-Flash (`deepseek-v4`) | `blackwell` | 4 | `deepseek_v4` / `deepseek_v4` (+`--tokenizer-mode deepseek_v4 --trust-remote-code`) | `deepseek_v4` / `deepseek_v31`† |
| MiniMax-M2.7 (`minimax-m2.7`) | `blackwell` | 4 | `minimax_m2` / `minimax_m2` (+`--trust-remote-code`) | `minimax_m2` / `minimax` |
| Kimi-K2.6 int4 (`kimi-k2.6`) | `blackwell` | 4 | `kimi_k2` / `kimi_k2` (+`--trust-remote-code`) | `kimik2` / `kimi_k25`† |

> **gpt-oss has no SMG tool-call-parser.** SMG handles gpt-oss through its harmony
> pipeline (`model_gateway/src/routers/grpc/harmony/`), auto-activated by
> `HarmonyDetector` on a `gpt-oss` model id / `GptOssForCausalLM` architecture. So
> the SMG arm passes **no** `--tool-call-parser` (`BFCL_SMG_TOOL_PARSER=""`). The
> `—` in **both** reasoning-parser columns for gpt-oss is likewise intentional:
> harmony carries its own reasoning channel, so neither arm sets a reasoning parser.
>
> **† Reasoning-parser fallbacks.** SMG's reasoning registry has no `deepseek_v4` or
> `kimi_k2` entry yet; the closest existing parsers (`deepseek_v31`, `kimi_k25`) are
> used. Confirm on the first nightly; adding exact parsers to `crates/reasoning_parser`
> is a follow-up if outputs diverge.
>
> The mid-2026 SKU ids and a couple of vLLM parser names may shift; confirm against
> the installed vLLM build: `vllm serve --help | grep -A40 tool-call-parser`.

## Matrix & runners

The nightly (`.github/workflows/nightly-bfcl.yml`) runs the A/B as a GitHub Actions
matrix — one leg per model, `fail-fast: false`, each on its own runner:

- `4-gpu-h100` — Qwen3.6-27B and gpt-oss-120b, TP=2 per arm (GPUs 0,1 + 2,3).
- `blackwell` (8×B300) — DeepSeek-V4-Flash, MiniMax-M2.7, Kimi-K2.6, TP=4 per arm
(GPUs 0-3 + 4-7).

All legs run **both arms concurrently**; no sequential path is needed because the
"flash"/int4 checkpoints fit half a node. Per the A/B's premise, model size is
irrelevant — a smaller same-family checkpoint exercises the identical parser, so the
matrix uses DeepSeek-V4-Flash and int4 Kimi-K2.6 to validate the `deepseek_v4` /
`kimi_k2` parsers without paying for the full production weights.

`workflow_dispatch` can target one leg via the `only` input and override
`model`/`bfcl_model`/parsers per run. PRs touching this pipeline run **all** legs
(H100 + Blackwell) as an end-to-end sanity check, but cheaply — the PR category set
is a tiny non-live subset (`simple_python,irrelevance`) for every leg.

## Gotchas discovered while bringing this up (read before debugging)

Expand Down
Loading
Loading