Repository navigation
feat(bfcl): multi-model nightly A/B matrix on per-model runners #1791
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 14 commits
78559c3
77a07f3
de62d32
e5a16fa
0cbc70e
1a95dd6
5c923ee
27dfd69
b861468
ed1a623
a5d6edc
ea0ce36
6df6622
5dc5b4e
bda57e2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,30 +22,34 @@ on: | |
| - ".github/workflows/nightly-bfcl.yml" | ||
| workflow_dispatch: | ||
| inputs: | ||
| only: | ||
| description: "Run only this matrix leg (qwen3.6|gpt-oss|deepseek-v4|minimax-m2.7|kimi-k2.6); empty = all" | ||
| required: false | ||
| default: "" | ||
| model: | ||
| description: "HF model id (weights) served on both arms" | ||
| description: "Override HF model id (weights) for the selected leg; empty = matrix default. Override model and bfcl_model together — mismatched weights/handler pair if only one is overridden." | ||
| required: false | ||
| default: "Qwen/Qwen3.6-27B" | ||
| default: "" | ||
| bfcl_model: | ||
| description: "BFCL FC handler name (must end in -FC; auto-registered if new)" | ||
| description: "Override BFCL FC handler name (must end in -FC); empty = matrix default. Override model and bfcl_model together — mismatched weights/handler pair if only one is overridden." | ||
| required: false | ||
| default: "Qwen/Qwen3.6-27B-FC" | ||
| default: "" | ||
| categories: | ||
| description: "Comma-separated BFCL test categories" | ||
| required: false | ||
| default: "simple_python,simple_java,simple_javascript,multiple,parallel,parallel_multiple,irrelevance,live_simple,live_multiple,live_parallel,live_parallel_multiple,live_irrelevance,live_relevance,multi_turn_base,multi_turn_miss_func,multi_turn_miss_param,multi_turn_long_context" | ||
| vllm_tool_parser: | ||
| description: "pure-vLLM --tool-call-parser" | ||
| description: "Override pure-vLLM --tool-call-parser; empty = matrix default" | ||
| required: false | ||
| default: "qwen3_xml" | ||
| default: "" | ||
| smg_tool_parser: | ||
| description: "SMG --tool-call-parser" | ||
| description: "Override SMG --tool-call-parser; empty = matrix default (empty also = SMG auto-detect)" | ||
| required: false | ||
| default: "qwen_xml" | ||
| default: "" | ||
| reasoning_parser: | ||
| description: "reasoning parser for both arms (empty for non-thinking SKUs)" | ||
| description: "Override reasoning parser for BOTH arms; empty = matrix defaults" | ||
| required: false | ||
| default: "qwen3" | ||
| default: "" | ||
|
|
||
| concurrency: | ||
| group: nightly-bfcl-${{ github.event_name }}-${{ github.ref }} | ||
|
|
@@ -91,20 +95,116 @@ jobs: | |
| path: bindings/python/dist/*.whl | ||
| retention-days: 7 | ||
|
|
||
| # Compute the model matrix as JSON so the dispatch `only` leg-filter can depend | ||
| # on inputs — the `matrix` context is not available in a job-level `if`. | ||
| setup: | ||
| # No needs: build-wheel — this job is pure CPU JSON generation with no wheel | ||
| # dependency. bfcl-ab still gates on build-wheel.result == 'success'. | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| outputs: | ||
| matrix: ${{ steps.build.outputs.matrix }} | ||
| steps: | ||
| - id: build | ||
| env: | ||
| ONLY: ${{ github.event.inputs.only }} | ||
| MODEL_OVERRIDE: ${{ github.event.inputs.model }} | ||
| BFCL_OVERRIDE: ${{ github.event.inputs.bfcl_model }} | ||
| run: | | ||
| set -euo pipefail | ||
| # Enforce the paired-override contract: `model` and `bfcl_model` must be | ||
| # given together or not at all (an overridden model with the default FC | ||
| # handler — or vice versa — is a mismatched weights/handler pair). | ||
| if { [ -n "$MODEL_OVERRIDE" ] && [ -z "$BFCL_OVERRIDE" ]; } || \ | ||
| { [ -z "$MODEL_OVERRIDE" ] && [ -n "$BFCL_OVERRIDE" ]; }; then | ||
| echo "::error::Set BOTH 'model' and 'bfcl_model' (or neither): model='$MODEL_OVERRIDE' bfcl_model='$BFCL_OVERRIDE'" | ||
| exit 1 | ||
| fi | ||
| python3 - <<'PY' >> "$GITHUB_OUTPUT" | ||
| import json, os | ||
| legs = [ | ||
| # H100 legs (TP=2 per arm, both arms concurrent: GPUs 0,1 + 2,3). | ||
| {"name": "qwen3.6", "runner": "4-gpu-h100", "tp": 2, | ||
| "gpu_a": "0,1", "gpu_b": "2,3", | ||
| "model": "Qwen/Qwen3.6-27B", "bfcl_model": "Qwen/Qwen3.6-27B-FC", | ||
| "vllm_tool": "qwen3_xml", "vllm_reason": "qwen3", | ||
| "smg_tool": "qwen_xml", "smg_reason": "qwen3", | ||
| "vllm_extra": "", "gpu_mem": "0.85", "startup_timeout": "600", | ||
| "model_cache": "/models"}, | ||
| # gpt-oss SMG arm passes NO tool parser (smg_tool empty) -> harmony auto-detect. | ||
| {"name": "gpt-oss", "runner": "4-gpu-h100", "tp": 2, | ||
| "gpu_a": "0,1", "gpu_b": "2,3", | ||
| "model": "openai/gpt-oss-120b", "bfcl_model": "openai/gpt-oss-120b-FC", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For new legs such as gpt-oss/DeepSeek/MiniMax/Kimi that Useful? React with 👍 / 👎. |
||
| "vllm_tool": "openai", "vllm_reason": "", | ||
| "smg_tool": "", "smg_reason": "", | ||
| "vllm_extra": "", "gpu_mem": "0.85", "startup_timeout": "600", | ||
| "model_cache": "/models"}, | ||
| # Blackwell legs (B200, TP=4 per arm, both arms concurrent: 0-3 + 4-7). | ||
| # Models are pre-staged on NVMe at /raid/models/<id> (no download). | ||
| {"name": "deepseek-v4", "runner": "blackwell", "tp": 4, | ||
| "gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7", | ||
| "model": "deepseek-ai/DeepSeek-V4-Flash", "bfcl_model": "deepseek-ai/DeepSeek-V4-Flash-FC", | ||
|
Comment on lines
+145
to
+147
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For these new Blackwell matrix legs, the selected checkpoints are TP=4 models, but the later Useful? React with 👍 / 👎. |
||
| "vllm_tool": "deepseek_v4", "vllm_reason": "deepseek_v4", | ||
| "smg_tool": "deepseek_v4", | ||
| "smg_reason": "deepseek_v31", # TODO(confirm): no deepseek_v4 SMG reasoning parser yet | ||
| # --kv-cache-dtype fp8 + --block-size 256 REQUIRED: V4's FlashMLA fp8 | ||
| # kernel asserts fp8 kv-cache (else EngineCore dies at startup). | ||
| "vllm_extra": "--tokenizer-mode deepseek_v4 --trust-remote-code --kv-cache-dtype fp8 --block-size 256", | ||
| "gpu_mem": "0.90", "startup_timeout": "2400", "model_cache": "/raid/models"}, | ||
| {"name": "minimax-m2.7", "runner": "blackwell", "tp": 4, | ||
| "gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7", | ||
| "model": "MiniMaxAI/MiniMax-M2.7", "bfcl_model": "MiniMaxAI/MiniMax-M2.7-FC", | ||
| "vllm_tool": "minimax_m2", "vllm_reason": "minimax_m2", | ||
| "smg_tool": "minimax_m2", "smg_reason": "minimax", | ||
| "vllm_extra": "--trust-remote-code", "gpu_mem": "0.90", "startup_timeout": "2400", | ||
| "model_cache": "/raid/models"}, | ||
| # Kimi-K2.6 loads native int4 weights (~500GB) -> fits TP=4 half-node. | ||
| {"name": "kimi-k2.6", "runner": "blackwell", "tp": 4, | ||
| "gpu_a": "0,1,2,3", "gpu_b": "4,5,6,7", | ||
| "model": "moonshotai/Kimi-K2.6", "bfcl_model": "moonshotai/Kimi-K2.6-FC", | ||
| "vllm_tool": "kimi_k2", "vllm_reason": "kimi_k2", | ||
| "smg_tool": "kimik2", | ||
| "smg_reason": "kimi_k25", # TODO(confirm): no kimi_k2 SMG reasoning parser yet | ||
| # Generous startup: 1T MoE at TP=4 needs minutes for load+compile+quant (420s timed out). | ||
| "vllm_extra": "--trust-remote-code", "gpu_mem": "0.90", "startup_timeout": "2400", | ||
| "model_cache": "/raid/models"}, | ||
| ] | ||
| only = os.environ.get("ONLY", "") | ||
| valid_names = [l["name"] for l in legs] | ||
| if only: | ||
| legs = [l for l in legs if l["name"] == only] | ||
| # All legs (incl. Blackwell) run on PRs; cheap because the PR CATEGORIES | ||
| # below are a tiny non-live subset for every leg. | ||
| if only and not legs: | ||
| raise SystemExit(f"unknown leg {only!r}; valid: {valid_names}") | ||
| print("matrix=" + json.dumps({"include": legs})) | ||
| PY | ||
|
|
||
| bfcl-ab: | ||
| needs: build-wheel | ||
| if: ${{ !cancelled() && needs.build-wheel.result == 'success' }} | ||
| runs-on: 4-gpu-h100 # TP=2 per arm, both arms concurrent (GPUs 0,1 + 2,3) | ||
| # Generous: the nightly set adds multi_turn (multi-turn state simulation), | ||
| # which is much slower than the single-turn AST/live categories. | ||
| # Concise leg label (default would list every matrix field): model + runner + TP. | ||
| name: bfcl-ab ${{ matrix.model }} (${{ matrix.runner }} TP${{ matrix.tp }}) | ||
| needs: [build-wheel, setup] | ||
| if: ${{ !cancelled() && needs.build-wheel.result == 'success' && needs.setup.result == 'success' }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: ${{ fromJSON(needs.setup.outputs.matrix) }} | ||
| runs-on: ${{ matrix.runner }} | ||
| # Generous: the nightly set adds multi_turn (state simulation), much slower | ||
| # than the single-turn AST/live categories. | ||
| timeout-minutes: 240 | ||
| permissions: | ||
| contents: read | ||
| env: | ||
| ROUTER_LOCAL_MODEL_PATH: /models | ||
| HF_HOME: /models | ||
| MODEL: ${{ github.event.inputs.model || 'Qwen/Qwen3.6-27B' }} | ||
| BFCL_MODEL_FC: ${{ github.event.inputs.bfcl_model || 'Qwen/Qwen3.6-27B-FC' }} | ||
| # Per-runner model store: /models on h100, /raid/models (NVMe) on blackwell. | ||
| # Models pre-staged at <cache>/<id> load locally; anything missing downloads | ||
| # into the same dir's HF cache. | ||
| ROUTER_LOCAL_MODEL_PATH: ${{ matrix.model_cache }} | ||
| HF_HOME: ${{ matrix.model_cache }} | ||
| MODEL: ${{ github.event.inputs.model || matrix.model }} | ||
| BFCL_MODEL_FC: ${{ github.event.inputs.bfcl_model || matrix.bfcl_model }} | ||
|
Comment on lines
+204
to
+205
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
For a Useful? React with 👍 / 👎. |
||
| # Auth for gated/private HF repos + higher download rate limits. | ||
| HF_TOKEN: ${{ secrets.HF_TOKEN }} | ||
| # PR = quick non-live sanity subset; schedule/dispatch = non_live + live + multi_turn. | ||
| CATEGORIES: ${{ github.event.inputs.categories || (github.event_name == 'pull_request' && 'simple_python,irrelevance' || 'simple_python,simple_java,simple_javascript,multiple,parallel,parallel_multiple,irrelevance,live_simple,live_multiple,live_parallel,live_parallel_multiple,live_irrelevance,live_relevance,multi_turn_base,multi_turn_miss_func,multi_turn_miss_param,multi_turn_long_context') }} | ||
| steps: | ||
|
|
@@ -123,27 +223,40 @@ jobs: | |
| - name: Register model in bfcl (no-op if bfcl already ships a handler) | ||
| run: python scripts/bfcl/register_bfcl_model.py --model-id "$MODEL" || true | ||
| - name: Stage model weights | ||
| run: bash scripts/ci_download_model.sh --gpu-tier 2 | ||
| # Skip if pre-staged locally at <cache>/<id> (e.g. blackwell NVMe); else | ||
| # download just this leg's model (not the whole GPU tier, which pulls | ||
| # unrelated, sometimes-gated repos this A/B never serves). | ||
| run: | | ||
| if [ -d "$ROUTER_LOCAL_MODEL_PATH/$MODEL" ]; then | ||
| echo "model present at $ROUTER_LOCAL_MODEL_PATH/$MODEL — skipping download" | ||
| else | ||
| bash scripts/ci_download_model.sh "$MODEL" | ||
| fi | ||
| - name: Launch arms + run official BFCL A/B | ||
| env: | ||
| BFCL_MODEL: ${{ env.MODEL }} | ||
| VLLM_BIN: vllm | ||
| VLLM_PYTHON: python | ||
| SMG_LAUNCH: "python -m smg.launch_router" | ||
| BFCL_VLLM_TOOL_PARSER: ${{ github.event.inputs.vllm_tool_parser || 'qwen3_xml' }} | ||
| BFCL_SMG_TOOL_PARSER: ${{ github.event.inputs.smg_tool_parser || 'qwen_xml' }} | ||
| BFCL_VLLM_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || 'qwen3' }} | ||
| BFCL_SMG_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || 'qwen3' }} | ||
| BFCL_TP: "2" | ||
| BFCL_VLLM_TOOL_PARSER: ${{ github.event.inputs.vllm_tool_parser || matrix.vllm_tool }} | ||
| BFCL_SMG_TOOL_PARSER: ${{ github.event.inputs.smg_tool_parser || matrix.smg_tool }} | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| BFCL_VLLM_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || matrix.vllm_reason }} | ||
| BFCL_SMG_REASONING_PARSER: ${{ github.event.inputs.reasoning_parser || matrix.smg_reason }} | ||
| BFCL_VLLM_EXTRA: ${{ matrix.vllm_extra }} | ||
| BFCL_TP: ${{ matrix.tp }} | ||
| BFCL_MAX_MODEL_LEN: "16384" | ||
| BFCL_GPU_MEM_UTIL: "0.85" | ||
| BFCL_GPU_MEM_UTIL: ${{ matrix.gpu_mem }} | ||
| # Per-leg health-wait budget: big MoEs need minutes for load + compile + quant. | ||
| BFCL_STARTUP_TIMEOUT: ${{ matrix.startup_timeout }} | ||
| # Arm logs + pidfiles under RUNNER_TEMP (per-job, uploadable) not /tmp. | ||
| BFCL_RUN_DIR: ${{ runner.temp }}/bfcl_run | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When this CI step sets Useful? React with 👍 / 👎. |
||
| run: | | ||
| set -euo pipefail | ||
| # TP=2 per arm: Arm A (pure vLLM) on GPUs 0,1; Arm B (SMG -> vLLM gRPC) on GPUs 2,3. | ||
| # Ports are left unset so launch_arm.sh picks free ones — lets two bfcl | ||
| # runners share a node (bin-packed, hostNetwork) without port collisions. | ||
| A_URL=$(BFCL_GPU=0,1 bash scripts/bfcl/launch_arm.sh a) | ||
| B_URL=$(BFCL_GPU=2,3 bash scripts/bfcl/launch_arm.sh b) | ||
| # Both arms concurrent: arm A (pure vLLM) on the first GPU half, arm B | ||
| # (SMG -> vLLM gRPC) on the second. Ports left unset so launch_arm.sh | ||
| # picks free ones (lets bin-packed runners share a node). | ||
| A_URL=$(BFCL_GPU=${{ matrix.gpu_a }} bash scripts/bfcl/launch_arm.sh a) | ||
| B_URL=$(BFCL_GPU=${{ matrix.gpu_b }} bash scripts/bfcl/launch_arm.sh b) | ||
| echo "arm A: $A_URL arm B: $B_URL" | ||
| python scripts/bfcl/run_ab.py \ | ||
| --baseline "vllm=$A_URL" \ | ||
|
|
@@ -157,16 +270,42 @@ jobs: | |
| --tolerance 0.02 || echo "BFCL_AB_REGRESSION=1" >> "$GITHUB_ENV" | ||
| - name: Render summary | ||
| if: always() | ||
| run: cat "$RUNNER_TEMP/bfcl_ab.md" >> "$GITHUB_STEP_SUMMARY" || echo "no report produced" >> "$GITHUB_STEP_SUMMARY" | ||
| - name: Upload comparison | ||
| run: | | ||
| { | ||
| echo "## BFCL A/B — ${{ matrix.name }} (${{ matrix.model }})" | ||
| cat "$RUNNER_TEMP/bfcl_ab.md" | ||
| } >> "$GITHUB_STEP_SUMMARY" || echo "no report produced for ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY" | ||
| - name: Upload results (summary + per-case scores) | ||
| if: always() | ||
| uses: actions/upload-artifact@v7 | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Read-only check for mutable action refs in this workflow.
nl -ba .github/workflows/nightly-bfcl.yml | rg -n 'uses:\s*[^#]+@v[0-9]+'Repository: lightseekorg/smg Length of output: 101 🏁 Script executed: # First, check if the file exists and read around line 247
if [ -f ".github/workflows/nightly-bfcl.yml" ]; then
echo "=== File exists, reading around line 247 ==="
sed -n '240,255p' .github/workflows/nightly-bfcl.yml | cat -n
else
echo "File not found"
git ls-files | grep -i nightly
fiRepository: lightseekorg/smg Length of output: 830 🏁 Script executed: # Count lines and find all 'uses' directives with version tags
echo "=== Total lines and all 'uses' with version tags ==="
wc -l .github/workflows/nightly-bfcl.yml
echo ""
echo "=== All 'uses' directives with `@v` or `@tags` ==="
grep -n "uses:" .github/workflows/nightly-bfcl.yml | head -20Repository: lightseekorg/smg Length of output: 538 🌐 Web query:
💡 Result: In GitHub Actions, pinning to a full-length commit SHA is considered the security gold standard, whereas pinning to a version tag is a matter of convenience that carries inherent risks [1][2]. The fundamental difference lies in immutability: 1. Commit SHA (Immutable): A full-length commit SHA is a unique, fixed identifier for a specific state of a repository [3][4]. Once pinned to a SHA, your workflow is guaranteed to execute the exact same code every time, making it immune to "tag retargeting" or "force-push" attacks where a malicious actor alters the code behind a seemingly legitimate tag [1][2][5]. 2. Version Tag (Mutable): Tags (e.g., v4, v4.1.0) are references that can be moved or deleted by repository maintainers [1][5]. Even if you trust an action's author, a compromised account or repository could allow an attacker to update a tag to point to malicious code, which your workflows would then automatically adopt [1][2][4]. Best Practices: - Always use the full-length commit SHA (e.g., actions/checkout@3df4ab1) instead of an abbreviated one [3][4]. - Maintain readability and auditability by including the version tag as a comment in your workflow file (e.g., uses: actions/checkout@3df4ab1 # v4.0.0) [4][6]. - Automate the maintenance of these pins. Tools like Renovate (using presets such as helpers:pinGitHubActionDigests) or Dependabot can automatically open pull requests to update your SHAs when new releases are published, allowing you to review changes before merging [7][4][5]. - If you are an administrator, consider enabling organization or repository-level policies that enforce commit SHA pinning, which will cause any non-pinned actions to fail [1][4][8]. While pinning to a tag is permissible if you strictly trust the creator, it remains a secondary choice compared to the immutability and security provided by commit SHAs [1][2]. Citations:
🌐 Web query:
💡 Result: Pinning actions/upload-artifact to a commit SHA is a recommended security best practice for GitHub Actions workflows [1][2][3]. As of June 2026, the latest version is v7.0.1 [4][5]. Security Context Using a tag like Citations:
Pin Line 247 uses a mutable version tag ( Note: This same pattern exists elsewhere in the workflow (e.g., lines 72, 75, 92, 190–191). Consider fixing all mutable action references. 🧰 Tools🪛 zizmor (1.25.2)[error] 247-247: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) 🤖 Prompt for AI AgentsSource: Linters/SAST tools |
||
| with: | ||
| name: bfcl-ab-${{ github.run_id }} | ||
| name: bfcl-ab-${{ matrix.name }}-${{ github.run_id }} | ||
| # Summary + per-category score files (which list each failed case for | ||
| # both arms). Raw generate dumps (bfcl_ab/<arm>/result/**) excluded as bulky. | ||
| path: | | ||
| ${{ runner.temp }}/bfcl_ab.md | ||
| ${{ runner.temp }}/bfcl_ab.json | ||
| ${{ runner.temp }}/bfcl_ab/*/score/** | ||
| if-no-files-found: warn | ||
| retention-days: 30 | ||
| - name: Upload arm logs (failure only) | ||
| if: failure() | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: bfcl-ab-${{ matrix.name }}-logs-${{ github.run_id }} | ||
| # vLLM/SMG logs — only useful on crash/timeout; noise on success. | ||
| path: ${{ runner.temp }}/bfcl_run/*.log | ||
| if-no-files-found: ignore | ||
| retention-days: 14 | ||
| - name: Teardown arms | ||
| if: always() | ||
| run: bash scripts/bfcl/launch_arm.sh stop || true | ||
| - name: Release GPUs (nuke + verify, same util as nightly-benchmark) | ||
| if: always() | ||
| # Backstop the pidfile teardown above: kill ANY process still holding the | ||
| # GPUs (by occupancy, not name) and assert the node is actually free. This | ||
| # catches root-owned, setsid-detached vLLM workers that survive a | ||
| # hard-killed launch step — exactly the leak that pinned ~250 GiB/GPU | ||
| # across jobs. Each bfcl leg uses the whole node (TP fills it), so no | ||
| # co-tenant job shares these GPUs. | ||
| run: bash scripts/ci_killall_sglang.sh nuke_gpus || true | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When Useful? React with 👍 / 👎. |
||
Uh oh!
There was an error while loading. Please reload this page.