chore(e2e): include model size in gpt-oss nightly benchmark slug by CatherineSue · Pull Request #384 · smg-project/smg

CatherineSue · 2026-02-09T20:05:06Z

Description

Problem

The nightly benchmark slug for openai/gpt-oss-20b is GptOss, which omits the model size. Every other model includes its parameter count in the slug (Llama8b, Qwen7b, Deepseek7b, etc.). This makes job names vague and ambiguous—especially since a gpt-oss-120b variant also exists in the codebase.

Solution

Rename the slug from GptOss to GptOss20b so the generated test class names and CI job names explicitly include the model size.

Changes

e2e_test/benchmarks/test_nightly_perf.py: "GptOss" → "GptOss20b" in _NIGHTLY_MODELS
.github/workflows/nightly-benchmark.yml: update test_class references in both single-worker and multi-worker matrices (TestNightlyGptOssSingle → TestNightlyGptOss20bSingle, TestNightlyGptOssMulti → TestNightlyGptOss20bMulti)

Test Plan

No functional change—only renames the slug used for test class generation and CI job naming.

Checklist

cargo +nightly fmt passes
cargo clippy --all-targets --all-features -- -D warnings passes
(Optional) Documentation updated

Summary by CodeRabbit

Chores
- Updated benchmarking test infrastructure and model configurations for improved test classification and tracking.

Rename GptOss → GptOss20b so the nightly job names include the parameter count, consistent with every other model (Llama8b, Qwen7b, Deepseek7b, etc.).

gemini-code-assist · 2026-02-09T20:05:23Z

Summary of Changes

Hello @CatherineSue, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an ambiguity in nightly benchmark naming by standardizing the slug for the openai/gpt-oss-20b model. By incorporating the model's parameter count into its slug, the change ensures consistency with other models and prevents confusion, especially given the existence of other gpt-oss variants. This improves the clarity and specificity of generated test class names and CI job names.

Highlights

Model Slug Renaming: The GptOss slug for the openai/gpt-oss-20b model has been updated to GptOss20b to explicitly include its model size, improving clarity in benchmark naming.
CI Workflow Updates: References to the old GptOss test class names in the .github/workflows/nightly-benchmark.yml file have been updated to reflect the new GptOss20b naming convention for both single and multi-worker configurations.

Changelog

e2e_test/benchmarks/test_nightly_perf.py
- Updated the model slug for openai/gpt-oss-20b from "GptOss" to "GptOss20b" within the _NIGHTLY_MODELS configuration.

Ignored Files

Ignored by pattern: .github/workflows/** (1)
- .github/workflows/nightly-benchmark.yml

Activity

No specific activity (comments, reviews, progress) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

coderabbitai · 2026-02-09T20:05:23Z

📝 Walkthrough

Walkthrough

The PR renames test class identifiers for the OpenAI GPT-OSS 20B model across nightly benchmark CI workflow and test configuration, standardizing the naming from generic "GptOss" to explicit "GptOss20b" for both single-worker and multi-worker test classes.

Changes

Cohort / File(s)	Summary
Nightly Benchmark Configuration `.github/workflows/nightly-benchmark.yml`	Updated test_class values for OpenAI GPT-OSS 20B model: `TestNightlyGptOssSingle` → `TestNightlyGptOss20bSingle` and `TestNightlyGptOssMulti` → `TestNightlyGptOss20bMulti` in workflow matrix definitions.
Nightly Performance Tests `e2e_test/benchmarks/test_nightly_perf.py`	Updated model entry naming fragment from "GptOss" to "GptOss20b" in the _NIGHTLY_MODELS configuration, aligning test class generation with CI workflow updates.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

refactor(ci): split nightly benchmark into per-model jobs #327: Modifies the same nightly benchmark matrix and test-class naming for the openai/gpt-oss-20b model across CI workflow and test configuration files.

Suggested labels

ci, tests

Suggested reviewers

key4ng
XinyueZhang369

Poem

🐰 A rabbit hops through benchmarks bright,
Renaming classes left and right,
From "GptOss" plain to "GptOss20b",
Each test class now knows what it'll be! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title accurately summarizes the main change: updating the gpt-oss nightly benchmark slug to include the model size (20b), making test class names and CI jobs less ambiguous.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch chore/nightly-gpt-oss-20b-slug

No actionable comments were generated in the recent review. 🎉

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

gemini-code-assist

Code Review

The pull request effectively addresses the ambiguity in the gpt-oss model slug by explicitly including the model size, 20b, in its name. This change enhances clarity and consistency across the benchmark suite, aligning with the naming conventions of other models and preventing potential confusion with different gpt-oss variants. The update is straightforward and improves the maintainability of the benchmark configuration.

gemini-code-assist · 2026-02-09T20:09:10Z

    ("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", "Deepseek7b", 4, ["http", "grpc"], {}),
    ("Qwen/Qwen3-30B-A3B", "Qwen30b", 4, ["http", "grpc"], {}),
    ("mistralai/Mistral-7B-Instruct-v0.3", "Mistral7b", 4, ["http", "grpc"], {}),
-    ("openai/gpt-oss-20b", "GptOss", 4, ["http", "grpc"], {}),


This change correctly updates the slug to include the model size, 20b. This improves clarity and consistency with other model naming conventions, making it easier to distinguish between different gpt-oss variants in job names and test classes.

Suggested change

("openai/gpt-oss-20b", "GptOss", 4, ["http", "grpc"], {}),

("openai/gpt-oss-20b", "GptOss20b", 4, ["http", "grpc"], {}),

Signed-off-by: ppraneth <pranethparuchuri@gmail.com>

chore(e2e): include model size in gpt-oss nightly benchmark slug

8d26111

Rename GptOss → GptOss20b so the nightly job names include the parameter count, consistent with every other model (Llama8b, Qwen7b, Deepseek7b, etc.).

CatherineSue requested review from XinyueZhang369, key4ng and slin1237 as code owners February 9, 2026 20:05

github-actions Bot added ci CI/CD configuration changes tests Test changes labels Feb 9, 2026

coderabbitai Bot approved these changes Feb 9, 2026

View reviewed changes

gemini-code-assist Bot reviewed Feb 9, 2026

View reviewed changes

slin1237 approved these changes Feb 9, 2026

View reviewed changes

slin1237 merged commit 6957d4e into main Feb 9, 2026
16 of 17 checks passed

slin1237 deleted the chore/nightly-gpt-oss-20b-slug branch February 9, 2026 20:18

This was referenced Feb 10, 2026

chore(e2e): overhaul nightly benchmark summary and trim model list #392

Merged

fix(e2e): migrate genai-bench to Docker and fix router pipe hang #403

Merged

ppraneth pushed a commit that referenced this pull request Feb 18, 2026

chore(e2e): include model size in gpt-oss nightly benchmark slug (#384)

2ba184e

Signed-off-by: ppraneth <pranethparuchuri@gmail.com>

This was referenced Mar 27, 2026

Add GLM-4.6 nightly benchmark coverage #954

Closed

feat(ci): add Devstral 2 nightly benchmark #1022

Closed

coderabbitai Bot mentioned this pull request Sep 21, 2026

fix(multimodal): let the engine decide if it can take media references #2604

Closed

Navigation Menu

Repository navigation

chore(e2e): include model size in gpt-oss nightly benchmark slug - #384

chore(e2e): include model size in gpt-oss nightly benchmark slug#384
slin1237 merged 1 commit into
mainfrom
chore/nightly-gpt-oss-20b-slug

CatherineSue commented Feb 9, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

gemini-code-assist Bot commented Feb 9, 2026

Uh oh!

coderabbitai Bot commented Feb 9, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Poem

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Feb 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	("openai/gpt-oss-20b", "GptOss", 4, ["http", "grpc"], {}),
	("openai/gpt-oss-20b", "GptOss20b", 4, ["http", "grpc"], {}),

Conversation

CatherineSue commented Feb 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Problem

Solution

Changes

Test Plan

Summary by CodeRabbit

Uh oh!

gemini-code-assist Bot commented Feb 9, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

coderabbitai Bot commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Poem

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CatherineSue commented Feb 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Feb 9, 2026 •

edited

Loading