Skip to content

[Benchmark] Make image dataset seeds reproducible across processes - #31581

Closed
mickqian wants to merge 1 commit into
mainfrom
codex/image-benchmark-seeded-rng
Closed

mickqian wants to merge 1 commit into
mainfrom
codex/image-benchmark-seeded-rng

Conversation

@mickqian

@mickqian mickqian commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • sort tokenizer vocabulary IDs before seeded synthetic prompt sampling, removing process hash/dictionary-order dependence
  • reset Python and NumPy RNG state after multimodal processor initialization, which may otherwise consume global RNG state
  • add regression tests for processor RNG side effects and vocabulary-order independence

Motivation

Seeded random-image bench_serving runs were not fully reproducible across fresh client processes. With the same --seed 777 and identical sampled image-size statistics, two runs reported different token totals:

  • before: total/text/vision = 4611/48/4563 vs 4609/44/4565
  • after: 4610/47/4563 vs 4610/47/4563

This makes strict SGLang/vLLM random-image comparisons unreliable even when the nominal benchmark arguments match.

Validation

  • test_benchmark_datasets_api.py: 40 passed, 3 subtests passed
  • full pre-commit on the three changed files: passed
  • two independent live Qwen3.6-VL client processes with the same seed produced identical image statistics and text/vision/total input-token counts

This only changes benchmark workload generation; it does not touch the serving hot path or claim a runtime performance change.


CI States

Latest PR Test (Base): ❌ Run #29570655883
Latest PR Test (Extra): ❌ Run #29570655646

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures deterministic benchmark dataset generation by sorting the available tokens from the tokenizer vocabulary and resetting the random seeds (both random and numpy.random) during ImageDataset.load to prevent processor initialization from affecting the global RNG state. It also adds corresponding unit tests to verify these behaviors. There are no review comments to address, and we have no additional feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@mickqian mickqian closed this Jul 28, 2026
@Jiminator
Jiminator deleted the codex/image-benchmark-seeded-rng branch September 14, 2026 04:41
@alexnails
alexnails restored the codex/image-benchmark-seeded-rng branch September 14, 2026 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant