docs: update benchmarking docs for new CLI flags, mocker/hitter tools, and revised result schema

[Pratham-Mishra04]

Summary

Updates the benchmarking documentation to reflect the current CLI interface and feature set of the Bifrost benchmarking tool, including two new companion tools (mocker and hitter), a revised flag schema, and new benchmark scenarios.

Changes

• Added documentation for the mocker (mock LLM provider) and hitter (load generator) companion tools shipped with the benchmarking repo
• Replaced the now-required -port flag with .env-based port configuration (BIFROST_PORT, etc.) and updated all examples accordingly
• Made -rate or -users a required mutually exclusive pair, replacing the old optional -rate with a default; added a "Rate vs. Users Mode" explanation section
• Documented new flags: -timeout, -cooldown, -model, -host, -path, -suffix, -request-type, -prompt-file, -ramp-up, -ramp-up-duration, -debug
• Removed -include-provider-in-request and -endpoint flags, which no longer exist
• Corrected the Go version prerequisite from 1.26.1+ to 1.24+
• Added a new "Concurrent Users with Ramp-Up" benchmark scenario
• Replaced the "Different Payload Sizes" advanced scenario with an "Embeddings Benchmarking" scenario covering -request-type embedding, -prompt-file, and the bundled prompt fixture files
• Updated the results JSON schema to match current output fields (drop_reasons, server_peak_memory_mb, status_code_counts, etc.)
• Updated multi-instance testing examples to use environment variable overrides instead of custom provider names
• Added new troubleshooting entries for missing -rate/-users, "No process found on port", and attack timeout errors
• Replaced the generic "Try hitting gateway requests to a Mock provider" tip with a direct link to the mocker tool

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

Review the rendered documentation to confirm:

• All CLI examples use -rate or -users instead of the removed -port flag
• The .env configuration section accurately describes port-based gateway configuration
• New flags appear in the configuration tables
• Companion tool links resolve to the correct paths in the benchmarking repo

Breaking changes

• Yes
• No

Related issues

Security considerations

None. Documentation-only change.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Pratham-Mishra04]

• docs: add conflicting credentials troubleshooting and clarify MCP auth mode credential exclusivity #4910
• docs: clarify adaptive load balancing weight calculation and cross-node coordination #4744
• docs: update benchmarking docs for new CLI flags, mocker/hitter tools, and revised result schema #4909 👈 (View in Graphite)
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Documentation
  • Updated benchmarking guide to use a root .env file (including BIFROST_PORT) for gateway port configuration, replacing -port/--endpoint-style usage.
  • Reworked Quick Start, scenario examples (basic, stress, large payload, endurance, ramp-up, comparative, and embeddings), and daily benchmark commands to emphasize -provider selection.
  • Added/updated guidance for real OpenAI runs, cost-free execution via the repo’s mocker, revised result/memory metrics wording, and expanded troubleshooting (including -rate vs -users constraints).

Walkthrough

Documentation for benchmark runs is updated to use a root .env file for gateway ports instead of -port and related endpoint flags. The guide also revises prerequisites, examples, result fields, advanced scenarios, automation commands, and troubleshooting text.

Changes

Benchmarking documentation update

Layer / File(s)	Summary
Prerequisites and intro tooling updates docs/benchmarking/run-your-own-benchmarks.mdx	Adds new testable items, introduces mocker/hitter companion tools, and updates Go version requirement to 1.24+ with provider configuration guidance.
Quick Start and configuration options rewrite docs/benchmarking/run-your-own-benchmarks.mdx	Rewrites Quick Start to use .env for ports, updates commands to -provider/-rate/-duration, and revises Configuration Options tables including flag mutual exclusivity.
Benchmark scenario examples and results schema docs/benchmarking/run-your-own-benchmarks.mdx	Updates basic, stress, large payload, endurance, ramp-up, and comparative benchmark examples to remove -port, and revises the results JSON schema and memory/drop-reason descriptions.
Advanced scenarios and automation script updates docs/benchmarking/run-your-own-benchmarks.mdx	Updates burst/multi-instance and embeddings examples to use BIFROST_PORT/.env, and updates the daily automation script to remove -port arguments.
Troubleshooting guidance updates docs/benchmarking/run-your-own-benchmarks.mdx	Updates flag requirement messaging and rewrites troubleshooting for .env port configuration, memory stat skipping, and mock provider recommendation.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the docs update, CLI flag changes, companion tools, and result schema revisions.
Description check	✅ Passed	The description covers the template’s required sections, including summary, changes, type, affected areas, testing, breaking changes, and security.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 07-04-docs_benchmarking_docs_update

---

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Confidence Score: 3/5

Two code examples in the doc will consistently fail when copied: the ramp-up scenario and two commands in the continuous-benchmarking script use durations that exceed the default 300-second timeout, triggering the tool's own "Attack timed out" error. Fix those examples before merging to avoid broken documentation for users.

The ramp-up example (duration=600, default timeout=300) and two continuous-benchmarking commands (durations of 600 s and 300 s with no explicit timeout) will be killed early by the tool's own attack timeout, which the doc itself says must exceed duration plus backend latency. These are concrete, reproducible failures for any user who follows the examples as written.

docs/benchmarking/run-your-own-benchmarks.mdx — the ramp-up scenario example (line 167) and the continuous-benchmarking script (lines 328–330) need explicit -timeout flags that exceed their respective -duration values.

Important Files Changed

Filename	Overview
docs/benchmarking/run-your-own-benchmarks.mdx	Documentation updated with new CLI flags, companion tools, and revised result schema. Two code examples (ramp-up scenario and continuous benchmarking script) use -duration values that exceed the default -timeout 300, causing benchmark runs to be killed early. The -timeout flag is also mislabelled as "Request timeout" when its semantics are those of a total-attack timeout.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User runs ./benchmark] --> B{-rate or -users provided?}
    B -- No --> C[Error: Either --rate or --users must be provided]
    B -- Yes: -rate --> D[Fixed RPS attack via vegeta]
    B -- Yes: -users --> E[Fixed concurrency — N requests in flight]
    E --> F{-ramp-up flag?}
    F -- Yes --> G[Ramp from 1 to N users over -ramp-up-duration seconds]
    F -- No --> H[Immediately maintain N concurrent users]
    G --> I[Hold at N users for remainder of -duration]
    H --> I
    D --> I
    I --> J{timeout exceeded?}
    J -- Yes --> K[Attack timed out error]
    J -- No --> L[Write results JSON keyed by provider]
    L --> M{-provider omitted?}
    M -- Yes --> N[Loop through all providers sequentially with -cooldown between each]
    M -- No --> O[Single provider result]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[User runs ./benchmark] --> B{-rate or -users provided?}
    B -- No --> C[Error: Either --rate or --users must be provided]
    B -- Yes: -rate --> D[Fixed RPS attack via vegeta]
    B -- Yes: -users --> E[Fixed concurrency — N requests in flight]
    E --> F{-ramp-up flag?}
    F -- Yes --> G[Ramp from 1 to N users over -ramp-up-duration seconds]
    F -- No --> H[Immediately maintain N concurrent users]
    G --> I[Hold at N users for remainder of -duration]
    H --> I
    D --> I
    I --> J{timeout exceeded?}
    J -- Yes --> K[Attack timed out error]
    J -- No --> L[Write results JSON keyed by provider]
    L --> M{-provider omitted?}
    M -- Yes --> N[Loop through all providers sequentially with -cooldown between each]
    M -- No --> O[Single provider result]

Loading

_{Reviews (2): Last reviewed commit: "docs: benchmarking docs update" | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/benchmarking/run-your-own-benchmarks.mdx`:
- Around line 357-359: The port check troubleshooting snippet is hardcoded to
8080 and no longer matches the configurable BIFROST_PORT setup. Update the
benchmark verification example in the relevant markdown section to reference the
configured port value instead of a fixed port, so it stays aligned with the
documented environment-based configuration.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 943ee1e3-c5fa-4bcf-9bc6-f80e3f54d90e

📥 Commits

Reviewing files that changed from the base of the PR and between 4462c49 and 520e833.

📒 Files selected for processing (1)

• docs/benchmarking/run-your-own-benchmarks.mdx

[akshaydeo]

z

[akshaydeo]

Merge activity

• Jul 4, 4:52 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jul 4, 4:53 PM UTC: @akshaydeo merged this pull request with Graphite.