Add simple utility to dump tensors for debugging

[fzyzcjy]

Motivation

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[gemini-code-assist]

Hello @fzyzcjy, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello! Gemini here, providing a summary of this pull request by @fzyzcjy.

This PR introduces a new utility module, debug_utils.py, specifically designed to help with debugging by providing a simple mechanism to dump tensor values and other data during execution. The primary goal is to make it easier to inspect intermediate values within the SGLang runtime, which can be particularly useful for verifying model behavior or tracking down issues.

The utility is controlled via environment variables, allowing users to enable/disable dumping, specify an output directory, and choose whether to save the dumped values to files or just print summary information.

Highlights

• New Debugging Utility: Adds a new module python/sglang/srt/debug_utils.py containing a _Dumper class and a global dumper instance for easy access.
• Tensor Dumping: Provides a dumper.dump(name, value, **kwargs) method to dump arbitrary values, with special handling for torch.Tensor.
• Configurable Behavior: The dumper's behavior is controlled by environment variables: SGLANG_DUMPER_ENABLE (default 'true'), SGLANG_DUMPER_DIR (default '/tmp'), and SGLANG_DUMPER_WRITE_FILE (default '1').
• Output and File Saving: When enabled, it prints information about the dumped value (type, shape, dtype, sample value) and optionally saves the value to a .pt file in a structured directory path (<base_dir>/sglang_dump_<timestamp>_<rank>/<kwargs_filename>.pt).
• Sample Value for Large Tensors: For large tensors (more than 200 elements), it prints and potentially saves only a slice (first 5 elements along dimensions larger than 200) to avoid excessive output/file size.
• Distributed Support: Includes the tensor model parallel rank in the output directory structure and print statements.

Changelog

• python/sglang/srt/debug_utils.py
  • Added a new file debug_utils.py.
  • Implemented the _Dumper class with methods __init__, dump, and _get_sample_value.
  • Added environment variable parsing for SGLANG_DUMPER_ENABLE, SGLANG_DUMPER_DIR, and SGLANG_DUMPER_WRITE_FILE.
  • Implemented logic to print value information and save torch.Tensor values to files using torch.save.
  • Added logic to get and include the tensor model parallel rank in output paths and prints.
  • Implemented sampling logic for large tensors in _get_sample_value.
  • Instantiated the _Dumper class as a global dumper object.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Tensors flow and twist,
Dump them out, see what's missed,
Debug finds the way.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a _Dumper utility, a helpful tool for debugging by saving tensors and printing their metadata. The implementation is clear and uses environment variables for configuration, which offers good flexibility. The code is modern, utilizing pathlib and f-strings.

While the utility is well-structured, there are a few areas for improvement concerning default behavior, filename robustness, and console output verbosity. Additionally, as indicated in the pull request checklist, adding unit tests for this new functionality would be beneficial for long-term maintainability.

Summary of Findings

• Default Dumper State: The dumper is enabled by default (SGLANG_DUMPER_ENABLE defaults to true). It's generally safer for debugging tools with I/O impact to be opt-in (default to false).
• Filename Robustness: Filenames are generated from kwargs values. If these values are very long or contain special characters, it could lead to invalid or problematic filenames. Consider sanitizing, truncating, or hashing these values for more robust filename generation.
• Console Output Verbosity: Printing the sample_value (which can be a tensor) directly to the console can lead to very verbose output. It might be better to print a summary (e.g., shape, basic statistics) instead of the full tensor representation in the log message.
• Unit Testing: The pull request checklist mentions adding unit tests. For this new utility, unit tests would be valuable to verify its core functionality (e.g., file creation, naming conventions, sampling logic) and to prevent regressions. Please consider adding these.
• Minor: Unused Parameter (Not Commented): The name parameter in the _get_sample_value method is unused. This was not added as a direct review comment due to its low severity and the current review settings.
• Minor: Magic Numbers (Not Commented): The _get_sample_value method uses magic numbers (e.g., 200 for numel/dim_size checks, 5 for slice size). Consider defining these as named constants for better readability and maintainability. This was not added as a direct review comment due to its low severity and the current review settings.

Merge Readiness

This pull request introduces a useful tensor dumping utility with a generally sound implementation. However, there are a few medium-severity concerns regarding the default enabled state, filename robustness from keyword arguments, and potential console output verbosity that would be good to address. Additionally, incorporating unit tests, as suggested by the PR checklist, would greatly enhance the utility's reliability and maintainability.

Given these points, I recommend that these changes be made before merging. As I am not authorized to approve pull requests, please ensure further review and approval from the maintainers once the feedback has been incorporated.

[gemini-code-assist]

The SGLANG_DUMPER_ENABLE environment variable defaults to "true", which means the tensor dumping functionality is active by default if this module is used. Debugging utilities, especially those involving I/O operations like saving tensors to disk, can introduce performance overhead and consume disk space.

Is it intentional for this dumper to be enabled by default? It's often a safer practice for such debugging tools to be opt-in (i.e., default to disabled) to prevent accidental resource consumption or performance degradation in environments where active dumping isn't explicitly needed.

Suggested change

	self._enable = get_bool_env_var("SGLANG_DUMPER_ENABLE", "true")
	self._enable = get_bool_env_var("SGLANG_DUMPER_ENABLE", "false")

[gemini-code-assist]

The full_filename is constructed by joining key-value pairs from full_kwargs, where values v are directly converted to strings (e.g., f"{k}={v}"). If a value v has a very long string representation or contains characters that are problematic for filenames (e.g., /, \, :, *, ?, \", <, >, |), this could lead to errors during file creation or result in filenames that are difficult to manage or incompatible with certain operating systems or path length limits.

To enhance robustness, have you considered any of these approaches?

• Sanitizing str(v): Replace or remove problematic characters.
• Truncating str(v): Limit the length if it exceeds a reasonable threshold.
• Hashing str(v): Use a hash if values are expected to be complex or very long, ensuring a fixed-length, safe component.
• Documenting expectations: Clearly state the expected types or formats for kwargs values to ensure they are filename-friendly.

[gemini-code-assist]

The print statement includes f"sample_value={sample_value}". When sample_value is a tensor, even a small one obtained after sampling, its default string representation can be quite verbose and span multiple lines in the console output (e.g., a 5x5 tensor already takes several lines).

Could this verbosity potentially clutter the console, making it harder to read other logs, especially if dump() is called frequently?

Some alternatives to consider for the console output:

• Print only metadata like shape, dtype, and device of sample_value if it's a tensor.
• Provide a compact summary of the tensor's content (e.g., min, max, mean values if it's a numerical tensor).
• Make the verbosity of this specific part of the log message configurable, perhaps via another environment variable.

[zhyncs]

only for debug usage