[Doc][Misc] Fix msprobe_guide.md documentation issues #6965
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -55,21 +55,20 @@ To restrict the operators that are captured, configure the `list` block: | |||||
|
|
||||||
| ```json | ||||||
| "scope": ["Module.conv1.Conv2d.forward.0", "Module.fc2.Linear.forward.0"] | ||||||
| "scope": ["Cell.conv1.Conv2d.forward.0", "Cell.fc2.Dense.forward.0"] | ||||||
| "scope": ["Tensor.add.0.forward", "Functional.square.2.forward"] | ||||||
| ``` | ||||||
|
|
||||||
| The `level` setting determines what can be provided—modules when `level=L0`, APIs when `level=L1`, and either modules or APIs when `level=mix`. | ||||||
|
|
||||||
| - `list` (list[str]): Custom operator list. Options include: | ||||||
| - Supply the full names of specific APIs in PyTorch pynative scenarios to only dump those APIs. Example: `"list": ["Tensor.permute.1.forward", "Tensor.transpose.2.forward", "Torch.relu.3.forward"]`. | ||||||
| - When `level=mix`, you can provide module names so that the dump expands to everything produced while the module is running. Example: `"list": ["Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0"]`. | ||||||
| - Provide a substring such as `"list": ["relu"]` to dump every API whose name contains the substring. When `level=mix`, modules whose names contain the substring are also expanded. | ||||||
|
|
||||||
| Example configuration: | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "task": "statistics", | ||||||
| "dump_path": "/home/data_dump", | ||||||
|
|
@@ -86,10 +85,9 @@ cat <<'JSON' > /data/msprobe_config.json | |||||
| "summary_mode": "statistics" | ||||||
| } | ||||||
| } | ||||||
| ``` | ||||||
|
Comment on lines
69
to
88
There was a problem hiding this comment. After removing the heredoc example, the guide no longer shows what filename/path to save this JSON config to, but the later |
||||||
|
|
||||||
| ## 3. Enable `msprobe` in vllm-ascend | ||||||
|
|
||||||
| 1. Start vLLM in eager mode by adding `--enforce-eager` (static-graph scenarios are not supported yet) and pass the config path through `--additional-config`: | ||||||
|
|
||||||
|
|
@@ -102,7 +100,7 @@ JSON | |||||
| --additional-config '{"dump_config_path": "/data/msprobe_config.json"}' & | ||||||
| ``` | ||||||
|
|
||||||
| ## 4. Send requests and collect dumps | ||||||
|
|
||||||
| 1. Send inference requests as usual, for example: | ||||||
|
|
||||||
|
|
@@ -117,7 +115,7 @@ JSON | |||||
| }' | python -m json.tool | ||||||
| ``` | ||||||
|
|
||||||
| 2. Each request drives the sequence `msprobe: start -> forward -> stop -> step`. The runner invokes `step()` on every code path, so you always get a complete dataset even if inference returns early. | ||||||
|
|
||||||
| 3. Dump files are written into `dump_path`. They usually contain: | ||||||
| - Tensor files grouped by operator/module. | ||||||
|
|
@@ -131,12 +129,10 @@ JSON | |||||
| │ ├── step0 | ||||||
| │ │ ├── rank0 | ||||||
| │ │ │ ├── dump_tensor_data | ||||||
| │ │ │ │ ├── Tensor.permute.1.forward.pt # Format: {api_type}.{api_name}.{call_count}.forward.{input/output}.{arg_index}. | ||||||
|
There was a problem hiding this comment. The example filename
Suggested change
|
||||||
| │ │ │ │ │ # arg_index is the nth input or output of the API. If an input is a list, keep numbering with decimals (e.g., 1.1 is the first element of the first argument). | ||||||
| │ │ │ │ ├── Module.conv1.Conv2d.forward.0.input.0.pt # Format: {Module}.{module_name}.{class_name}.forward.{call_count}.{input/output}.{arg_index}. | ||||||
| │ │ │ │ └── Module.conv1.Conv2d.forward.0.parameters.bias.pt # Module parameter data: {Module}.{module_name}.{class_name}.forward.{call_count}.parameters.{parameter_name}. | ||||||
| │ │ │ │ # When the `model` argument passed to dump is a List[torch.nn.Module] or Tuple[torch.nn.Module], module-level data names also include the index inside the list ({Module}.{index}.*), e.g., Module.0.conv1.Conv2d.forward.0.input.0.pt. | ||||||
| │ │ │ ├── dump.json | ||||||
| │ │ │ ├── stack.json | ||||||
|
|
@@ -159,20 +155,20 @@ JSON | |||||
|
|
||||||
| - `rank`: Device ID. Each card writes its data to the corresponding `rank{ID}` directory. In non-distributed scenarios the directory is simply named `rank`. | ||||||
| - `dump_tensor_data`: Tensor payloads that were collected. | ||||||
| - `dump.json`: Statistics for the forward data of each API or module, including names, dtype, shape, max, min, mean, L2 norm (square root of the L2 variance), and CRC-32 when `summary_mode="md5"`. See [dump.json file description](#dumpjson-file-description) for details. | ||||||
| - `dump_error_info.log`: Present only when the dump tool encountered an error and records the failure log. | ||||||
| - `stack.json`: Call stacks for APIs/modules. | ||||||
| - `construct.json`: Hierarchical structure description. Empty when `level=L1`. | ||||||
|
|
||||||
| ## 5. Analyze the results | ||||||
|
|
||||||
| ### 5.1 Prerequisites | ||||||
|
|
||||||
| You typically need two dump datasets: one from the "problem side" (the run that exposes the accuracy or numerical error) and another from the "benchmark side" (a good baseline). These datasets do not have to be identical—they can come from different branches, framework versions, or even alternative implementations (operator substitutions, different graph-optimization switches, etc.). As long as they use the same or similar inputs, hardware topology, and sampling points (step/token), `msprobe` can compare them and locate the divergent nodes. If you cannot find a perfectly clean benchmark, start by capturing the problem-side data, craft the smallest reproducible case by hand, and perform a self-comparison. Below we assume the problem dump is `problem_dump` and the benchmark dump is `bench_dump`. | ||||||
|
|
||||||
| ### 5.2 Visualization | ||||||
|
|
||||||
| Use `msprobe -f pytorch graph` to generate results that can be opened inside `tb_graph_ascend`. | ||||||
|
|
||||||
| 1. Ensure the dump contains `construct.json` (i.e., `level = L0` or `level = mix`). | ||||||
| 2. Prepare a comparison file such as `compare.json`. Its format and generation flow are described in section 3.1.3 of `msprobe_visualization.md`. Example (minimal runnable snippet): | ||||||
|
|
@@ -185,7 +181,7 @@ Use `msprobe graph_visualize` to generate results that can be opened inside `tb_ | |||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| Replace the paths with your dump directories before invoking `msprobe -f pytorch graph`. **If you only need to build a single graph**, omit `bench_path` to visualize one dump. | ||||||
| Multi-rank scenarios (single rank, multi-rank, or multi-step multi-rank) are also supported. `npu_path` or `bench_path` must contain folders named `rank+number`, and every rank folder must contain a non-empty `construct.json` together with `dump.json` and `stack.json`. If any `construct.json` is empty, verify that the dump level includes `L0` or `mix`. When comparing graphs, both `npu_path` and `bench_path` must contain the same set of rank folders so they can be paired one-to-one. | ||||||
|
|
||||||
| ```shell | ||||||
|
|
@@ -195,7 +191,7 @@ Use `msprobe graph_visualize` to generate results that can be opened inside `tb_ | |||||
| | | | ├── Tensor.permute.1.forward.pt | ||||||
| | | | ├── MyModule.0.forward.input.pt | ||||||
| | | | ... | ||||||
| | | | └── Functional.linear.5.forward.output.pt | ||||||
| | | ├── dump.json # Tensor metadata | ||||||
| | | ├── stack.json # Operator call stack information | ||||||
| | | └── construct.json # Hierarchical structure; empty when `level=L1` | ||||||
|
|
@@ -213,7 +209,7 @@ Use `msprobe graph_visualize` to generate results that can be opened inside `tb_ | |||||
| 3. Run: | ||||||
|
|
||||||
| ```bash | ||||||
| msprobe -f pytorch graph \ | ||||||
| --input_path ./compare.json \ | ||||||
| --output_path ./graph_output | ||||||
| ``` | ||||||
|
|
@@ -236,7 +232,7 @@ Use `msprobe graph_visualize` to generate results that can be opened inside `tb_ | |||||
| - Search/filter: Use the search box to filter nodes by operator name, etc. | ||||||
| - Manual mapping: Automatic mapping cannot cover every case, so the tool lets you manually map nodes between the problem and benchmark graphs before generating comparison results. | ||||||
|
|
||||||
| ## 6. Troubleshooting | ||||||
|
|
||||||
| - `RuntimeError: Please enforce eager mode`: Restart vLLM and add the `--enforce-eager` flag. | ||||||
| - No dump files: Confirm that the JSON path is correct and every node has write permission. In distributed scenarios set `keep_all_ranks` so that every rank writes its own dump. | ||||||
|
|
@@ -250,15 +246,13 @@ Use `msprobe graph_visualize` to generate results that can be opened inside `tb_ | |||||
|
|
||||||
| #### L0 level | ||||||
|
|
||||||
| An L0 `dump.json` contains forward I/O for modules together with parameters. Using PyTorch's `Conv2d` as an example, the network code looks like: | ||||||
|
|
||||||
| `output = self.conv2(input) # self.conv2 = torch.nn.Conv2d(64, 128, 5, padding=2, bias=True)` | ||||||
|
|
||||||
| `dump.json` contains the following entries: | ||||||
|
|
||||||
| - `Module.conv2.Conv2d.forward.0`: Forward data of the module. `input_args` represents positional inputs, `input_kwargs` represents keyword inputs, `output` stores forward outputs, and `parameters` stores weights/biases. | ||||||
|
|
||||||
| **Note**: When the `model` parameter passed to the dump API is `List[torch.nn.Module]` or `Tuple[torch.nn.Module]`, module-level names include the index inside the list (`{Module}.{index}.*`). Example: `Module.0.conv1.Conv2d.forward.0`. | ||||||
|
|
||||||
|
|
@@ -339,89 +333,16 @@ An L0 `dump.json` contains forward/backward I/O for modules together with parame | |||||
| } | ||||||
| } | ||||||
| }, | ||||||
|
Contributor
There was a problem hiding this comment. This JSON example is invalid due to a trailing comma. Since the
Suggested change
|
||||||
| } | ||||||
| } | ||||||
| } | ||||||
|
Comment on lines
334
to
338
There was a problem hiding this comment. The L0 |
||||||
| ``` | ||||||
|
|
||||||
| #### L1 level | ||||||
|
|
||||||
| An L1 `dump.json` records forward I/O for APIs. Using PyTorch's `relu` function as an example (`output = torch.nn.functional.relu(input)`), the file contains: | ||||||
|
|
||||||
| - `Functional.relu.0.forward`: Forward data of the API. `input_args` are positional inputs, `input_kwargs` are keyword inputs, and `output` stores the forward outputs. | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
|
|
@@ -469,43 +390,6 @@ An L1 `dump.json` records forward/backward I/O for APIs. Using PyTorch's `relu` | |||||
| } | ||||||
| ] | ||||||
| }, | ||||||
|
Contributor
There was a problem hiding this comment. This JSON example is invalid due to a trailing comma. Since the
Suggested change
|
||||||
| } | ||||||
| } | ||||||
| } | ||||||
|
Comment on lines
391
to
395
There was a problem hiding this comment. The L1 |
||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
scopeexamples are inside ajsoncode block but repeat the"scope"key three times, which is not valid JSON and may confuse readers trying to copy/paste. Consider turning these into separate code blocks (each showing a full snippet) or a single valid example with explanatory text outside the JSON block.