Fix github io pages

docs/source/conf.py

@@ -22,7 +22,7 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
 templates_path = ['_templates']
-exclude_patterns = []
+exclude_patterns = ['performance/performance-tuning-guide/introduction.md']
 
 extensions = [
     'sphinx.ext.duration',

docs/source/index.rst

@@ -116,7 +116,7 @@ Welcome to TensorRT-LLM's Documentation!
 
    performance/perf-overview.md
    Benchmarking <performance/perf-benchmarking.md>
-   Best Practices <performance/introduction.md>
+   performance/performance-tuning-guide/index
    performance/perf-analysis.md
 
 

docs/source/performance/benchmarking-default-performance.md → docs/source/performance/performance-tuning-guide/benchmarking-default-performance.md

@@ -88,7 +88,7 @@ The README in the examples folder for supported models walks through building en
 
 `python benchmarks/cpp/prepare_dataset.py --stdout --tokenizer /path/to/hf/Llama-3.3-70B-Instruct/ token-norm-dist --input-mean 2048 --output-mean 2048 --input-stdev 0 --output-stdev 0 --num-requests 1000 > synthetic_2048_2048.txt`
 
-`trtllm-bench` can also take in real data, see [`trtllm-bench` documentation](./perf-benchmarking.md) for more details on the required format.
+`trtllm-bench` can also take in real data, see [`trtllm-bench` documentation](../perf-benchmarking.md) for more details on the required format.
 
 ### Running Throughput and Latency Benchmarks
 

docs/source/performance/deciding-model-sharding-strategy.md → docs/source/performance/performance-tuning-guide/deciding-model-sharding-strategy.md

@@ -11,12 +11,12 @@ Splitting your model weights between multiple GPUs requires them to communicate
 
 In pipeline parallelism, the model is split into sets of contiguous layers and each GPU houses one of these sets. In this setup, the only required communication is for each GPU to send the outputs of its set to the GPU with the next set.
 
-![Pipeline Parallel Visualization](../media/Pipeline_Parallel_Vis.svg)
+![Pipeline Parallel Visualization](../../media/Pipeline_Parallel_Vis.svg)
 
 
  On the other hand, tensor parallelism takes each layer of the model and splits it between the GPUs. This means that every GPU houses a portion of every layer. However since each layer needs the full outputs of the previous layer as an input, each GPU has to perform the heavier All-Reduce communication operation to share its results with all other GPUs before it can begin processing the next layer. While this seems disadvantageous, because each GPU only holds partial layers, it also performs smaller matrix multiplications, allowing it to compute its outputs quicker.
 
- ![Tensor Parallel Visualization](../media/Tensor_Parallelism_Vis.svg)
+ ![Tensor Parallel Visualization](../../media/Tensor_Parallelism_Vis.svg)
 
 
  Ultimately deciding the best strategy comes down to whether the extra overhead from the All-Reduce operation overshadows the gains from the smaller matrix multiplications. If the interconnects between the GPUs are sufficiently fast, the gains from the reduced computation burden per layer can outweigh the additional communication cost. Consequently, a general rule of thumb is that if your GPUs have fast connections between them like NVLink then tensor parallel is likely a good choice. However if the communication will go over slow connections (across nodes for example) pipeline parallel is likely better. Overall we provide the following guidelines:

docs/source/performance/performance-tuning-guide/index.rst

@@ -0,0 +1,15 @@
+Performance Tuning Guide
+=======================
+
+.. include:: introduction.md
+   :parser: myst_parser.sphinx_
+
+.. toctree::
+   :maxdepth: 1
+
+   benchmarking-default-performance
+   useful-build-time-flags
+   tuning-max-batch-size-and-max-num-tokens
+   deciding-model-sharding-strategy
+   fp8-quantization
+   useful-runtime-flags

docs/source/performance/introduction.md → docs/source/performance/performance-tuning-guide/introduction.md

@@ -1,7 +1,3 @@
-(perf-best-practice)=
-
-# Introduction
-
 While defaults are expected to provide solid performance, TensorRT-LLM has several configurable options that can improve performance for your particular workload. This guide is meant to help you tune TensorRT-LLM to extract the best performance for your use case. It covers several of the most helpful tunable parameters and provides intuition for thinking about them. This guide also doubles as an example of how to work with TensorRT-LLM's LLM-API and its TRTLLM-Bench benchmarking workflow.
 
 This guide uses Llama-3.3-70b on 4 H100-sxm-80GB connected via NVLink as a case study and focuses on optimizing performance on input sequence length/output sequence length of 2048/2048. Case study sections throughout this guide reference internal performance testing and results to help reinforce the conclusions and recommendations given.
@@ -18,10 +14,4 @@ This guide expects you have some familiarity with the following concepts
  Please refer to [Mastering LLM Techniques - Inference Optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/) for an introduction to these concepts.
 
 ## Table of Contents
-
- 1. [Benchmarking Default Performance](./benchmarking-default-performance.md)
- 2. [Useful Build Time Flags](./useful-build-time-flags.md)
- 3. [Tuning Max-Num Tokens and Max-Batch Size](./tuning-max-batch-size-and-max-num-tokens.md)
- 4. [Deciding Model Sharding Strategy (Tensor Parallel, Pipeline Parallel)](./deciding-model-sharding-strategy.md)
- 5. [FP8 Quantization](./fp8-quantization.md)
- 6. [Useful Runtime Options](./useful-runtime-flags.md)
+<!--Actual table of contents with links is not written here because sphinx autogenerates it.-->

docs/source/performance/tuning-max-batch-size-and-max-num-tokens.md → docs/source/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md

@@ -10,26 +10,26 @@ One of TensorRT-LLM's key features is it's inflight batching scheduler and runti
 
 This section visualizes how TensorRT-LLM schedules requests based on max-batch size and max-num tokens. The example starts out with a newly initialized engine as well as a few unscheduled requests that have come in. For the sake of this example, toy values are set to `max batch size = 4` and `max num tokens = 12`. Each square block represents a token, and its color represents which request it belongs to.
 
-![TRT-LLM Scheduler Visualization 1](../media/TRTLLM_Scheduler_Vis_1.svg)
+![TRT-LLM Scheduler Visualization 1](../../media/TRTLLM_Scheduler_Vis_1.svg)
 
 
 Now the scheduler takes the first two requests, Request 1 and Request 2, and schedules them to execute the context phase. However, it cannot schedule any more requests because the prompts of the first two requests had 5 tokens each, leaving a budget of 2 tokens due to the max num tokens limit. Since all remaining requests have more than 2 prompt tokens none of them can be scheduled (context chunking can help in this situation, see the paged context attention section below). The tokens are marked with a "C" on them to represent that they are prompt tokens that were processed in the context phase.
 
 > Note: The tokens for different requests are shown on different rows simply for visualization purposes and are not representative of actual memory layouts
 
-![TRT-LLM Scheduler Visualization 2](../media/TRTLLM_Scheduler_Vis_2.svg)
+![TRT-LLM Scheduler Visualization 2](../../media/TRTLLM_Scheduler_Vis_2.svg)
 
 Now the engine runs an iteration of execution, completing the context phases for both of the scheduled requests. After it is done, the kv-cache of the prompts for both requests have been created and the first token has been generated. Tokens that were generated are marked with "G(n)" - for example a token marked "G1" represents that it is the first token generated for its request.
 
 TRT-LLM prioritizes scheduling requests in generation phase first so the two generated tokens are queued to be processed in the next iteration. Now, since the two previously scheduled requests have entered generation phase and only take up two tokens out of the max num token budget of 12, the scheduler is able to schedule two additional requests, Request 3 and Request 4. It cannot schedule the last request, Request 5, even though there is space for it in the max num tokens budget because of the max batch size limit of 4.
 
-![TRT-LLM Scheduler Visualization 3](../media/TRTLLM_Scheduler_Vis_3.svg)
+![TRT-LLM Scheduler Visualization 3](../../media/TRTLLM_Scheduler_Vis_3.svg)
 
 After the next iteration of execution, the second tokens for Requests 1 and 2 have been generated, and the first tokens for Request 3 and 4 have been generated. Lets say that G2 that was generated for Request 1 is the stop token, signifying that Request 1 is completed. In this case the scheduler would evict Request 1 before performing another execution iteration and prepare to return it to the user. This eviction puts the state of the engine below the max batch size limit and allows Request 5 to be scheduled.
 
 Another thing to note is that G1 that was generated for Request 2 has been added to the kv-cache for request 2, representing how kv-cache for a request grows as more and more tokens are generated.
 
-![TRT-LLM Scheduler Visualization 4](../media/TRTLLM_Scheduler_Vis_4.svg)
+![TRT-LLM Scheduler Visualization 4](../../media/TRTLLM_Scheduler_Vis_4.svg)
 
 Overall, the max batch size and max num tokens limits play a key part in deciding when requests are actually executed, and tuning them can have significant impacts on throughput numbers as well as how the engine balances previously scheduled requests in generation phase with context phase on new requests
 
@@ -101,7 +101,7 @@ For this particular workload max num tokens of 2048 provides the best performanc
 
 The [visualization](#understanding-the-trt-llm-scheduler) of the TensorRT-LLM scheduler showed that initially Request 3 couldn't be scheduled because it would put the scheduler over the max-num tokens limit. However with context chunking, this is no longer the case, and the first chunk of Request 3 would be able to be scheduled.
 
-![TRT-LLM Scheduler Visualization Chunked Context 1](../media/TRTLLM_Scheduler_Vis_Chunked_Context_1.svg)
+![TRT-LLM Scheduler Visualization Chunked Context 1](../../media/TRTLLM_Scheduler_Vis_Chunked_Context_1.svg)
 
 This is extremely beneficial for several reasons. Firstly it eliminates the possibility of requests with large prompts relative to max num tokens being unable to be scheduled due to other requests that are already in-flight. In production workloads, this can help improve worst case TTFT numbers. Secondly it allows for setting smaller values of max num tokens since you no longer need max num tokens to be at least as large as the longest prompt you want to support. For long-context cases this is extremely important, because setting extremely large values of max-num tokens takes away from memory available to be used as kv-cache. Given that in the worst case scenario chunked context has minimal impact on performance but can significantly benefit it in many scenarios, it's recommended that you always enable it.
 

docs/source/performance/useful-runtime-flags.md → docs/source/performance/performance-tuning-guide/useful-runtime-flags.md

@@ -203,7 +203,7 @@ If you wanted to set `max_tokens_in_paged_kv_cache` instead, you would replace `
 The `max_attention_window_size` flag sets the maximum number of tokens that are
 attended to in order to generate one token when using techniques like sliding window
 attention. See this
-[Document](../advanced/gpt-attention.md#sliding-window-attention-cyclic-rolling-buffer-kv-cache)
+[Document](../../advanced/gpt-attention.md#sliding-window-attention-cyclic-rolling-buffer-kv-cache)
 for more details. It defaults to the maximum sequence length
 (`max_seq_len` when building the engine), which means
 that the feature is disabled by default.