Commit 72784b2
committed
[Pass] Attach memory-planning attributes for dynamic func output
This PR adds a pass into the model compilation pipeline, which
attach an attribute `"relax.memory_plan_dynamic_func_output"`
for each Relax function in the IRModule. This attribute suggests
that the Relax functions' output tensors, though having dynamic
shapes, are statically plannable.
This enhancement makes sure that in serving scenarios, our
memory allcoation is completely static after stablized. So we
will not be worried about continuing memory usage growth, and
can allocate more memory for KV cache.
This PR can be early merged, but it will not take effects until
apache/tvm#16111 is merged.1 parent b58d32d commit 72784b2
1 file changed
+1
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
127 | 127 | | |
128 | 128 | | |
129 | 129 | | |
| 130 | + | |
130 | 131 | | |
131 | 132 | | |
132 | 133 | | |
| |||
0 commit comments