Fix meta tensor materialization in SequentialPipeline #1930

ronantakizawa · 2025-10-14T17:04:11Z

SUMMARY:

Problem

Sequential quantization (GPTQ/AWQ) fails on vision-language models with: RuntimeError: Tensor.item() cannot be called on meta tensors.

This occurs in SequentialPipeline because PyTorch FX tracing creates meta-tensors
during graph partitioning for offloaded modules. When the traced code executes
operations like .item(), it fails on these meta tensors.

Solution

Added meta tensor materialization at two levels:

Model-level (helpers.py): Materialize meta tensors in model parameters/buffers
before subgraph execution
Input-level (pipeline.py): Materialize meta tensors in inputs before passing
to traced code

This defense-in-depth approach ensures meta-tensors are materialized regardless of
where they originate.

Changes

src/llmcompressor/pipelines/sequential/helpers.py:
- Added _materialize_model_meta_tensors() method to Subgraph class
- Added _materialized flag to prevent duplicate materialization
- Handles integer dtype tensors by converting to buffers instead of parameters
src/llmcompressor/pipelines/sequential/pipeline.py:
- Added input materialization before both calibration and propagation passes

TEST PLAN:
Implemented unit test file tests/llmcompressor/pipelines/test_sequential_vlm.py
Successfully quantized HuggingFaceTB/SmolVLM-Instruct (idefics3 architecture):

GPTQ W4A16: 168 layers quantized, 4.4 GB → 1.97 GB (55.1% reduction)
AWQ W4A16: 168 layers quantized, 4.4 GB → 1.97 GB (55.1% reduction)

github-actions · 2025-10-14T17:04:19Z

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

gemini-code-assist · 2025-10-14T17:04:31Z

Summary of Changes

Hello @ronantakizawa, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a robust solution to a RuntimeError that occurs when PyTorch FX tracing generates meta tensors during sequential quantization of vision-language models. The core issue is that operations like .item() fail on these meta tensors. The fix employs a two-pronged approach: materializing meta tensors within the model's parameters and buffers, and also materializing meta tensors found in the input data before execution. This ensures that all necessary tensors are concrete, allowing quantization pipelines to proceed without error and enabling efficient model compression.

Highlights

Problem Resolution: Fixes RuntimeError: Tensor.item() cannot be called on meta tensors encountered during sequential quantization (GPTQ/AWQ) of vision-language models, caused by PyTorch FX tracing creating meta tensors.
Model-Level Materialization: Implemented a new method _materialize_model_meta_tensors in Subgraph to materialize meta tensors in model parameters and buffers before subgraph execution, preventing issues with offloaded modules.
Input-Level Materialization: Added logic to materialize meta tensors within input data structures (tensors, dicts, lists, tuples) before passing them to the traced code during both calibration and and propagation passes.
Integer Dtype Handling: Specifically addresses integer dtype parameters by converting them to buffers during materialization, as they cannot require gradients.
Performance Verified: The fix has been successfully tested with GPTQ and AWQ quantization on a complex vision-language model, demonstrating correct functionality and significant memory reduction.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request addresses a RuntimeError with meta tensors during sequential quantization by materializing them at both the model and input levels. The overall approach is sound and provides a robust fix. My review includes suggestions to improve the implementation by removing hardcoded device assignments in favor of dynamically determined ones, and refactoring duplicated code to enhance maintainability. These changes will make the solution more robust and easier to manage.

src/llmcompressor/pipelines/sequential/helpers.py

src/llmcompressor/pipelines/sequential/pipeline.py

kylesayrs · 2025-10-14T17:54:50Z

Hi @ronantakizawa,

Do you have a model example which uses Tensor.item?

These kind of Tensor.item calls typically shouldn't be present in model definitions anyways. I think in this scenario, it might make more sense to add a patching layer to patch the model definition to avoid the item call.

ronantakizawa · 2025-10-14T19:46:55Z

@kylesayrs here is the project I was working on that caused the meta tensor error. My patch fixed it.

https://colab.research.google.com/drive/15L5F-p01M5mBv4UDueyw8C57KxkFaqjS?usp=sharing

The .item() call doesn't occur in the model definition itself - it happens inside the traced subgraph code that PyTorch FX generates during sequential pipeline partitioning.

When SequentialPipeline partitions the model using PyTorch FX tracing, device-offloaded modules get replaced with meta tensors during symbolic execution. The traced code then executes rotary_emb(), which internally calls .item() on tensor dimensions/position indices. When those are meta tensors, it crashes.

Signed-off-by: ronantakizawa <[email protected]>

- Materialize meta tensors in model parameters/buffers before subgraph execution - Materialize meta tensors in inputs before passing to traced code - Fixes 'Tensor.item() on meta tensors' error from offloaded modules - Enables GPTQ quantization for idefics3-based models like SmolVLM Signed-off-by: ronantakizawa <[email protected]>

- Handle integer dtype parameters that can't require gradients - Convert integer parameters to buffers instead of Parameters - Prevents 'Only floating point tensors can require gradients' error Signed-off-by: ronantakizawa <[email protected]>

- Use _parameters and _buffers dicts directly to avoid conflicts - Properly remove parameter before converting to buffer - Prevents 'attribute already exists' KeyError Signed-off-by: ronantakizawa <[email protected]>

- Only materialize meta tensors once per subgraph - Add existence checks before modifying _parameters and _buffers - Add try-catch blocks with warnings for materialization failures - Prevents KeyError from duplicate buffer registration Signed-off-by: ronantakizawa <[email protected]>

Signed-off-by: ronantakizawa <[email protected]>

- Use model_device instead of hardcoded cuda:0 for multi-GPU compatibility - Define _materialize_meta_tensors once before loop to avoid duplication - Improves maintainability and correctness in multi-device environments Signed-off-by: ronantakizawa <[email protected]>

Run make style to format code according to project standards Signed-off-by: ronantakizawa <[email protected]>

brian-dellabetta · 2025-10-16T20:50:19Z

@kylesayrs does this look good to you? A user reported it resolved their issue with Qwen 3 VL and AWQ -- #1939 (comment)

I plan to validate Qwen 3 VL + AWQ use case tomorrow (along with updating the e2e tests from qwen 2.5 vl to qwen 3 vl)

brian-dellabetta

Thank you for the very impressive contribution! I tried this and it does in fact resolve the issue using Qwen 3 inside the sequential pipeline.

The code looks really good, but I have a question and a couple change requests for the tests

brian-dellabetta · 2025-10-17T20:21:23Z

src/llmcompressor/pipelines/sequential/helpers.py

+
+        for module in model.modules():
+            # Materialize parameters
+            for name, param in list(module.named_parameters(recurse=False)):


why don't we need to recurse here?

brian-dellabetta · 2025-10-17T20:24:22Z

tests/llmcompressor/pipelines/test_sequential_vlm.py

+
+
+@pytest.mark.unit
+def test_meta_tensor_materialization():


this test looks like it might have been vibe coded 😄

brian-dellabetta · 2025-10-17T20:27:13Z

tests/llmcompressor/pipelines/test_sequential_vlm.py

+
+@pytest.mark.integration
+@requires_gpu
+def test_sequential_pipeline_with_meta_tensors(tmp_path):


rather than include calls to oneshot, a better unit test here would just be to make sure the model, after materializing, has parameters than aren't meta type?

dsikka · 2025-10-17T21:31:35Z

Are you using latest transformers? @ronantakizawa

brian-dellabetta · 2025-10-17T21:31:58Z

Hi @ronantakizawa , it seems like materialized tensors might be a useful thing in general when tracing subgraphs, but I tried running a sequential pipeline after upgrading my torch/transformers versions, and it works without your PR. Can you try and see if it works on your side without your changes?

ronantakizawa closed this Oct 14, 2025

gemini-code-assist bot reviewed Oct 14, 2025

View reviewed changes

src/llmcompressor/pipelines/sequential/helpers.py Show resolved Hide resolved

src/llmcompressor/pipelines/sequential/pipeline.py Outdated Show resolved Hide resolved

src/llmcompressor/pipelines/sequential/pipeline.py Outdated Show resolved Hide resolved

ronantakizawa reopened this Oct 14, 2025

ronantakizawa force-pushed the ronantakizawa/sequentialpipeline branch 2 times, most recently from 2ee7493 to 4e69a2e Compare October 15, 2025 05:13

ronantakizawa added 8 commits October 14, 2025 23:16

fix

5be9f1a

Signed-off-by: ronantakizawa <[email protected]>

Fix integer dtype parameter materialization

e993200

- Handle integer dtype parameters that can't require gradients - Convert integer parameters to buffers instead of Parameters - Prevents 'Only floating point tensors can require gradients' error Signed-off-by: ronantakizawa <[email protected]>

Fix duplicate attribute error during materialization

7be23fe

- Use _parameters and _buffers dicts directly to avoid conflicts - Properly remove parameter before converting to buffer - Prevents 'attribute already exists' KeyError Signed-off-by: ronantakizawa <[email protected]>

Add test for sequential pipeline meta tensor fix

e249c18

Signed-off-by: ronantakizawa <[email protected]>

Apply code formatting to pipeline.py

ee7c416

Run make style to format code according to project standards Signed-off-by: ronantakizawa <[email protected]>

ronantakizawa force-pushed the ronantakizawa/sequentialpipeline branch from 4e69a2e to ee7c416 Compare October 15, 2025 05:17

JartX mentioned this pull request Oct 16, 2025

HOW To: Quantization: Qwen/Qwen3-VL-30B-A3B-Instruct AWQ #1939

Open

brian-dellabetta requested a review from kylesayrs October 16, 2025 20:50

brian-dellabetta requested changes Oct 17, 2025

View reviewed changes

brian-dellabetta mentioned this pull request Oct 17, 2025

[Bug]: Qwen3-VL-4B 'Tensor' object has no attribute 'get'. Did you mean: 'det'? #1941

Open

brian-dellabetta requested a review from dsikka October 17, 2025 20:34



		@pytest.mark.unit
		def test_meta_tensor_materialization():

Fix meta tensor materialization in SequentialPipeline #1930

Are you sure you want to change the base?

Fix meta tensor materialization in SequentialPipeline #1930

Conversation

ronantakizawa commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Changes

Uh oh!

github-actions bot commented Oct 14, 2025

Uh oh!

gemini-code-assist bot commented Oct 14, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kylesayrs commented Oct 14, 2025

Uh oh!

ronantakizawa commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brian-dellabetta commented Oct 16, 2025

Uh oh!

brian-dellabetta left a comment

Choose a reason for hiding this comment

Uh oh!

brian-dellabetta Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

brian-dellabetta Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

brian-dellabetta Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

dsikka commented Oct 17, 2025

Uh oh!

brian-dellabetta commented Oct 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ronantakizawa commented Oct 14, 2025 •

edited

Loading

ronantakizawa commented Oct 14, 2025 •

edited

Loading