Add input-embedding quantization example

[KKothuri]

Purpose

Adds examples/quantization_embedding/ — an example showing how to quantize a model's input embedding table to weight-only intN (WNA16) with a data-free QuantizationModifier. vLLM loads these checkpoints (embedding-quant support added in vllm-project/vllm#44340) and runs a fused gather + dequant over the looked-up rows, so the packed table is never densified.

The recipe targets the Embedding module by class name (["Embedding"]), which is portable across architectures and independent of a model's module prefix. This matters because name-based targets (e.g. re:.*embed_tokens$) require the model to forward prefix to its VocabParallelEmbedding, which not all vLLM models do (see vllm-project/vllm#45535).

Embedding quantization is weight-only and data-free (no calibration set), near-lossless, and most useful for large-vocabulary models where the embedding table is a meaningful fraction of memory.

Changes

• examples/quantization_embedding/llama3_example.py — data-free embedding quant (int4, group size 64), sample generation, compressed save.
• examples/quantization_embedding/README.md — walkthrough, channel / 8-bit variants, how to compose with linear-weight quantization, and an accuracy table.

Testing

Ran the example flow end-to-end (load → oneshot → dispatch_model → generate → save); exits clean. Accuracy (lm-eval) on pythia-1.4b shows embedding quantization is near-lossless:

scheme	wikitext ppl	arc_easy acc
baseline (fp16)	14.733	0.6048
embedding W8 channel	14.732	0.6052
embedding W4 group-64	14.752	0.6061

The example references meta-llama/Meta-Llama-3-8B-Instruct per the repo convention (each scheme folder has a llama3 example); the identical recipe was validated on pythia-1.4b and Mistral-7B-v0.1, and the resulting checkpoints load and generate in vLLM.

---

This change was developed with AI assistance (Claude Code). All changed lines were reviewed by the submitter.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8a1fd738-9b7b-4fd9-9b2b-afe50d20d563

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Walkthrough

This PR adds a new embedding quantization example directory with a runnable Llama3 8B Instruct script and comprehensive documentation. The example demonstrates data-free int4 weight quantization of input embeddings using QuantizationModifier and includes sample accuracy results and composition patterns.

Changes

Embedding Quantization Example

Layer / File(s)	Summary
Llama3 embedding quantization runnable example examples/quantization_embedding/llama3_example.py	Loads Llama3 8B Instruct model and tokenizer, defines a QuantizationModifier recipe to quantize Embedding weights to int4 (grouped, symmetric, group size 64), applies quantization via oneshot, generates sample text from "Hello my name is", and saves the quantized model and tokenizer to disk with a descriptor in the output directory name.
Embedding quantization documentation examples/quantization_embedding/README.md	Documentation page covers installation, quickstart, code walkthrough (model loading, data-free quantization with strategy notes, checkpoint saving), composition with linear quantization, and a Pythia 1.4B accuracy comparison table.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

enhancement, llama, w4a16

Suggested reviewers

• HDCharles
• kylesayrs

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly describes the main addition of input-embedding quantization examples to the codebase, accurately reflecting the changeset.
Description check	✅ Passed	The description comprehensively explains the purpose, changes, and testing of the embedding quantization examples, closely aligned with the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Linked repositories: Your configuration references 1 linked repositories, but your current plan allows 0. Analyzed ``, skipped vllm-project/compressed-tensors.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Require two reviews

Wonderful, this rule succeeded.

PRs labelled "two-reviews" must have at least two approving reviews before merging.

• #approved-reviews-by >= 2
• #changes-requested-reviews-by = 0

[gemini-code-assist]

Code Review

This pull request introduces a new example and documentation for quantizing a model's input embedding table using llm-compressor. The feedback suggests adding device_map="auto" when loading the model in both the example script and the README to prevent potential CPU Out-Of-Memory (OOM) issues. Additionally, it is recommended to replace the invalid Python syntax in the README's composition example with a valid placeholder to ensure copy-pasted code remains functional.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/quantization_embedding/README.md`:
- Around line 24-26: Update the quickstart command in the README so it runs from
the repository root: replace the current line "python3 llama3_example.py" with
either an explicit path "python3
examples/quantization_embedding/llama3_example.py" or add a preceding "cd
examples/quantization_embedding && python3 llama3_example.py" step so the
example is runnable as documented.
- Around line 38-40: Add a short note to the README next to the model usage
(referencing model_id and the AutoModelForCausalLM.from_pretrained /
AutoTokenizer.from_pretrained calls) stating that
"meta-llama/Meta-Llama-3-8B-Instruct" is gated on Hugging Face and that users
must have account access and authenticate locally (for example via
huggingface-cli login or setting HF_TOKEN) before running the example; keep it
concise and placed immediately after the example snippet.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 913a4f30-8bec-4a4b-9d9d-f6fc59b4d271

📥 Commits

Reviewing files that changed from the base of the PR and between 6d2a090 and efd26ab.

📒 Files selected for processing (2)

• examples/quantization_embedding/README.md
• examples/quantization_embedding/llama3_example.py

[kylesayrs]

Looks great, thanks for the contribution!

[brian-dellabetta]

Thanks for the contribution @KKothuri !