[Bugfix] SparseGPT, Pipelines

[kylesayrs]

Purpose

• SparseGPT
  • Fix behavior where targets specifies which modules to sparsity, not which layers to target
  • Fix broken behavior with _infer_owl_layer_sparsity and add test
  • Fix owl argument validation
  • Add type hints and abstract methods for clarity
• Pipelines
  • Fix bug revealed by decorators added to the llama model definition in the latest transformers release
    • [Backend support] Allow num_logits_to_keep as Tensor and change it to logits_to_keep + add flag huggingface/transformers#35757
    • For the sequential pipeline, this revealed a bug in torch.fx._symbolic_trace where wrapped functions were not being handled properly
    • Future work could involve upstreaming a bug fix
  • Fix issue caused by changes to llama model definition
    • 🧹 Remove deprecated RotaryEmbedding parts in the Attention layers huggingface/transformers#34858
    • For the layer sequential pipeline, this challenges the assumption that each layer input is the previous layer's output (which was known to be a fragile assumption)
  • Fix issue related to basic pipeline slowdowns and inaccuracy

Changes

• SparseGPT
  • Fully separate targets and sequential_targets
    • Modify hooks adding logic to reflect this change
  • Fix behavior of _infer_owl_layer_sparsity and add test
  • Code clarity
    • Add additional type hints
    • Designate calibrate_module as an abstract method on the sgpt mixin
• Pipelines
  • Sequential pipeline: unwrap model forward function to avoid issues with pytorch function patching
  • Layer Sequential Pipeline: Add maybe_inject_pos_embeddings to sequential pipeline to hackily support models with position_embeddings
  • Basic Pipeline: Fix on_sequential_batch_end to call on the end of epoch, rather than every batch
    • Calling every batch was likely causing slowdowns

Followups

• Remove deprecated sequential_update option from examples and tests

Testing

• Added tests/llmcompressor/transformers/obcq/test_obcq_owl.py
• Tested OBCQ+llama with sequential, layer sequential, and basic pipelines independently

Regression Evaluations

Models were compressed using examples/sparse_2of4_quantization_fp8/llama3_8b_2of4.py without fp8 option

sparsegpt

Main

vllm (pretrained=/home/kyle/llm-compressor/Meta-Llama-3-8B-InstructSparseGPTModifierMAIN,dtype=bfloat16,add_bos_token=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.6243|±  |0.0136|

This branch

vllm (pretrained=/home/kyle/llm-compressor/Meta-Llama-3-8B-InstructSparseGPTModifierFEATURE,dtype=bfloat16,add_bos_token=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.6306|±  |0.0136|

To test wanda, the SparseGPTModifier was replaced with the WandaPruningModifier

wanda

Main

vllm (pretrained=/home/kyle/llm-compressor/Meta-Llama-3-8B-InstructWandaPruningModifierMAIN,dtype=bfloat16,add_bos_token=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.5912|±  |0.0138|

This branch

vllm (pretrained=/home/kyle/llm-compressor/Meta-Llama-3-8B-InstructWandaPruningModifierFEATURE,dtype=bfloat16,add_bos_token=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande|      1|none  |     5|acc   |↑  |0.5817|±  |0.0139|

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[dsikka]

I don't agree with the notion that adding the same variables through inheritance helps with code clarity or user readability.

Updating the docstring in the inherited classes would be sufficient rather than relying on redundant code.

[rahul-tuli]

LGTM! (pending tests)