feat: Implement Q-GaLore optimizer and custom embedding learning rate…#4511

Merged

danielhanchen merged 12 commits into

unslothai:mainfrom

OnePunchMonk:main

Mar 25, 2026

Commits on Mar 25, 2026

feat: Implement Q-GaLore optimizer and custom embedding learning rate in the Unsloth trainer.

OnePunchMonk
authored and
danielhanchen
committed
feat: Implement QGaLoreAdamW8bit optimizer with 8-bit states, GaLore low-rank gradient projection, and optional INT8 weight quantization, along with supporting projector and tests.

OnePunchMonk
authored and
danielhanchen
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
danielhanchen
committed
feat: Introduce Q-GaLore AdamW optimizer with low-rank quantized gradient projection and integrate into the trainer, along with dedicated tests.

OnePunchMonk
authored and
danielhanchen
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
danielhanchen
committed
feat: Implement Q-GaLore AdamW optimizer with gradient projection and quantization, including trainer integration and corresponding tests.

OnePunchMonk
authored and
danielhanchen
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
danielhanchen
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
danielhanchen
committed
Fix 3 bugs in Q-GaLore optimizer and add weight_quant forward hooks
danielhanchen
committed
Default weight_quant to False in QGaloreConfig
danielhanchen
committed
Optimize Q-GaLore projector and optimizer step performance
danielhanchen
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot]
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Implement Q-GaLore optimizer and custom embedding learning rate…#4511

feat: Implement Q-GaLore optimizer and custom embedding learning rate…#4511
danielhanchen merged 12 commits into
unslothai:mainfrom
OnePunchMonk:main

Commits on Mar 25, 2026

feat: Implement Q-GaLore optimizer and custom embedding learning rate in the Unsloth trainer.

feat: Implement QGaLoreAdamW8bit optimizer with 8-bit states, GaLore low-rank gradient projection, and optional INT8 weight quantization, along with supporting projector and tests.

[pre-commit.ci] auto fixes from pre-commit.com hooks

feat: Introduce Q-GaLore AdamW optimizer with low-rank quantized gradient projection and integrate into the trainer, along with dedicated tests.

[pre-commit.ci] auto fixes from pre-commit.com hooks

feat: Implement Q-GaLore AdamW optimizer with gradient projection and quantization, including trainer integration and corresponding tests.

[pre-commit.ci] auto fixes from pre-commit.com hooks

[pre-commit.ci] auto fixes from pre-commit.com hooks

Fix 3 bugs in Q-GaLore optimizer and add weight_quant forward hooks

Default weight_quant to False in QGaloreConfig

Optimize Q-GaLore projector and optimizer step performance

[pre-commit.ci] auto fixes from pre-commit.com hooks