[Liger] liger DPO support by kashif · Pull Request #2568 · huggingface/trl

kashif · 2025-01-14T13:14:28Z

What does this PR do?

Add support for Liger-kernel losses for the DPO Kernel

Peft support: #3065

HuggingFaceDocBuilderDev · 2025-01-15T11:15:17Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

tests/test_dpo_trainer.py

trl/trainer/dpo_trainer.py

qgallouedec · 2025-01-17T16:50:25Z

liger loss isn't compatible with ref precomputing right? If so we could add a warning or an error.

docs/source/reducing_memory_usage.md

VProv · 2025-03-26T16:27:33Z

This PR needs to use _FSDPForwardRedirection or another solution to work with FSDP correctly
linkedin/Liger-Kernel#615
https://github.com/linkedin/Liger-Kernel/blob/2bb8dcfc18f10ff90f942f238b5cfe16c12749b6/src/liger_kernel/transformers/trainer/orpo_trainer.py#L18-L66

kashif · 2025-03-26T16:35:36Z

@VProv, at the moment, I was having issues getting the same outputs/metrics with and without liger in the trainer.

VProv · 2025-03-26T17:18:59Z

@VProv, at the moment, I was having issues getting the same outputs/metrics with and without liger in the trainer.

What setup are you using?

vaibhavjindal · 2025-04-22T22:03:51Z

Hi, I am working on fixing the output/metrics issue.
Added a PR in liger-kernel: linkedin/Liger-Kernel#676

vaibhavjindal · 2025-04-23T09:33:18Z

@kashif @qgallouedec can you please review the following PR which fixes the output/metrics issue? Thanks :)
#3346

kashif · 2025-05-05T08:18:40Z

thanks @hanbyul-kim for the report

vaibhavjindal · 2025-06-09T22:54:01Z

@kashif just wanted to circle back and see if we can merge this now? We wanted to try it out internally at Linkedin.

qgallouedec · 2025-06-11T13:38:38Z

trl/trainer/dpo_trainer.py

    import wandb


+def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int) -> torch.Tensor:


pad_token_id isn't used?

trl/trainer/dpo_trainer.py

qgallouedec · 2025-06-12T08:02:52Z

trl/trainer/dpo_trainer.py



+def shift_tokens_right(input_ids: torch.Tensor, decoder_start_token_id: int) -> torch.Tensor:
+    """Shift input ids one token to the right, and pad with pad_token_id"""


this docstring ain't accurate I think

Co-authored-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co> Co-authored-by: Vaibhav Jindal <32337828+vaibhavjindal@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <gallouedec.quentin@gmail.com>

initial liger support

f50e74d

kashif mentioned this pull request Dec 22, 2024

[Tracking issue] Integrate native liger-kernel losses #2495

Open

7 tasks

kashif added 3 commits January 15, 2025 12:05

fix outputs

e3eebd3

fix config merge conflict

2d82b39

Merge branch 'main' into liger-dpo

50d341e

kashif added 2 commits January 15, 2025 12:19

fix comment

8ae06b1

fix peft training

cc2b7b9

qgallouedec reviewed Jan 17, 2025

View reviewed changes

tests/test_dpo_trainer.py Outdated Show resolved Hide resolved

qgallouedec added 3 commits January 17, 2025 16:31

use parametrized

03fd005

raise error as soon as dep is not met

5f4110f

move param to the right section

b22eb24

qgallouedec reviewed Jan 17, 2025

View reviewed changes

trl/trainer/dpo_trainer.py Outdated Show resolved Hide resolved

reducing memory doc

b8e6f8c

qgallouedec reviewed Jan 17, 2025

View reviewed changes

docs/source/reducing_memory_usage.md Show resolved Hide resolved

kashif added 8 commits January 21, 2025 14:57

use liger specifc method

6310dbd

Merge branch 'main' into liger-dpo

bdca4f1

Merge branch 'main' into liger-dpo

5efe4d0

update return signature

dbece54

Merge branch 'main' into liger-dpo

d21bd81

fix typo

c441925

fix tests

f1af5d6

truncation and logits to keep

2814228

VProv mentioned this pull request Mar 26, 2025

[WIP] PEFT 🤝 Liger DPO #3065

Closed

5 tasks

Merge branch 'main' into liger-dpo

94422db

kashif added 3 commits May 5, 2025 10:21

update liger to fix dpo bug

614e5d9

skip test for python 3.9

8939796

fix asserts

50a4adc

kashif and others added 7 commits June 10, 2025 09:41

Merge branch 'main' into liger-dpo

11fa70c

formatting

d47ab9f

fix issue due to wraparound from roll

30ee209

revert back tol

4a46a0d

fix skip test

9fbdf80

use unwrapped model

9178c67

fix and expand doc

68aba88

qgallouedec reviewed Jun 11, 2025

View reviewed changes

trl/trainer/dpo_trainer.py Show resolved Hide resolved

qgallouedec reviewed Jun 11, 2025

View reviewed changes

trl/trainer/dpo_trainer.py Show resolved Hide resolved

qgallouedec and others added 4 commits June 11, 2025 13:45

nits in test

6de6070

style

e067a15

fix doc

28963ce

fix for review

588ebf1

qgallouedec reviewed Jun 12, 2025

View reviewed changes

qgallouedec approved these changes Jun 12, 2025

View reviewed changes

kashif added 2 commits June 12, 2025 11:11

Merge branch 'main' into liger-dpo

f04100c

add Flush and truncate to liger

3e5802a

kashif merged commit 53c4a7c into main Jun 12, 2025
11 checks passed

kashif deleted the liger-dpo branch June 12, 2025 10:25

qgallouedec mentioned this pull request Jan 16, 2026

Refactor DPO #3906

Merged

		import wandb


		def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int) -> torch.Tensor:



		def shift_tokens_right(input_ids: torch.Tensor, decoder_start_token_id: int) -> torch.Tensor:
		"""Shift input ids one token to the right, and pad with pad_token_id"""

Comments

Conversation

kashif commented Jan 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Jan 15, 2025

Uh oh!

Uh oh!

Uh oh!

qgallouedec commented Jan 17, 2025

Uh oh!

Uh oh!

VProv commented Mar 26, 2025

Uh oh!

kashif commented Mar 26, 2025

Uh oh!

VProv commented Mar 26, 2025

Uh oh!

vaibhavjindal commented Apr 22, 2025

Uh oh!

vaibhavjindal commented Apr 23, 2025

Uh oh!

kashif commented May 5, 2025

Uh oh!

vaibhavjindal commented Jun 9, 2025

Uh oh!

qgallouedec Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

qgallouedec Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kashif commented Jan 14, 2025 •

edited

Loading