Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inference pipeline #555

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

Inference pipeline #555

wants to merge 32 commits into from

Conversation

Dahoas
Copy link
Collaborator

@Dahoas Dahoas commented Sep 4, 2023

Implementation of multi-generation RL in trlX

Suggested (but optional) external inference pipeline wrapper can be found here

Dahoas and others added 30 commits July 18, 2023 12:00
@Dahoas
Implementing support for dense rewards
f04446d
@SharathRaparthy @Dahoas
added "num_return_sequences" param which corresponds to n in Best-of-…
13a01fc 
…N sampling
@SharathRaparthy @Dahoas
updates to "num_return_sequences" param
5421a73
@SharathRaparthy @Dahoas
BoN implementation
2f3ac28
@SharathRaparthy @Dahoas
Changed back to default.
2f1dace
@SharathRaparthy @Dahoas
TopK sampling instead of Top1
f58170d
@SharathRaparthy @Dahoas
summed along dim=1
be8bc1a
@SharathRaparthy @Dahoas
Generating samples in chunks
608d812
@SharathRaparthy @Dahoas
added gen_chunk_size parameter
d8557e7
@SharathRaparthy @Dahoas
chunking in forward prop
8ef9c36
@SharathRaparthy @Dahoas
chunking generations in train and eval
4c1d82d
@Dahoas
Implementing support for dense rewards
ecd5107
@Dahoas
Fix distributed ref_mean, ref_var bug for dense rewards
4071604
@Dahoas
Make generation respect max seq length
5f41413
@Dahoas
Make experience before first round of training
22ae83f
@Dahoas
Refactoring .generate/.generate_eval
7d0a4be
@Dahoas
Fix BoN metric support
b79dd19
@Dahoas
Enforce chunk_size param for eval generation when present
cb49dc5
@Dahoas
Fix: Don't shuffle prompt dataset
e290412
@Dahoas
Move inputs to device
391d04c
@Dahoas
Fix style
8de84e4
@Dahoas
Fix: Do not shuffle empty experience dataloader
404ef14
@Dahoas
Make experience before first round of training
67b711a
@Dahoas
Refactoring .generate/.generate_eval
34e185a
@Dahoas
Refactored decode, make_experience and added support for external ref…
11e1e95 
…erence models
@Dahoas
Fix BoN sampling after big refactor
527ba23
@Dahoas
Fixing style
676a1cd
@Dahoas
Fix: attention_mask indexing error
e43f7b1
@Dahoas
Feature: Support pipelined rollouts
1018748
@Dahoas
Fix: Add mask arg to advantage whitening
3ec27ac
@Dahoas
Copy link
Collaborator Author

Dahoas commented Sep 4, 2023

Depends on #529

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxreciprocate maxreciprocate Awaiting requested review from maxreciprocate

@cat-state cat-state Awaiting requested review from cat-state

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Dahoas @SharathRaparthy