Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix context overflow #131

Merged
merged 24 commits into from
Dec 15, 2022
Merged

Fix context overflow #131

merged 24 commits into from
Dec 15, 2022

Conversation

maxreciprocate
Copy link
Collaborator

@maxreciprocate maxreciprocate commented Dec 9, 2022 •
edited
Loading

Fix of #130, this PR

  • limits generation's size to max_new_tokens
  • truncates prompts to seq_length - max_new_tokens
  • enables separate logging of prompts and generations

PPO
ILQL

@jon-tow jon-tow self-requested a review December 9, 2022 22:40
jon-tow
jon-tow reviewed Dec 10, 2022
View reviewed changes
Copy link
Collaborator

@jon-tow jon-tow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally and looks great! I left a few comments and questions 🙏

trlx/model/nn/ilql_models.py Show resolved Hide resolved
trlx/model/nn/ilql_models.py Show resolved Hide resolved
trlx/model/accelerate_base_model.py Show resolved Hide resolved
examples/randomwalks/configs/ppo_randomwalks.yml Show resolved Hide resolved
jon-tow
jon-tow approved these changes Dec 12, 2022
View reviewed changes
Copy link
Collaborator

@jon-tow jon-tow left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks for clearing up my questions.

Note: Awaiting logs before merging.

@LouisCastricato
Copy link
Contributor

When are we merging

Dahoas
Dahoas reviewed Dec 14, 2022
View reviewed changes
trlx/model/nn/ppo_models.py
@@ -162,10 +163,12 @@ def loss(
old_values - self.cliprange_value,
old_values + self.cliprange_value,
)
n = mask.sum()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure but shouldn't n be vector valued where each component is the size of the ith generation? (Before endoftext)

Copy link
Collaborator Author

@maxreciprocate maxreciprocate Dec 14, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is used only in reductions to scalars so no

@jon-tow jon-tow mentioned this pull request Dec 14, 2022
Merged
jon-tow
jon-tow reviewed Dec 15, 2022
View reviewed changes
Copy link
Collaborator

@jon-tow jon-tow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left one tiny nit if you could take a look!

trlx/model/nn/ppo_models.py Outdated Show resolved Hide resolved
@maxreciprocate maxreciprocate merged commit dc0e060 into main Dec 15, 2022
@maxreciprocate maxreciprocate deleted the fix-context-overflow branch December 15, 2022 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jon-tow jon-tow jon-tow approved these changes

@Dahoas Dahoas Dahoas left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@maxreciprocate @LouisCastricato @Dahoas @jon-tow