Repository navigation
Miscellaneous inference cleanup #2955
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 16 commits
Commits
Show all changes
52 commits
Select commit
Hold shift + click to select a range
85efc1f
WIP engine and context cleanup
santhnm2 ca8b278
Tests pass
santhnm2 fafb2d2
Merge with main
santhnm2 01a8237
Undo attention.py change
santhnm2 faa935b
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 6da70fb
Fix unit tests
santhnm2 2f67856
Fix typo and clean up run_inference_performance_test.py
santhnm2 c8d3934
Merge branch 'main' into engine_context_cleanup
santhnm2 b801c9c
Address reviewer feedback
santhnm2 cff3c1e
Fix copyright
santhnm2 b6d9c93
Add back deprecated args with exception
santhnm2 ed6f96a
Merge with main
santhnm2 98d5acc
Add deprecation exception for GPTInferenceWrapper
santhnm2 ec2fad2
Fix wandb test
santhnm2 2dea9b3
Address reviewer comments
santhnm2 e5edb9a
Bug fixes
santhnm2 4b6cb9c
Major refactor - introduce DynamicInferenceConfig
santhnm2 8e8225a
Merge with main
santhnm2 bf66e63
Add inference config unit test
santhnm2 d14f476
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 f1af84f
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 188dcf2
Address reviewer feedback
santhnm2 92b1a8d
Fix copyright
santhnm2 c76d953
Fix import
santhnm2 71e790c
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 952dd38
Fix mamba test
santhnm2 84c6123
Fix unit tests
santhnm2 c50d844
Merge branch 'main' into engine_context_cleanup
santhnm2 ef50892
Fix example scripts
santhnm2 39c92c6
Update tools/run_inference_performance_test.py
santhnm2 9665804
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 fbe9db6
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 bdde060
Add explicit deprecation error
santhnm2 9bf08a8
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 964302c
Remove --inference-max-batch-size
santhnm2 ad80dc7
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 97cb2d6
RL fixes
santhnm2 58f5be5
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 70e4e3a
Fix perf test
santhnm2 c40e18e
Merge with main
santhnm2 0e4e286
Merge branch 'main' into engine_context_cleanup
santhnm2 6d85bf2
Merge with main
santhnm2 7080773
Merge with main
santhnm2 c3e10a2
Fix static functional test
santhnm2 b9fd22a
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 a57af29
Try removing arguments that are now in TransformerConfig
santhnm2 21c6d2d
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 dbb8614
Bug fixes
santhnm2 5474dea
Merge remote-tracking branch 'upstream/main' into engine_context_cleanup
santhnm2 ca27708
Fix formatting
santhnm2 80ddc9a
More bug fixes
santhnm2 84b443d
Merge branch 'main' into engine_context_cleanup
santhnm2 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
something about passing global args makes me really uncomfortable ... I feel like we should probably have an
DynamicInferenceContextConfig(or justInferenceContextConfig), which would be primarily for the dynamic context at this pointThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see your point, but I think ideally we should have a
from_argsfunction somewhere or else we'll be back at the same place we're at now where we have to manually initialize theDynamicInferenceContextConfigin multiple places. Maybe one compromise option is we bring backmegatron/inferenceas a folder that exists for any inference utilities that shouldn't live inmegatron/core/inference(analogous to currentmegatron/training), and then put the factory functionget_dynamic_inference_context_config_from_argsthere. What do you think?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We currently use
examples/inference/gptas the location of similar utility functions but I think it would be nice to have a more central placeUh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also is passing the
MegatronModuledirectly into core still ok?EDIT: It's defined in
megatron/core/transformer/module.pyso I think this should be fineThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the idea of a utility method to do
context_config = from_args(args)that lives outside of core. Great question about amegatron/inferencedirectory ... it feels likeexamples/inferencehas become the de-facto place for non-core inference. I think it makes sense to startmegatron/inference, and I'd imagine we can even move some sub-components of the example scripts in there.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok sounds good, I'll make these changes. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @kvareddy @jaredcasper for viz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have now added
DynamicInferenceConfigand aget_dynamic_inference_config_from_model_and_argsfunction inmegatron/inference/utils.py