fit : add --fit-show-mem to print probe table at INFO#23232
Open
Bikkies wants to merge 1 commit into
Open
Conversation
- Opt-in flag that surfaces the per-device free-memory probe table (currently TRACE-only after ggml-org#23021) at INFO when set; default off.
Contributor
|
The console output for What should or should not be shown by default is I think another issue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds a flag -fsm / --fit-show-mem [on|off] that surfaces the per-device free-memory table from common_params_fit_impl at the INFO log level.
This is the table that used to show up by default before #23021 demoted it to TRACE. I found it really handy when manually adjusting params like tensor splits and context. Being able to see quickly how much VRAM was free per device made iterating on tensor splits to balance vram quicker without going through a full model load each time.
Adding this flag lets this table show and be easily visible without disappearing into the sea of other output when using higher verbose log output.
Personally, I think it should be on by default. I didn't even know it existed until it was showing up in my
terminal output, and others may now miss out on it the same way now that it's hidden behind TRACE. But
since #23021 deliberately moved it, this PR adds it as opt-in rather than overriding that decision.
Additional information
Requirements