[NF4] .to() fixes
#1312
[NF4] .to() fixes
#1312
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1312
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ No FailuresAs of commit f309aea with merge base 26648c2 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
|
||
|
|
||
| @implements_torch_function(torch.Tensor.cuda) | ||
| def function_cuda(*args, **kwargs): |
There was a problem hiding this comment.
does this func not work call_from_inner_tensors
There was a problem hiding this comment.
call_from_inner_tensors() does not call the method on .scaler_mean and .nf4 attribute, hence I use __tensor_flatten__ instead.
There was a problem hiding this comment.
I see. That's another anti-pattern since call_from_inner_tensors is typically applied for all the tensors and not just specific to sharding properties. But this makes sense. Maybe we should just like update it and have a flag that says ignore sharding or not.
There was a problem hiding this comment.
Otherwise, thank you. Yeah, I really didn't like the to calling dequant secretly, so I think this more aligns with what we've seen in the rest of the library.
There was a problem hiding this comment.
Yea I also penned some of my thoughts in #1310.
To clarify, this PR does not change the behavior "to calling dequant secretly (when dtype is specified)". Apart from fixes for .scaler_mean and .nf4 attributes, this PR only changes .cuda() behavior to not dequantize (previously .cuda() will propagate to aten._to_copy, which dequantize), so it's more consistent with .cpu() as well as the general "not dequantize when dtype is not specified".
|
@ebsmothers Are there any ways to validate this against some tune flows to make sure it doesn't break anything? |
Fixes #1310