Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix slow training issue when using mask #1114

Merged
merged 1 commit into from
Dec 13, 2022
Merged

Fix slow training issue when using mask #1114

merged 1 commit into from
Dec 13, 2022

Conversation

ttaoREtw
Copy link
Contributor

@ttaoREtw ttaoREtw commented Dec 12, 2022 •
edited
Loading

After fixing this bug, the training iter time improves from 600ms to 30ms.

[Info]
GPU: 3090
torch: 1.12.1+cu113
nerfacc: 0.2.1
nerfstudio: 0.1.12 (commit 90447ab)

@tancik
Copy link
Contributor

tancik commented Dec 13, 2022 •
edited
Loading

The .to_device() was added to address the issue in #1061
I wonder why it is different in these two cases?

@ttaoREtw
Copy link
Contributor Author

@hturki said the device here is cpu in #1061 (comment)
Is it the expected behavior? It seems to me computing on gpu is more efficient.
Not sure why device mismatch occurs in his case .

@tancik
Copy link
Contributor

tancik commented Dec 13, 2022

Is the device always on the gpu for you? If so, I though to_device doesn't do anything. But perhaps I've misunderstood.

@ttaoREtw
Copy link
Contributor Author

Is the device always on the gpu for you? If so, I though to_device doesn't do anything. But perhaps I've misunderstood.

I just printed out the device, it is indeed cpu.

tancik
tancik approved these changes Dec 13, 2022
View reviewed changes
Copy link
Contributor

@tancik tancik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I wonder why you aren't running into the save device error. I'm inclined to approve this PR given the speed up (others have also noted that it has been slow for them lately). We can investigate the device error more thoroughly if it re-emerges.

@tancik tancik merged commit 1a70665 into nerfstudio-project:main Dec 13, 2022
tancik pushed a commit to dozeri83/nerfstudio that referenced this pull request Jan 20, 2023
@ttaoREtw @tancik
Fix slow training issue when using mask (nerfstudio-project#1114)
aaff1b4
chris838 pushed a commit to chris838/nerfstudio that referenced this pull request Apr 22, 2023
@ttaoREtw
Fix slow training issue when using mask (nerfstudio-project#1114)
681c890
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tancik tancik tancik approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ttaoREtw @tancik