Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ns-train semantic-nerfw (friends dataset) output goes and stays black between step 82000 and 84000 #1072

Closed
machenmusik opened this issue Dec 2, 2022 · 3 comments
Closed

ns-train semantic-nerfw (friends dataset) output goes and stays black between step 82000 and 84000 #1072

machenmusik opened this issue Dec 2, 2022 · 3 comments

Comments

@machenmusik
Copy link
Contributor

Describe the bug
ns-train semantic-nerfw (friends dataset) output goes and stays black between step 82000 and 84000

To Reproduce
Steps to reproduce the behavior:

  1. Download or manually copy the friends dataset.
  2. ns-train semantic-nerfw --trainer.save-only-latest-checkpoint False
  3. Open the Web interface to see what's going on
  4. At/after step 84000 (not exactly, perhaps a bit before; takes quite a while to get there) render is black, aargh
  5. ns-train semantic-nerfw --trainer.load-dir outputs\data\friends\TBBT-big_living_room\semantic-nerfw\YOUR_TIMESTAMP_HERE\nerfstudio_models --viewer.start-train False --trainer.load-step 82000
  6. Open / refresh the Web interface to see what's going on
  7. Notice that render is fine (not black)

Expected behavior
render should not have gone black...
@tancik
Copy link
Contributor

tancik commented Dec 2, 2022

This is related to #873
Until the NAN error is resolved, we do not recommend training past 30k iters.

@machenmusik
Copy link
Contributor Author

thanks - then perhaps max-num-iterations should be lowered temporarily?

machenmusik added a commit to machenmusik/nerfstudio that referenced this issue Dec 2, 2022
@machenmusik
limit semantic-nerfw iterations to 30000
fa39a52 
Per discussion at nerfstudio-project#1072 (comment)
@machenmusik machenmusik mentioned this issue Dec 2, 2022
Merged
tancik pushed a commit that referenced this issue Dec 2, 2022
@machenmusik
limit semantic-nerfw iterations to 30000 (#1074)
d9395f2 
Per discussion at #1072 (comment)
@machenmusik
Copy link
Contributor Author

Closed via #1074

tancik pushed a commit to dozeri83/nerfstudio that referenced this issue Jan 20, 2023
@machenmusik @tancik
limit semantic-nerfw iterations to 30000 (nerfstudio-project#1074)
7702ee6 
Per discussion at nerfstudio-project#1072 (comment)
chris838 pushed a commit to chris838/nerfstudio that referenced this issue Apr 22, 2023
@machenmusik
limit semantic-nerfw iterations to 30000 (nerfstudio-project#1074)
6db4337 
Per discussion at nerfstudio-project#1072 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@machenmusik @tancik