Fix benchmarking scripts #1005
Merged
Fix benchmarking scripts #1005
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
devernay
commented
Nov 22, 2022
•
devernay
commented
- closes nerfacto overfitting on blender scenes? #1000
- see also Poor performance on NeRF Blender Synthetic Data & potential bugs #806
- launch_train_blender.sh:
- add -s option to launch a single job per GPU
- add -v option to use tensorboard instead of wandb
- set nerfacto options according to Poor performance on NeRF Blender Synthetic Data & potential bugs #806 (comment)
- use a single timestamp for all training jobs
- last GPU was ignored
- kill all subprocesses when script is terminated
- print the eval script command-line
- launch_eval_blender.sh:
- add shebang
- add -s option to launch a single job per GPU
- last GPU was ignored
- kill all subprocesses when script is terminated
- update benchmarking doc
- add shebang - add -s option to launch a single job per GPU - last GPU was ignored - kill all subprocesses when script is terminated
- add -s option to launch a single job per GPU - add -v option to use tensorboard instead of wandb - set nerfacto options according to #806 (comment) - use a single timestamp for all training jobs - last GPU was ignored - kill all subprocesses when script is terminated - print the eval script command-line
tancik
reviewed
Nov 22, 2022
| @@ -71,6 +73,7 @@ The flags used in the benchmarking script are defined as follows: | |||
| - `-m`: config name (e.g. `instant-ngp`). This should be the same as what was passed in for -c in the train script. | |||
| - `-o`: base output directory for where all of the benchmarks are stored (e.g. `outputs/`). Corresponds to the `--output-dir` in the base `Config` for training. | |||
| - `-t`: timestamp of benchmark; also the identifier (e.g. `2022-08-10_172517`). | |||
| - `-s`: Launch a single job per GPU. | |||
| - `-g`: specifies the gpus to use and if not specified (no -g flag), will automaticaly search for available gpus. | |||
There was a problem hiding this comment.
That's a flag I added. It basically launches one job per GPU, then waits on the first one to be finished before relaunching on that GPU.
In the previous version, all jobs were launched in parallel at script launch (and Ctrl-C didn't kill the jobs). That works fine if you have several GPUs and lots of GPU memory, but not if you have a single 16Gb GPU.
There was a problem hiding this comment.
Ahh sorry, I was trying to highlight the -g flag.
There was a problem hiding this comment.
well this flag was simply ignored in the previous version of the scripts. It just took the list of remaining arguments as the list of GPUs, so I kept it that way and removed the unused flag. I'll adjust this doc.
tancik
pushed a commit
to dozeri83/nerfstudio
that referenced
this pull request
Jan 20, 2023
* launch_eval_blender.sh: fix script - add shebang - add -s option to launch a single job per GPU - last GPU was ignored - kill all subprocesses when script is terminated * launch_train_blender.sh: fix script - add -s option to launch a single job per GPU - add -v option to use tensorboard instead of wandb - set nerfacto options according to nerfstudio-project#806 (comment) - use a single timestamp for all training jobs - last GPU was ignored - kill all subprocesses when script is terminated - print the eval script command-line * launch_eval_blender.sh: fix script * Update benchmarking.md * launch_train_blender.sh: add -s to eval command-line * Update launch_train_blender.sh * Update benchmarking.md
chris838
pushed a commit
to chris838/nerfstudio
that referenced
this pull request
Apr 22, 2023
* launch_eval_blender.sh: fix script - add shebang - add -s option to launch a single job per GPU - last GPU was ignored - kill all subprocesses when script is terminated * launch_train_blender.sh: fix script - add -s option to launch a single job per GPU - add -v option to use tensorboard instead of wandb - set nerfacto options according to nerfstudio-project#806 (comment) - use a single timestamp for all training jobs - last GPU was ignored - kill all subprocesses when script is terminated - print the eval script command-line * launch_eval_blender.sh: fix script * Update benchmarking.md * launch_train_blender.sh: add -s to eval command-line * Update launch_train_blender.sh * Update benchmarking.md
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.