Fix splatfacto crash in eval when using viewer #3430
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This addresses the issue in #3253 where splatfacto training could crash during the evaluation step when interacting with the viewer (e.g., with --vis viewer+wandb). The problem occurs because interacting with the viewer renders a new image in
_render_img()
, wheremodel.eval()
is called at the start andmodel.train()
at the end, regardless of the initial model state (see nerfstudio/viewer/render_state_machine.py#L151). If this happens during the actual eval step of the model, it flips the mode to training during evaluation, causing errors.Changes made:
eval_iteration()
to use thetrain_lock
, ensuring a consistent model state during evaluationTesting
Closes #3253