Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option for refining the TSDF using the mesh's AABB #3459

Merged
merged 1 commit into from
Oct 7, 2024

Conversation

nepfaff
Copy link
Contributor

@nepfaff nepfaff commented Oct 2, 2024

I found the TSDF mesh quality to be highly dependent on the TSDF grid location and resolution. Currently, the user is responsible for providing a tight bounding box around the object to ensure good TSDF grid resolution in the areas of interest. However, this is often inconvenient and slow. It might also be hard to achieve when the NeRF is trained on a server for which the viewer isn't available.

In this PR, I suggest a simple solution for refining a rough bounding box estimate using the computed bounding box of the original mesh. In practice, I found this to lead to big improvements.

Note that this assumes that the density estimates are good enough to prevent floaters far from the object. I set the option to False by default, as this will be an issue if no special care is taken.

I trained nerfacto with alpha-transparent training on images of a mustard bottle:

ns-train nerfacto \
--pipeline.model.background_color "random" \
--pipeline.model.disable-scene-contraction True \
--data data

Exporting with rough estimate + no TSDF refinement:

ns-export tsdf --load-config outputs/.../config.yml \
--output-dir exports \
--num-pixels-per-side 2048 \
--resolution 250 250 250 \
--use-bounding-box True \
--bounding-box-min -.5 -.5 -.5 \
--bounding-box-max .5 .5 .5

no_refinement01

Exporting with rough estimate + TSDF refinement:

ns-export tsdf --load-config outputs/.../config.yml \
--output-dir exports \
--num-pixels-per-side 2048 \
--resolution 250 250 250 \
--use-bounding-box True \
--bounding-box-min -.5 -.5 -.5 \
--bounding-box-max .5 .5 .5 \
--refine-mesh-using-initial-aabb-estimate True

refined00

The refined mesh is much smoother.

Copy link
Collaborator

@brentyi brentyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, thanks @nepfaff!

@brentyi brentyi merged commit 8e27e5f into nerfstudio-project:main Oct 7, 2024
3 checks passed
@nepfaff nepfaff deleted the tsdf_aabb_refinement branch October 7, 2024 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants