WIP : support for hloc toolbox (superpoint, superglue, netvlad,...) #1026

RandomPrototypes · 2022-11-26T07:21:52Z

This adds support for hloc which brings support for more recent features and matches such as superpoint and superglue.

I added command-line options to select the sfm toolbox (colmap, hloc), feature (sift, superpoint,...), and matcher (NN, superglue,...).

It requires hloc as dependency.

…olmap), just a first test, not ready yet

pull from head repository

nerfstudio/process_data/hloc_utils.py

tancik

Thanks for adding support for hloc. My main comment atm is around notifying users how to install hloc if it is not included. We should also add documentation for these alternative matches in the web docs.

RandomPrototypes · 2022-11-28T00:09:26Z

ok, I'll prepare a new version with these try/except for dependency, notification, and documentation update

…rfstudio into hcol_support

merge from main repo

…rfstudio into hcol_support

RandomPrototypes · 2022-11-28T17:31:52Z

I updated the PR :
hloc is now an optional dependency. If it is not installed, it will not cause any error except if we try to use a feature from it (superpoint, superglue,...), it will output a red error message : "Error: To use this set of parameters (superpoint_aachen/superglue/hloc), you must install hloc toolbox!!"

The documentation of ns-process-data has a new line "You may also want to install hloc for more feature detector and matcher options." with a link to the github page. The new cli arguments (sfm_tool, feature_type, matcher_type) have a description in the doc too.

merge from main repo

Fix torch spherical harmonics (nerfstudio-project#1054)

tancik

Thanks for the updates! Almost ready, just a few small additional comments.

scripts/process_data.py

… type hints and Args/Returns doc

tancik · 2022-12-04T01:45:45Z

scripts/process_data.py

@@ -127,7 +103,7 @@ def main(self) -> None:
        if not self.skip_colmap:
            colmap_dir.mkdir(parents=True, exist_ok=True)

-            (sfm_tool, feature_type, matcher_type) = find_tool_feature_matcher_combination(
+            (sfm_tool, feature_type, matcher_type) = process_data_utils.find_tool_feature_matcher_combination(


Can you raise an error if no valid tool is found.

find_tool_feature_matcher_combination returns (None,None,None) when no valid tool is found, which brings to the message CONSOLE.log("[bold yellow]Warning: Invalid combination of sfm_tool, feature_type, and matcher_type ").

I'll add a sys.exit(1) after the message is printed.

I changed the message to "[bold red]Invalid combination of sfm_tool, feature_type, and matcher_type, exiting" and added sys.exit(1)

tancik · 2022-12-04T01:45:52Z

scripts/process_data.py

@@ -251,7 +227,7 @@ def main(self) -> None:
        if not self.skip_colmap:
            colmap_dir.mkdir(parents=True, exist_ok=True)

-            (sfm_tool, feature_type, matcher_type) = find_tool_feature_matcher_combination(
+            (sfm_tool, feature_type, matcher_type) = process_data_utils.find_tool_feature_matcher_combination(


same as other message

…d matcher

tancik

LGTM, Thanks for adding this functionality! Sorry for the delayed reviews.

…erfstudio-project#1026) * basic support for hloc (includes netvlad, super point, superglue in colmap), just a first test, not ready yet * add exhaustive match option for hloc * add command line option to choose the type of feature, matcher,... * make hloc an optional dependency, update doc, and small bug fixes * fix missing imports * fix warnings, pylint,... * fix overwrite previous commits * fix import order from isort * move find_tool_feature_matcher_combination to process_data_utils, add type hints and Args/Returns doc * error message when no valid combination of sfm tool, feature type, and matcher Co-authored-by: RandomPrototypes <[email protected]>

Diana-Zeng · 2023-04-10T11:38:25Z

When I use the command:
ns-process-data images --data /media/diana/001/c
--output /home/diana/NeRF/nerfstudio/selfdata/20230404/cc
--camera perspective
--matching vocab_tree
--sfm-tool hloc
--feature-type superpoint_aachen
--matcher-type superglue
--colmap-cmd /home/diana/COLMAP/colmap/build/src/exe/colmap

This error occurred:
scipy.io.matlab._miobase.MatReadError: Mat file appears to be empty.

These are complete information:
process_data_utils.py:363
[2023/04/10 18:58:47 hloc INFO] Extracting local features with configuration:
{'model': {'max_keypoints': 4096, 'name': 'superpoint', 'nms_radius': 3},
'output': 'feats-superpoint-n4096-r1024',
'preprocessing': {'grayscale': True, 'resize_max': 1024}}
[2023/04/10 18:58:47 hloc INFO] Skipping the extraction.
[2023/04/10 18:58:47 hloc INFO] Extracting local features with configuration:
{'model': {'name': 'netvlad'},
'output': 'global-feats-netvlad',
'preprocessing': {'resize_max': 1024}}
[2023/04/10 18:58:47 hloc INFO] Found 1440 images in root /home/diana/NeRF/nerfstudio/selfdata/20230404/cc/images.
Traceback (most recent call last):
File "/home/diana/anaconda3/envs/nerfstudio/bin/ns-process-data", line 8, in
sys.exit(entrypoint())
File "/home/diana/NeRF/nerfstudio/scripts/process_data.py", line 833, in entrypoint
tyro.cli(Commands).main()
File "/home/diana/NeRF/nerfstudio/scripts/process_data.py", line 165, in main
self._run_colmap(image_dir, colmap_dir)
File "/home/diana/NeRF/nerfstudio/scripts/process_data.py", line 231, in _run_colmap
hloc_utils.run_hloc(
File "/home/diana/NeRF/nerfstudio/nerfstudio/process_data/hloc_utils.py", line 110, in run_hloc
retrieval_path = extract_features.main(retrieval_conf, image_dir, outputs)
File "/home/diana/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/diana/Tools/Hierarchical-Localization/hloc/extract_features.py", line 251, in main
model = Model(conf['model']).eval().to(device)
File "/home/diana/Tools/Hierarchical-Localization/hloc/utils/base_model.py", line 17, in init
self._init(conf)
File "/home/diana/Tools/Hierarchical-Localization/hloc/extractors/netvlad.py", line 82, in _init
mat = loadmat(checkpoint, struct_as_record=False, squeeze_me=True)
File "/home/diana/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/scipy/io/matlab/_mio.py", line 225, in loadmat
MR, _ = mat_reader_factory(f, **kwargs)
File "/home/diana/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/scipy/io/matlab/_mio.py", line 74, in mat_reader_factory
mjv, mnv = _get_matfile_version(byte_stream)
File "/home/diana/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/scipy/io/matlab/_miobase.py", line 234, in _get_matfile_version
raise MatReadError("Mat file appears to be empty")
scipy.io.matlab._miobase.MatReadError: Mat file appears to be empty

Does anybody meet the same problem? Know the reason or know how to solve this? ? It is important and urgent for me!!! Thanks a lot!!!

…erfstudio-project#1026) * basic support for hloc (includes netvlad, super point, superglue in colmap), just a first test, not ready yet * add exhaustive match option for hloc * add command line option to choose the type of feature, matcher,... * make hloc an optional dependency, update doc, and small bug fixes * fix missing imports * fix warnings, pylint,... * fix overwrite previous commits * fix import order from isort * move find_tool_feature_matcher_combination to process_data_utils, add type hints and Args/Returns doc * error message when no valid combination of sfm tool, feature type, and matcher Co-authored-by: RandomPrototypes <[email protected]>

delmottea and others added 5 commits November 23, 2022 01:12

basic support for hloc (includes netvlad, super point, superglue in c…

2ee385d

…olmap), just a first test, not ready yet

add exhaustive match option for hloc

28f3818

add command line option to choose the type of feature, matcher,...

bdb5ad3

Merge pull request #1 from nerfstudio-project/main

db8c105

pull from head repository

Merge branch 'main' into hcol_support

c58ecbc

tancik reviewed Nov 28, 2022

View reviewed changes

nerfstudio/process_data/hloc_utils.py Outdated Show resolved Hide resolved

tancik reviewed Nov 28, 2022

View reviewed changes

nerfstudio/process_data/hloc_utils.py Outdated Show resolved Hide resolved

tancik reviewed Nov 28, 2022

View reviewed changes

delmottea and others added 5 commits November 29, 2022 01:57

make hloc an optional dependency, update doc, and small bug fixes

71ef810

Merge branch 'hcol_support' of https://github.com/RandomPrototypes/ne…

fee4977

…rfstudio into hcol_support

Merge pull request #2 from nerfstudio-project/main

0442286

merge from main repo

fix missing imports

8659fc3

Merge branch 'hcol_support' of https://github.com/RandomPrototypes/ne…

6cc95d9

…rfstudio into hcol_support

delmottea and others added 11 commits November 29, 2022 21:25

fix warnings, pylint,...

579e74d

Merge pull request #3 from nerfstudio-project/main

8e550f1

merge from main repo

fix overwrite previous commits

9f1bdd9

fix import order from isort

d6cd658

Merge branch 'main' into hcol_support

c73cb84

Merge pull request #4 from nerfstudio-project/main

48f9cd6

Fix torch spherical harmonics (nerfstudio-project#1054)

Merge branch 'main' into hcol_support

cf187ee

Merge branch 'main' into hcol_support

2d4c705

Merge branch 'main' into hcol_support

079c442

Merge branch 'main' into hcol_support

319a728

Merge branch 'main' into hcol_support

eec47a6

tancik reviewed Dec 2, 2022

View reviewed changes

scripts/process_data.py Outdated Show resolved Hide resolved

scripts/process_data.py Outdated Show resolved Hide resolved

scripts/process_data.py Outdated Show resolved Hide resolved

RandomPrototypes and others added 3 commits December 3, 2022 10:49

Merge branch 'main' into hcol_support

c7ab6c9

Merge branch 'main' into hcol_support

b009e7d

move find_tool_feature_matcher_combination to process_data_utils, add…

0cf1530

… type hints and Args/Returns doc

tancik reviewed Dec 4, 2022

View reviewed changes

error message when no valid combination of sfm tool, feature type, an…

d916f09

…d matcher

tancik approved these changes Dec 4, 2022

View reviewed changes

tancik merged commit 740cdeb into nerfstudio-project:main Dec 4, 2022

LingzheZhao mentioned this pull request May 29, 2024

Using hloc for camera pose WU-CVGL/BAD-Gaussians#10

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP : support for hloc toolbox (superpoint, superglue, netvlad,...) #1026

WIP : support for hloc toolbox (superpoint, superglue, netvlad,...) #1026

RandomPrototypes commented Nov 26, 2022

tancik left a comment

RandomPrototypes commented Nov 28, 2022

RandomPrototypes commented Nov 28, 2022

tancik left a comment

tancik Dec 4, 2022

RandomPrototypes Dec 4, 2022

RandomPrototypes Dec 4, 2022

tancik Dec 4, 2022

RandomPrototypes Dec 4, 2022

tancik left a comment

Diana-Zeng commented Apr 10, 2023

WIP : support for hloc toolbox (superpoint, superglue, netvlad,...) #1026

WIP : support for hloc toolbox (superpoint, superglue, netvlad,...) #1026

Conversation

RandomPrototypes commented Nov 26, 2022

tancik left a comment

Choose a reason for hiding this comment

RandomPrototypes commented Nov 28, 2022

RandomPrototypes commented Nov 28, 2022

tancik left a comment

Choose a reason for hiding this comment

tancik Dec 4, 2022

Choose a reason for hiding this comment

RandomPrototypes Dec 4, 2022

Choose a reason for hiding this comment

RandomPrototypes Dec 4, 2022

Choose a reason for hiding this comment

tancik Dec 4, 2022

Choose a reason for hiding this comment

RandomPrototypes Dec 4, 2022

Choose a reason for hiding this comment

tancik left a comment

Choose a reason for hiding this comment

Diana-Zeng commented Apr 10, 2023