Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add nuScenes-based lidar examples #4407

Merged
merged 27 commits into from
Dec 7, 2023
Merged

Add nuScenes-based lidar examples #4407

merged 27 commits into from
Dec 7, 2023

Conversation

roym899
Copy link
Collaborator

@roym899 roym899 commented Nov 30, 2023 •
edited by github-actions bot
Loading

What

Adds two new examples based on the nuScenes dataset. One example visualizing most of the data (lidar, radar, camera, and 3D bounding boxes) in the nuScenes dataset, and a more minimal example that only visualizes the lidar data in the lidar reference frame.

Lidar example
lidar

nuScenes example
nuscenes

The download module is included in both examples for now; if we have a place so this could be shared across examples, that would be nice.

Closes #2510.

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested app.rerun.io (if applicable)
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

@roym899 roym899 added examples Issues relating to the Rerun examples include in changelog labels Nov 30, 2023
nikolausWest
nikolausWest reviewed Nov 30, 2023
View reviewed changes
examples/python/nuscenes/main.py
Comment on lines +101 to +102
point_distances = np.linalg.norm(points, axis=1)
point_colors = cmap(norm(point_distances))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Link to distance based color mapping

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

nikolausWest
nikolausWest reviewed Nov 30, 2023
View reviewed changes
examples/python/nuscenes/main.py Outdated
sensor_name = sample_data["channel"]
calibrated_sensor_token = sample_data["calibrated_sensor_token"]
calibrated_sensor = nusc.get("calibrated_sensor", calibrated_sensor_token)
rotation_xyzw = np.roll(calibrated_sensor["rotation"], shift=-1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add comment on why you do the shift

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

nikolausWest
nikolausWest reviewed Nov 30, 2023
View reviewed changes
examples/python/nuscenes/main.py Outdated
parser.add_argument(
"--root_dir",
type=pathlib.Path,
default="dataset",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense for this dataset dir to default to <script dir>/dataset ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed it.

nikolausWest
nikolausWest approved these changes Nov 30, 2023
View reviewed changes
Copy link
Member

@nikolausWest nikolausWest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Realized I accidentally left all my comments as "single comment", sorry! Either way, lgtm!

@nikolausWest
Copy link
Member

Actually, one more question after playing with it: is the nuScenes dataset missing an annotation context?

@nikolausWest
Copy link
Member

It would also be cool to log a car mesh that moves around where the car is supposed to be but definitely ok to skip

@roym899
Copy link
Collaborator Author

roym899 commented Nov 30, 2023 •
edited
Loading

I just tried adding an annotation context, but the labels are taking a lot of space. Is there any benefit in having the annotation context with just the ids?

Some classes get the same color right now, but it's quite rare; I'm not sure if that warrants adding manual colors for all classes.

@nikolausWest
Copy link
Member

I just tried adding an annotation context, but the labels are taking a lot of space. Is there any benefit in having the annotation context with just the ids?

Do you mean they take a lot of visual space in the 3D scene? In that case we should probably make showing the labels in 3D optional. It's still useful to be able to hover over the bbox and see what class it is. Could you open an issue?

@emilk
Copy link
Member

emilk commented Dec 6, 2023

Do I need to do something special to run this?

> pip install -r ./examples/python/requirements.txt
…
ERROR: Could not find a version that satisfies the requirement pycollad (from versions: none)
ERROR: No matching distribution found for pycollad

❯ examples/python/nuscenes/main.py                 
DEV ENVIRONMENT DETECTED! Re-importing rerun from: /Users/emilk/code/rerun/rerun/rerun_py/rerun_sdk
Traceback (most recent call last):
  File "/Users/emilk/code/rerun/rerun/examples/python/nuscenes/main.py", line 12, in <module>
    from nuscenes import nuscenes
ModuleNotFoundError: No module named 'nuscenes'

(Python 3.10.12)

@roym899
Copy link
Collaborator Author

roym899 commented Dec 6, 2023 •
edited
Loading

Hm, that's strange. I didn't need anything else. I guess pip install nuscenes-devkit fails for you?

I think the error you're getting might be from another requirements file (ros_node) and then it doesn't even attempt to install the other requirements?

@roym899 roym899 marked this pull request as ready for review December 6, 2023 15:44
@emilk
Copy link
Member

emilk commented Dec 7, 2023

Oh yeah, I was pip installing the wrong requirements.txt file 🤦 Now it seems to be working (downloading dataset…)

@roym899 roym899 changed the title nuScenes-Based Lidar Examples Add nuScenes-based lidar examples Dec 7, 2023
@roym899 roym899 merged commit f6781a9 into main Dec 7, 2023
41 checks passed
@roym899 roym899 deleted the leo/nuscenes branch December 7, 2023 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nikolausWest nikolausWest nikolausWest approved these changes

Assignees
No one assigned
Labels
examples Issues relating to the Rerun examples include in changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add Lidar example
3 participants
@roym899 @nikolausWest @emilk