Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New face detection example based on MediaPipe #2360

Merged
merged 18 commits into from
Jun 14, 2023
Merged

New face detection example based on MediaPipe #2360

merged 18 commits into from
Jun 14, 2023

Conversation

abey79
Copy link
Member

@abey79 abey79 commented Jun 10, 2023 •
edited by github-actions bot
Loading

What

Closes #2338

New example based on MediaPipe's Face Detector and Landmarks solutions.

image

In the future, this example could benefit from a custom blueprint when the API is stabilised enough:

  • split face detect and face landmark into separate views (both with the image in the background)
  • proper default orientation for the face 3D model
  • general layout order (cam views -> 3D view -> plots)

Checklist

PR Build Summary: https://build.rerun.io/pr/2360

Docs preview: https://rerun.io/preview/275e36b/docs
Examples preview: https://rerun.io/preview/275e36b/examples

@abey79 abey79 added the examples Issues relating to the Rerun examples label Jun 10, 2023
@abey79 abey79 self-assigned this Jun 10, 2023
@abey79 abey79 marked this pull request as ready for review June 12, 2023 14:33
emilk
emilk reviewed Jun 12, 2023
View reviewed changes
Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to use rr.log_view_coordinates to make the 3D face view orient better

examples/python/mp_face/README.md Outdated Show resolved Hide resolved
emilk
emilk reviewed Jun 12, 2023
View reviewed changes
Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool demo! But the plot gets really slow really quickly 😬 (not the examples fault, just shows a pre-existing performance problem)

@abey79 abey79 added the do-not-merge Do not merge this PR label Jun 12, 2023
@abey79
Copy link
Member Author

abey79 commented Jun 12, 2023 •
edited
Loading

Just added a log_transform3d call to better centre the 3d face by default, but apparently the transform doesn't affect the points3d's connections (from annotation):

image

Edit: #2402

nikolausWest
nikolausWest approved these changes Jun 12, 2023
View reviewed changes
examples/python/mp_face/main.py Outdated Show resolved Hide resolved
examples/python/mp_face/main.py Outdated Show resolved Hide resolved
examples/python/mp_face/main.py Outdated Show resolved Hide resolved
examples/python/mp_face/main.py Outdated Show resolved Hide resolved
examples/python/mp_face/main.py Outdated Show resolved Hide resolved
@abey79
Copy link
Member Author

abey79 commented Jun 13, 2023

You should be able to use rr.log_view_coordinates to make the 3D face view orient better

Done. I also had an issue where OpenCV would sometime return a blank first frame, leading to a spurious log that would keep the viewer's auto-entering mechanism to work properly.

@abey79
Misc fixes
7d5b995 
- added clarification comments
- renamed some function
- tqdm download progress
- skip blank frame
- made view coordinates timeless
@abey79
Copy link
Member Author

abey79 commented Jun 13, 2023

I still have cases where auto-centring doesn't work as expected—not sure what's going on. Anyways, this is easily addressed by reseting the blueprint so I think we can move forward.

@abey79 abey79 removed the do-not-merge Do not merge this PR label Jun 13, 2023
@abey79 abey79 mentioned this pull request Jun 14, 2023
Merged
7 tasks
@abey79 abey79 merged commit 07c2a84 into main Jun 14, 2023
@abey79 abey79 deleted the antoine/mp_face_example branch June 14, 2023 12:48
abey79 added a commit that referenced this pull request Jun 14, 2023
@nikolausWest @abey79
Update names, titles and tags for examples with real data (#2416)
7c59e05 
### What
Updates the titles, and adds tags to all "python examples with real
data" (a separate PR is needed for the "fake data" examples).

Renames:

| before | after |
| --- | --- |
| arkitscenes | arkit_scenes |
| colmap | structure_from_motion |
| deep_sdf | signed_distance_fields |
| dicom | dicom_mri |
| mp_pose | human_pose_tracking |
| nyud | rgbd |
| opencv_canny | live_camera_edge_detection |
| ros | ros_node |
| segment_anything | segment_anything_model |
| stable_diffusion | depth_guided_stable_diffusion |
| tracking_hf_opencv | detect_and_track_objects |

In order to minimise merge issues, this is best updated and merged
AFTER:
- #2419
- #2420
- #2418
- #2424
- #2426
- #2360 

TODO

- [x] Update names script names
- [x] Update any references to the scripts
- [ ] ~~Mirror updates for Rust examples~~ (none of the renamed examples
have rust counterparts)
- [x] Convert tags to list
- [x] Quick typo/grammar pass with pycharm's checker

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [ ] I've included a screenshot or gif (if applicable)

<!-- This line will get updated when the PR build summary job finishes.
-->
PR Build Summary: https://build.rerun.io/pr/2416

<!-- pr-link-docs:start -->
Docs preview: https://rerun.io/preview/3fb07ee/docs
Examples preview: https://rerun.io/preview/3fb07ee/examples
<!-- pr-link-docs:end -->

---------

Co-authored-by: Antoine Beyeler <[email protected]>
emilk pushed a commit that referenced this pull request Jun 15, 2023
@abey79 @emilk
New face detection example based on MediaPipe (#2360)
122832e 
### What

Closes #2338

New example based on
[MediaPipe](https://developers.google.com/mediapipe)'s Face Detector and
Landmarks solutions.

<img alt="image"
src="https://static.rerun.io/6efefd1de7c63166c4d99956711b6367206d9835_mp_face.png">

In the future, this example could benefit from a custom blueprint when
the API is stabilised enough:
- split face detect and face landmark into separate views (both with the
image in the background)
  - proper default orientation for the face 3D model
  - general layout order (cam views -> 3D view -> plots)

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)

<!-- This line will get updated when the PR build summary job finishes.
-->
PR Build Summary: https://build.rerun.io/pr/2360

<!-- pr-link-docs:start -->
Docs preview: https://rerun.io/preview/275e36b/docs
Examples preview: https://rerun.io/preview/275e36b/examples
<!-- pr-link-docs:end -->
emilk pushed a commit that referenced this pull request Jun 15, 2023
@nikolausWest @abey79 @emilk
Update names, titles and tags for examples with real data (#2416)
06fa7da 
Updates the titles, and adds tags to all "python examples with real
data" (a separate PR is needed for the "fake data" examples).

Renames:

| before | after |
| --- | --- |
| arkitscenes | arkit_scenes |
| colmap | structure_from_motion |
| deep_sdf | signed_distance_fields |
| dicom | dicom_mri |
| mp_pose | human_pose_tracking |
| nyud | rgbd |
| opencv_canny | live_camera_edge_detection |
| ros | ros_node |
| segment_anything | segment_anything_model |
| stable_diffusion | depth_guided_stable_diffusion |
| tracking_hf_opencv | detect_and_track_objects |

In order to minimise merge issues, this is best updated and merged
AFTER:
- #2419
- #2420
- #2418
- #2424
- #2426
- #2360

TODO

- [x] Update names script names
- [x] Update any references to the scripts
- [ ] ~~Mirror updates for Rust examples~~ (none of the renamed examples
have rust counterparts)
- [x] Convert tags to list
- [x] Quick typo/grammar pass with pycharm's checker

* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [ ] I've included a screenshot or gif (if applicable)

<!-- This line will get updated when the PR build summary job finishes.
-->
PR Build Summary: https://build.rerun.io/pr/2416

<!-- pr-link-docs:start -->
Docs preview: https://rerun.io/preview/3fb07ee/docs
Examples preview: https://rerun.io/preview/3fb07ee/examples
<!-- pr-link-docs:end -->

---------

Co-authored-by: Antoine Beyeler <[email protected]>
This was referenced Jun 15, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nikolausWest nikolausWest nikolausWest approved these changes

@emilk emilk emilk left review comments

Assignees

@abey79 abey79

Labels
examples Issues relating to the Rerun examples
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Live streaming facial landmarks example
3 participants
@abey79 @emilk @nikolausWest