Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: geaxgx/depthai_blazepose
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: the-hive-lab/depthai_blazepose
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: integration
Choose a head ref
Able to merge. These branches can be automatically merged.
  • 16 commits
  • 19 files changed
  • 1 contributor

Commits on Jun 13, 2023

  1. Created new script for collecting dynamic gesture data from blazepose…

    … running live on an OAKD
    drpjm committed Jun 13, 2023
    Copy the full SHA
    913b0df View commit details
  2. Copy the full SHA
    c8744cf View commit details
  3. Created data_collection module

    drpjm committed Jun 13, 2023
    Copy the full SHA
    b61758d View commit details

Commits on Jun 14, 2023

  1. Copy the full SHA
    7a12bb7 View commit details

Commits on Jun 15, 2023

  1. Copy the full SHA
    0681434 View commit details
  2. Merge pull request #3 from the-hive-lab/2-data-capture-revision

    Data capture revision
    drpjm authored Jun 15, 2023

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    3a7a28b View commit details
  3. Copy the full SHA
    9cad6c6 View commit details

Commits on Jun 16, 2023

  1. Copy the full SHA
    37c5bb6 View commit details
  2. Copy the full SHA
    00ce530 View commit details
  3. Copy the full SHA
    204ea0b View commit details
  4. Copy the full SHA
    acb8e5d View commit details

Commits on Jun 20, 2023

  1. Completed the repo refactoring

    drpjm committed Jun 20, 2023
    Copy the full SHA
    2789fb0 View commit details
  2. Merge pull request #4 from the-hive-lab/1-repo-refactor

    Repository refactor
    drpjm authored Jun 20, 2023

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    43aac39 View commit details
  3. Merge pull request #5 from the-hive-lab/main

    Fixing incorrect merge
    drpjm authored Jun 20, 2023

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    ba2ea54 View commit details

Commits on Jun 22, 2023

  1. Copy the full SHA
    3a39d51 View commit details

Commits on May 8, 2024

  1. Copy the full SHA
    19388d5 View commit details
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
lib/
lib64/
bin/
etc/
share/
include/
60 changes: 17 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Blazepose tracking with DepthAI

This repo is a **fork** of the core Blazepose repo for the OAK-D cameras.
It is refactored to adhere to typicaly python packaging schemes and will have additional capabilities for use in the [HIVE Lab](http://thehivelab.xyz).

The refactoring has the following structure:

- `depthai_blazepose` - Python module forthe DepthAI based Blazepose model
- `depthai_blazepose.utils` - Utility modules used within the project
- `examples` - Reorganized examples from the original repository
- `data_collection` - This module maintains HIVE Lab data collection modules

-----

Running Google Mediapipe single body pose tracking models on [DepthAI](https://docs.luxonis.com/en/gen2/) hardware (OAK-1, OAK-D, ...).

The Blazepose landmark models available in this repository are the version "full", "lite" and "heavy" of mediapipe 0.8.6 (2021/07),
@@ -12,7 +24,6 @@ For the challenger Movenet on DepthAI, please visit : [depthai_movenet](https://

For an OpenVINO version of Blazepose, please visit : [openvino_blazepose](https://github.com/geaxgx/openvino_blazepose)


- [Blazepose tracking with DepthAI](#blazepose-tracking-with-depthai)
- [Architecture: Host mode vs Edge mode](#architecture-host-mode-vs-edge-mode)
- [Inferred 3D vs Measured 3D](#inferred-3d-vs-measured-3d)
@@ -60,19 +71,19 @@ The image below demonstrates the 3 modes of 3D visualization:

## Install


Install the python packages (depthai, opencv, open3d) with the following command:
To install this module, it is recommended that you create a [virutual environment](https://docs.python.org/3/library/venv.html) first.
Then run the following:

```
python3 -m pip install -r requirements.txt
pip install -e .
```

## Run

**Usage:**
**Demo Usage:**

```
-> python3 demo.py -h
-> python3 examples/demo.py -h
usage: demo.py [-h] [-e] [-i INPUT] [--pd_m PD_M] [--lm_m LM_M] [-xyz] [-c]
[--no_smoothing] [-f INTERNAL_FPS]
[--internal_frame_height INTERNAL_FRAME_HEIGHT] [-s] [-t]
@@ -247,43 +258,6 @@ from BlazeposeDepthaiEdge import BlazeposeDepthai

This way, you can replace the renderer from this repository and write and personalize your own renderer (for some projects, you may not even need a renderer).

The file ```demo.py``` is a representative example of how to use these classes:
```
from BlazeposeDepthaiEdge import BlazeposeDepthai
from BlazeposeRenderer import BlazeposeRenderer
# The argparse stuff has been removed to keep only the important code
tracker = BlazeposeDepthai(input_src=args.input,
pd_model=args.pd_m,
lm_model=args.lm_m,
smoothing=not args.no_smoothing,
xyz=args.xyz,
crop=args.crop,
internal_fps=args.internal_fps,
internal_frame_height=args.internal_frame_height,
force_detection=args.force_detection,
stats=args.stats,
trace=args.trace)
renderer = BlazeposeRenderer(
pose,
show_3d=args.show_3d,
output=args.output)
while True:
# Run blazepose on next frame
frame, body = tracker.next_frame()
if frame is None: break
# Draw 2d skeleton
frame = renderer.draw(frame, body)
key = renderer.waitKey(delay=1)
if key == 27 or key == ord('q'):
break
renderer.exit()
tracker.exit()
```

For more information on:
- the arguments of the tracker, please refer to the docstring of class `BlazeposeDepthai` or `BlazeposeDepthaiEdge` in `BlazeposeDepthai.py` or `BlazeposeDepthaiEdge.py`;
- the attributes of the 'body' element you can exploit in your program, please refer to the doctring of class `Body` in `mediapipe_utils.py`.
Empty file added data_collection/__init__.py
Empty file.
130 changes: 130 additions & 0 deletions data_collection/dynamic_gesture_collect.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
from depthai_blazepose.BlazeposeRenderer import BlazeposeRenderer
import argparse
import pickle
import pandas as pd
import numpy as np
import time
import datetime
from pathlib import Path

def main():
parser = argparse.ArgumentParser()
parser.add_argument('-e', '--edge', action="store_true",
help="Use Edge mode (postprocessing runs on the device)")
parser_tracker = parser.add_argument_group("Tracker arguments")
parser_tracker.add_argument('-i', '--input', type=str, default="rgb",
help="'rgb' or 'rgb_laconic' or path to video/image file to use as input (default=%(default)s)")
parser_tracker.add_argument("--pd_m", type=str,
help="Path to an .blob file for pose detection model")
parser_tracker.add_argument("--lm_m", type=str,
help="Landmark model ('full' or 'lite' or 'heavy') or path to an .blob file")
parser_tracker.add_argument('-xyz', '--xyz', action="store_true", default=True,
help="Get (x,y,z) coords of reference body keypoint in camera coord system (only for compatible devices)")
parser_tracker.add_argument('-c', '--crop', action="store_true",
help="Center crop frames to a square shape before feeding pose detection model")
parser_tracker.add_argument('--no_smoothing', action="store_true",
help="Disable smoothing filter")
parser_tracker.add_argument('-f', '--internal_fps', type=int,
help="Fps of internal color camera. Too high value lower NN fps (default= depends on the model)")
parser_tracker.add_argument('--internal_frame_height', type=int, default=640,
help="Internal color camera frame height in pixels (default=%(default)i)")
parser_tracker.add_argument('-s', '--stats', action="store_true",
help="Print some statistics at exit")
parser_tracker.add_argument('-t', '--trace', action="store_true",
help="Print some debug messages")
parser_tracker.add_argument('--force_detection', action="store_true",
help="Force person detection on every frame (never use landmarks from previous frame to determine ROI)")

parser_renderer = parser.add_argument_group("Renderer arguments")
parser_renderer.add_argument('-3', '--show_3d', choices=[None, "image", "world", "mixed"], default=None,
help="Display skeleton in 3d in a separate window. See README for description.")


parser_renderer.add_argument("-o","--output", type=str, default="data", help="Name of the output data files")


args = parser.parse_args()

if args.edge:
from depthai_blazepose.BlazeposeDepthaiEdge import BlazeposeDepthai
else:
from depthai_blazepose.BlazeposeDepthai import BlazeposeDepthai
tracker = BlazeposeDepthai(input_src=args.input,
pd_model=args.pd_m,
lm_model=args.lm_m,
smoothing=not args.no_smoothing,
xyz=args.xyz,
crop=args.crop,
internal_fps=args.internal_fps,
internal_frame_height=args.internal_frame_height,
force_detection=args.force_detection,
stats=True,
trace=args.trace)

landmarks_world_list = []
visibility_list = []
presence_list = []
xyzs = []

now = datetime.datetime.now()
month = str(now.month).zfill(2)
day = str(now.day).zfill(2)
hour = str(now.hour).zfill(2)
minute = str(now.minute).zfill(2)
secs = str(now.second).zfill(2)
timestamp = f"{now.year}{month}{day}_{hour}{minute}{secs}"
data_dir = Path(__file__).cwd().resolve().joinpath(args.output + "_" + timestamp)
print(f"Data directory: {data_dir}")
data_dir.mkdir()

renderer = BlazeposeRenderer(
tracker,
show_3d=args.show_3d,
output=str(data_dir.joinpath(args.output+".mp4")))

time.sleep(2)

while True:
# Run blazepose on next frame
frame, body = tracker.next_frame()
if frame is not None and body is not None:

# Grab the relevant data from the Body data structure
lms_world = body.landmarks_world.flatten()
landmarks_world_list.append(lms_world)
visibility_list.append(body.visibility)
presence_list.append(body.presence)
xyzs.append(body.xyz)

# Draw 2d skeleton
frame = renderer.draw(frame, body)
key = renderer.waitKey(delay=1)
if key == 27 or key == ord('q'):
break

# Make the csv file for this recording
landmark_col_labels = [str(i) for i in range(np.array(landmarks_world_list).shape[1])]
dynamic_gesture_df = pd.DataFrame(landmarks_world_list, columns=landmark_col_labels)
gesture_out_path = data_dir.joinpath(args.output+"_landmarks.csv")
dynamic_gesture_df.to_csv(gesture_out_path)

# Probability of keypoints that exist and are *not* occluded
gesture_vis_col_labels = [str(i) for i in range(np.array(visibility_list).shape[1])]
gesture_visibility_df = pd.DataFrame(visibility_list, columns=gesture_vis_col_labels)
vis_out_path = data_dir.joinpath(args.output+"_visibility.csv")
gesture_visibility_df.to_csv(vis_out_path)

# Probability of keypoints that exist in the frame
gesture_pres_col_labels = [str(i) for i in range(np.array(presence_list).shape[1])]
gesture_presence_df = pd.DataFrame(presence_list, columns=gesture_pres_col_labels)
presence_out_path = data_dir.joinpath(args.output + "_presence.csv")
gesture_presence_df.to_csv(presence_out_path)

# Store the human's center xyz estimate
xyz_labels = ['x','y','z']
xyz_df = pd.DataFrame(xyzs, columns=xyz_labels)
xyz_out_path = data_dir.joinpath(args.output + "_xyz.csv")
xyz_df.to_csv(xyz_out_path)

renderer.exit()
tracker.exit()
40 changes: 40 additions & 0 deletions data_collection/video_to_images.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import cv2
import argparse
import logging
import os

def extract_frames(video_path, img_path, img_type='jpg'):
video_cap = cv2.VideoCapture(video_path)
video_name = video_path.split('.')[0]

frame_idx = 0

while True:
ret, frame = video_cap.read()

if not ret:
break

img_name = video_name + "_" + str(frame_idx).zfill(3) + ".jpg"
logging.info(f"Writing: {img_name}")
status = cv2.imwrite(f'{img_name}', frame)
logging.info(f'Image write status: {status}')
frame_idx += 1

video_cap.release()

def main():
logging.basicConfig(level=logging.INFO)
parser = argparse.ArgumentParser()
parser.add_argument('-i', '--input', help='Path to the video file', type=str, default='')
parser.add_argument('-o', '--output', help='Path to the output files', type=str, default='imgs')
args = parser.parse_args()

logging.info(f"Opening video: {args.input}")
logging.info(f"Writing images to: {args.output}")

extract_frames(args.input, args.output)
cv2.destroyAllWindows()

if __name__ == '__main__':
main()
11 changes: 6 additions & 5 deletions BlazeposeDepthai.py → depthai_blazepose/BlazeposeDepthai.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
import numpy as np
import mediapipe_utils as mpu
import cv2
from pathlib import Path
from FPS import FPS, now
from math import sin, cos
import depthai as dai
import time, sys

SCRIPT_DIR = Path(__file__).resolve().parent
import depthai_blazepose.utils.mediapipe_utils as mpu
from depthai_blazepose.utils.FPS import FPS, now

# SCRIPT_DIR = Path(__file__).resolve().parent
SCRIPT_DIR = Path.cwd()
POSE_DETECTION_MODEL = str(SCRIPT_DIR / "models/pose_detection_sh4.blob")
LANDMARK_MODEL_FULL = str(SCRIPT_DIR / "models/pose_landmark_full_sh4.blob")
LANDMARK_MODEL_HEAVY = str(SCRIPT_DIR / "models/pose_landmark_heavy_sh4.blob")
LANDMARK_MODEL_LITE = str(SCRIPT_DIR / "models/pose_landmark_lite_sh4.blob")


def to_planar(arr: np.ndarray, shape: tuple) -> np.ndarray:
return cv2.resize(arr, shape).transpose(2,0,1).flatten()

@@ -67,7 +68,7 @@ def __init__(self, input_src="rgb",
trace=False,
force_detection=False
):

self.pd_model = pd_model if pd_model else POSE_DETECTION_MODEL
print(f"Pose detection blob file : {self.pd_model}")
self.rect_transf_scale = 1.25
Original file line number Diff line number Diff line change
@@ -1,23 +1,24 @@
import numpy as np
import cv2
from numpy.core.fromnumeric import trace
import mediapipe_utils as mpu
from pathlib import Path
from FPS import FPS, now
import depthai as dai
import marshal
import sys
from string import Template
from math import sin, cos

SCRIPT_DIR = Path(__file__).resolve().parent
from numpy.core.fromnumeric import trace
import depthai_blazepose.utils.mediapipe_utils as mpu
from depthai_blazepose.utils.FPS import FPS, now

SCRIPT_DIR = Path.cwd()
POSE_DETECTION_MODEL = str(SCRIPT_DIR / "models/pose_detection_sh4.blob")
LANDMARK_MODEL_FULL = str(SCRIPT_DIR / "models/pose_landmark_full_sh4.blob")
LANDMARK_MODEL_HEAVY = str(SCRIPT_DIR / "models/pose_landmark_heavy_sh4.blob")
LANDMARK_MODEL_LITE = str(SCRIPT_DIR / "models/pose_landmark_lite_sh4.blob")
DETECTION_POSTPROCESSING_MODEL = str(SCRIPT_DIR / "custom_models/DetectionBestCandidate_sh1.blob")
DIVIDE_BY_255_MODEL = str(SCRIPT_DIR / "custom_models/DivideBy255_sh1.blob")
TEMPLATE_MANAGER_SCRIPT = str(SCRIPT_DIR / "template_manager_script.py")
TEMPLATE_MANAGER_SCRIPT = str(SCRIPT_DIR / "scripts/template_manager_script.py")


def to_planar(arr: np.ndarray, shape: tuple) -> np.ndarray:
13 changes: 4 additions & 9 deletions BlazeposeRenderer.py → depthai_blazepose/BlazeposeRenderer.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
import cv2
import numpy as np
from o3d_utils import Visu3D
import mediapipe_utils as mpu


from depthai_blazepose.utils.o3d_utils import Visu3D
import depthai_blazepose.utils.mediapipe_utils as mpu

# LINE_BODY and COLORS_BODY are used when drawing the skeleton in 3D.
rgb = {"right":(0,1,0), "left":(1,0,0), "middle":(1,1,0)}
@@ -21,9 +19,6 @@
"right","right","right","left","left","left"]
COLORS_BODY = [rgb[x] for x in COLORS_BODY]




class BlazeposeRenderer:
def __init__(self,
tracker,
@@ -36,9 +31,9 @@ def __init__(self,

# Rendering flags
self.show_rot_rect = False
self.show_landmarks = True
self.show_landmarks = False
self.show_score = False
self.show_fps = True
self.show_fps = False

self.show_xyz_zone = self.show_xyz = self.tracker.xyz

Empty file added depthai_blazepose/__init__.py
Empty file.
File renamed without changes.
Empty file.
File renamed without changes.
File renamed without changes.
6 changes: 3 additions & 3 deletions demo.py → examples/demo.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env python3

from BlazeposeRenderer import BlazeposeRenderer
from depthai_blazepose.BlazeposeRenderer import BlazeposeRenderer
import argparse

parser = argparse.ArgumentParser()
@@ -40,9 +40,9 @@
args = parser.parse_args()

if args.edge:
from BlazeposeDepthaiEdge import BlazeposeDepthai
from depthai_blazepose.BlazeposeDepthaiEdge import BlazeposeDepthai
else:
from BlazeposeDepthai import BlazeposeDepthai
from depthai_blazepose.BlazeposeDepthai import BlazeposeDepthai
tracker = BlazeposeDepthai(input_src=args.input,
pd_model=args.pd_m,
lm_model=args.lm_m,
Loading