feat(pose_estimation): support multiprocessing videos_to_poses #130
Conversation
| @@ -15,7 +15,7 @@ def load_video_frames(cap: cv2.VideoCapture): | |||
| cap.release() | |||
|
|
|||
|
|
|||
| def pose_video(input_path: str, output_path: str, format: str, additional_config: dict): | |||
| def pose_video(input_path: str, output_path: str, format: str, additional_config: dict = {'model_complexity': 1}, progress: bool = True): | |||
There was a problem hiding this comment.
I think additional_config should default to None. it knows how to handle it
There was a problem hiding this comment.
We had this line here, which would raise an error:
Do we still want by default {'model_complexity': 1}? I do not know the default behavior from Mediapipe and how much difference it brings.
There was a problem hiding this comment.
i see, in that case, keep it
| @@ -79,6 +111,21 @@ def get_corresponding_pose_path(video_path: Path, keep_video_suffixes: bool = Fa | |||
| return video_path.with_suffix(".pose") | |||
|
|
|||
|
|
|||
| def process_video(vid_path: Path, keep_video_suffixes: bool, pose_format: str, additional_config: dict) -> bool: | |||
| print(f'Estimating {vid_path} on CPU {psutil.Process().cpu_num()}') | |||
There was a problem hiding this comment.
We should probably switch to logging at some point rather than print statements, but apparently combining that with multiprocessing can be tricky.
https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes seems easiest.
| try: | ||
| pose_path = get_corresponding_pose_path(video_path=vid_path, keep_video_suffixes=keep_video_suffixes) | ||
| if pose_path.is_file(): | ||
| print(f"Skipping {vid_path}, corresponding .pose file already created.") |
There was a problem hiding this comment.
Here's an example where logging.debug might be good, as this could potentially output thousands of messages
J22Melody
changed the title
|
can be merged if it looks good to you |
|
Okay, the current script has a memory leak - the memory usage goes to > 100GB after running 64 processes on thousands of large videos (BOBSL videos, 30 to 45 minutes) for a night. Let me know if you have any clue about it. |
|
@J22Melody last time I was debugging memory usage I used memray: https://pypi.org/project/memray/ |
|
That was to do with DGS corpus/ the datasets library, I discovered in that case the tfds was expanding the entire video to frames and holding them in memory, which got really big really fast. I wonder if something similar is going on, that we've not tested with long videos and the pose estimation struggles with really long sequences of frames. |
|
Following that suspicion and making some calculations...
|
|
Gave it a quick test on Google Colab, found a bug. #135 |
|
OK, after a bit of messing around on Google Colab with memray (NB: you have to use --follow-fork or it won't work right with multiprocessing), I have some results. For two 5-minute videos, AKzh_rKNqzo.mp4 and AKzh_rKNqzo_copy.mp4 (I just copied the same file) from YouTube-ASL, it looks like it uses about 600 MB of memory at peak usage. As expected, the peak of that is in process_holistic, specifically this line. memray_flamegraphs_on_AKzh_rKNqzo_with_2_workers.zip https://colab.research.google.com/drive/10vMrhoC0zVXLMzhJHB9kUi_0LqBvFhz7?usp=sharing is the quick notebook I threw together. |
|
Thanks for the investigation. Yes, every video is like a few hundred MB and if you have processed a few hundred videos without releasing them, then the memory explodes. I guess the problem is with multiprocessing/ |
|
I looked over the code, I don't see why it would leak memory. We are duplicating the memory on the line @cleong110 refered to though. I think the first thing to investigate then is: Run pose estimation with a single process ( So in conclusion, if you run 64 processes, and each is 1GB, but we also duplicate the memory, it makes sense you ran out of memory. Starting solution: Option 2 here |
@cleong110 @AmitMY