DEPRECATED Repository

Use YouTube-Motion-Tracking

YouTube pose dream

Some examples from this Video source

For more examples and the current progress see #visualizeYouTube playlist

The logo was designed by a 3 year old girl.

YouTube pose dream is a Chrome AI extension to visualize videos. (Soon also available for Firefox.)

When you're old enough, you may remember Winamp. There is the possibility to visualize music.
However, this extension makes similar with videos. The human pose in the video is used as input for the visualization.
In the process, the existing video is transformed into a psychedelic work of art. But this depends on the video you are watching.

The goal of this project is for anyone, with or without programming experience, to be able to visualize YouTube.
In addition, animations should be interchangeable between users.

Pose estimation is done with TensorFlow.js
Particle animation is done with Proton

This project is a Google Chrome browser extension. That's why you need this browser to try it out.

Table of Contents

1. How it works
2. Features
   1. Fun with lines
      1. Skeleton
      2. Skeleton three times
      3. Skeleton five times
      4. Puppets player
      5. Spider web
   2. Replace head with image
      1. Cat
      2. Smiley
      3. Sun
      4. Monkey
      5. Anonymous
   3. Show particle animation
      1. Hand power balls
      2. Two head balls
      3. Right hand line
      4. Nose gravity
      5. Nose supernova
      6. Hands track from border
      7. Upper body glow
      8. Glow painting
      9. Particle painting
      10. Particle painting with random drift
      11. Comet thrower
      12. Body glow
      13. Burning Man
3. Installation
   1. Download this repository
   2. Open Chrome extensions
   3. Enable developer mode
   4. Load extension
   5. Check extension installation
   6. Check Chrome settings
   7. Pin the extension
4. Usage
5. Uninstall
6. Build
7. Further development
8. License

How it works.

The Chrome extension becomes active, when you watch a YouTube video.

The extension use MoveNet as model.
Visual representation of the keypoints:

Image from tensorflow.js pose-detection model
We use these keypoints to place animations or static content in the video.

Features

After the installation you can select the following visualizations via the extension menu.
You can change the visualization while the video is playing.

Or you can change the visualization directly in the video player.
This also works in full screen mode.
With the new button the popup can be displayed or disabled.

The video Fatboy Slim ft. Bootsy Collins - Weapon Of Choice Official 4k Video was used for all screenshots.

Fun with lines

Skeleton

This is TensorFlow.js in action.
MoveNet is used for the detection.
Use: all keypoints

Skeleton 3 times

Use: all keypoints

Skeleton 5 times

Use: all keypoints

Puppets player

Use: left_wrist, right_wrist, left_shoulder, right_shoulder, left_ankle, right_ankle, nose

Spider web

Use: all keypoints

Replace head with image

The eyes are used as a reference for displaying the image.
All use left_eye, right_eye

Cat

Smiley

Sun

Monkey

Anonymous

Show particle animation

Hand power balls

Every hand gets one energy ball.
Use: left_wrist, right_wrist

Two head balls

Two energy balls circle around the head.
Use: nose

Right hand line

Only the right hand gets a particle animation.
Use: right_wrist

Nose gravity

A gravitational field is created around the nose.
Use: nose

Nose supernova

Supernovas are constantly exploding on the nose.
Use: nose

Hands track from border

Use: left_wrist, right_wrist
The yellow ray is connected to the right hand.
The red ray is connected to the left hand.
If the hands are too far away from the edge, the connection is interrupted.

Upper body glow

Use: left_wrist, right_wrist, left_elbow, right_elbow, left_shoulder, right_shoulder

Glow painting

Use: left_wrist, right_wrist

Particle painting

Use: left_wrist, right_wrist

Particle painting with random drift

Use: left_wrist, right_wrist

Comet thrower

Use: left_wrist, right_wrist

Body glow

Use: left_wrist, right_wrist, left_elbow, right_elbow, left_shoulder, right_shoulder, left_ankle, right_ankle, left_hip, right_hip

Burning Man

Use: left_wrist, right_wrist, left_elbow, right_elbow, 2*left_shoulder, 2*right_shoulder, 2*left_ankle, 2*right_ankle, left_hip, right_hip, left_ear, right_ear, nose

Installation

Download this repository

git clone https://github.com/deep2universe/ChromePoseDream.git

Open Chrome extensions

Open this URL

chrome://extensions

Enable developer mode

In the upper right corner you have to activate the developer mode.

Load extension

Click this button to load the extension.
Then select the dist Folder from this repository.

Check extension installation

You should now see the following entry.

Check Chrome settings

Go to this URL

chrome://settings/accessibility

In the System settings make sure to enable this:

Pin the extension

In the upper right corner click this to pin the extension.

Usage

Open YouTube

Watch a video and have fun.

Uninstall

Just go to the chrome://extensions page and disable or delete the extension.
This will be possible later via the popup/settings page.

Build

All the code is in the src directory.
You need PARCEL for the build.

# install PARCEL
npm install -g parcel-bundler

# install dependencies
npm install

# build dist folder
npm run build

Further development

The following is still on the TODO list:

• Support for other video platforms (e.g. Vimeo)
• Support for multi-pose detection.
• Internationalization
• Export of the pose detection keypoints for other applications.
• Use pose detection keypoints to feed GAN or VAE model and display result in webpage.
• It should be possible to deactivate the extension. Currently, you must do this in chrome://extensions
• Fill settings page with content (default animation, disable extension, language settings etc.)
• Publish to Chrome extension store.
• Clean up code.
• Add support for three.proton to enable 3D particles.
• Check content.js video event listener, - sometimes you have to reload the page to start pose dream.
• Add more particle animations.
• Create Firefox extension and publish to Firefox Browser ADD-ONS

This is an example of how this project setup could be extended in the future.
Currently, the focus of the project is on exploring the existing architectural possibilities. With the help of artificial intelligence, many other enhancements are feasible. For example, the acoustic content of the video is not viewed and could also be used.

If you want to support, pull requests are always welcome.
If you find any bugs please report them.

License

Apache License 2.0