This is an experiment in incorporating transformers.js into ml5.js for use with p5.js. Done by @gohai, @chanel1130, @Lisa-HuangZijin and @Ricci-Liu in Summer of 2024.
We're making use of the detr-resnet-50-panoptic image segmentation model here, which we found to provide acceptable performance in the browser (albeit not quite fast enough to run it continuously from a webcam).
- Basic
- Webcam Collage by Chanel
- Webcam Drag & Drop Storytelling by Lisa
- Webcam Grid Storytelling by Lisa
- Webcam Parallax by Ruiqi
In the head section of your index.html, add this line after the line that loads ml5.js:
<script src="https://unpkg.com/[email protected]/src/ml5-extra-imagesegmentation.js"></script>
Load the image segmenter like so:
let segmentation;
function preload() {
segmentation = ml5.imageSegmentation({ feature_extractor_size: 256 });
}
Omitting the feature_extractor_size option will improve the accuracy over speed.
To start a detection, pass an image or video to the detect() method, together with the name of a callback function.
segmentation.detect(img, gotResults);
Once the segmentation has finished, the callback function is called with an array of segments as argument.
function gotResults(results) {
for (let i=0; i < results.length; i++) {
result = results[i];
console.log(result);
}
}
Each result has the following properties: label, score as well as a mask image. The mask is ready to be used with p5's mask() method.
Many thanks to @xenova - first and foremost for transformers.js, but also for kindly providing feedback during the research for this project!