Capturing a static image from the local MediaStream

[oliverlukesch]

Hi Iñaki, first I'd like to thank you for this great plugin. I'm currently building a Cordova-based video-chat solution using OpenTok.js, Crosswalk and your plugin and most of the things are working out great. I'll happily share the project once it's done.

Now to the issue at hand: I need to be able to capture static images from the MediaStream of the local client before the stream gets sent to the remote client and therefore gets compressed and loses quality. The stream needs to continue while capturing the static images.

The common WebRTC solution is to draw the content of the <video> element onto a <canvas> element and use <canvas>.toDataURL('image/png') to get the static image. However, due to the specifics of iosrtc, this is obviously not possible.

My guess is that this requires some sort of native implementation. While my Swift/Objective-C skills are limited, I'll happily give it a shot and create a pull-request if you can point me into the right direction. Or maybe you already have a solution at hand.

Many thanks!

[ibc]

There is no easy way to do that, and given that this plugin mimics the official WebRTC API I don't want a custom API to capture images from the video. The way to go should be adding <canvas> support to the plugin with all the Canvas API, but that would be an horrible work.

I'm sorry, I cannot point you in the right direction :)

[oliverlukesch]

Thanks for the update, I understand your approach. And yes, implementing the <canvas> API would be quite a different kind of beast :) I'll start writing on a patch for this functionality I can use in my own project.

[yocontra]

@ibc If you were to add an internal API to grab the image, I can work on integrating it into canvas so you can use the same code you would in a browser

[ibc]

@contra I'm so sorry, but currently I cannot get the time to work on that.

[yocontra]

@ibc Can you leave the issue open so somebody (ie me) can work on it?

[ibc]

Sure.

[derMani]

@oliverlukesch I had to do the same in my application and I've extended the plugin on JS and Swift side.

What you can do: Take a snapshot of the UIView element. You need the ID of the mediastreamrenderer

In cordova-plugin.iosrtc.js do (for example) the following trick

var keys = Object.keys(mediaStreamRenderers);
id = mediaStreamRenderers[keys[0]].id;

The ID contains your specific view element. It's mapped to the Swift dictionary

pluginMediaStreamRenderers

which contains the specific UIView.

Do a local snapshot in Swift with this code:

// Create the UIImage
UIGraphicsBeginImageContextWithOptions(view.bounds.size, true, 0.0)
view.drawViewHierarchyInRect(view.bounds, afterScreenUpdates: true)

view.layer.renderInContext(UIGraphicsGetCurrentContext()!)
let image = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
var imageData = UIImagePNGRepresentation(image)

and return it to the function as base64.

This should do the trick

Regards (Grüße nach Berlin)
Rolf

[oliverlukesch]

Hey Rolf, that's great, many thanks for sharing! Think I'll have time for implementing this solution next week, will keep you updated.

Grüße nach Osnabrück :)

[1N50MN14]

@derMani @oliverlukesch Awesome!

[gefangenimnetz]

Guys, this is exactly what I am missing. The ability to write the current stream to a canvas element. I want to use the camera as motion detector (see this super simple and awesome example: https://github.com/ReallyGood/js-motion-detection). If you manage to get this to work that would be awesome!

Grüße aus Stuttgart
(Seems like the Germans are taking over – again)

[ibc]

@gefangenimnetz Pull Requests are welcome.

[AntonioCurci]

@derMani I would be interested to capture the image, but the only "limit" I see on your method is that the resolution of the image will depend on the size of the UIView. So this means that you get a 320 or 640 HxW image... Am I wrong?

[1N50MN14]

@SirSeymour The resolution will depend on what you decide to use with the plugin, from there you can use the base64 image data to resize, crop etc.

Anyone on this thread actually working on a PR? @oliverlukesch are you intending to send in a PR?

[AntonioCurci]

@1N50MN14 if the local video UIView is 240x240, you'll get a 240x240 image. Don't you? I'm saying if I want to capture the full resolution image from local camera, with the method stated above I can't (ex. the video source is set a 1280x960 but the UIView is at 640x320, the image I'll get is the 640x320)

[1N50MN14]

@SirSeymour You're actually right this won't serve say as a 12M full image resolution capture, but it's good for stuff such as "chose your profile photo"

[1N50MN14]

Anyway I just need to know if anyone on this thread is working on a PR so I could decide whether to write it myself or not as "implementing the solution" is not clear if translates to submitting a PR

[derMani]

At least for me, I am not working on a PR.

Please note that the Screenshot-thing is kind of a heavy task (at least, that's what the documentation says)

For anything live / canvas related, it might be better to utilize somehow a AVCaptureConnection and AVCaptureStillImageOutput on the video connection.

[1N50MN14]

Ok cool thanks @derMani I'll squeeze this on my to-do list next Friday and submit a PR @SirSeymour I'll try to generate a full resolution image I'll have to see how it'll work out in relation to the constrains properties the plugin receives

[oliverlukesch]

@1N50MN14 Nope, haven't had any time to touch this issue so far. Great stuff you're taking a look!

[ikostic]

I am currently working with iosRTC plugin and i can test PR when available.

[ikostic]

@1N50MN14 Hi, i am just wondering if you had any luck creating the solution. DId you made any progress? Many thanks.

[1N50MN14]

@ikostic Apologies for not updating this, unfortunately I lost interest in this project / plugin, I'll no longer be participating here and will be focusing on my own implementation instead; I have some catching up to do but once I get there I can revisit the repo and send a PR if no one has implemented this until then... sorry about that.

[ikostic]

@1N50MN14 No problem, i understand. I will try to find some solution.

[scottmahr]

Is there any chance that someone will make this possible with iosrtc?

[1N50MN14]

@scottmahr I'm sending a PR for this one on Friday

[scottmahr]

Wow, that is amazing timing. In your implementation what sort of format will the image be coming out from the library? Right now I use canvas and canvas.toDataURL to pull a jpeg of the current video output.

Let me know if you want help testing to see if it works on different devices.

Thanks!

Scott

[1N50MN14]

@scottmahr Similar to toDataURL you'll receive base64 data back, the only caveat is that you're receiving back a "screenshot" having whatever dimensions the device has, not an actual frame from the video

[SomewhatCloudy]

You might have more joy with it than me! I am sure the theory is sound. Let me know if you have any success on it.

[danielearwicker]

Unfortunately after I added the plugin to my project it won't build (compiler error, a missing type). But if I figure out why that is and get it working I'll let you know!

[cometta]

anyone has already got this working?

[ibc]

No.

[hthetiot]

Related #253 (comment)

[hthetiot]

Done here:

• [Beta] update libwebrtc without iosrtc custom patch to ~M54 #385

Example:

var canvasEl;
function TestMediaRenderCatpure(videoEl) {
  canvasEl = document.createElement('canvas');
  var ctx = canvasEl.getContext("2d");
  canvasEl.width = "100";
  canvasEl.height = "100";
  canvasEl.style.position = 'absolute';
  canvasEl.style.left = 0;
  canvasEl.style.bottom = 0;
  appContainer.appendChild(canvasEl);
  var image = new Image();
  image.onload = function() {
    ctx.drawImage(image, 0, 0);
  };
  videoEl.render.save(function (data) {
    image.src = "data:image/jpg;base64," + data;
  });
} 

Test Source:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/task/libwebrtc-update/extra/renderer-and-libwebrtc-tests.js#L167

Implementation:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/task/libwebrtc-update/src/PluginMediaStreamRenderer.swift#L242
• https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/task/libwebrtc-update/js/MediaStreamRenderer.js#L114

[alexp25]

Can the video element be hidden? I only want to show the frames into the canvas. Is this possible?

[hthetiot]

6.0.15 support CanvasRenderingContext2D.drawImage on VideoElement with iosrtc MediaStream

03e4a0d

// Apply CanvasRenderingContext2D.drawImage monkey patch
var drawImage = CanvasRenderingContext2D.prototype.drawImage;
CanvasRenderingContext2D.prototype.drawImage = function (arg) {
	var args = Array.prototype.slice.call(arguments);
	var context = this;
	if (arg instanceof HTMLVideoElement && arg.render) {
		arg.render.save(function (data) {
		    var img = new window.Image();
		    img.addEventListener("load", function () {
		    	args.splice(0, 1, img);
		        drawImage.apply(context, args);
		    });
		    img.setAttribute("src", "data:image/jpg;base64," + data);
	  	});
	} else {
		return drawImage.apply(context, args);
	}

[gabrielenosso]

I just tried that but in Safari it gives me Type Error on "drawImage.apply(context, args);".

Any idea why?
In the debugger args seems correct: it's an array and the element 0 is a string "data:image...."

[hthetiot]

@gefangenimnetz This is not made for Safari but for cordova, the img is not a string but an image tag loaded, also I have no idea if Safari support draw on video.

If you want really help with this issue please test with Cordova, you need to pass the proper argument (videoEl, 0, 0) but since you did not provided any details on what you exactly called it's impossible to know what to expect.

This script above is not made for Safari,

[gabrielenosso]

I am using Cordova, debugging the WKWebView with Safari.
Sorry, my message was not clear.

This is my code:

  if (navigator.mediaDevices.getUserMedia) {
      navigator.mediaDevices.getUserMedia({ video: true, audio: false })
        .then((stream) => {
          const video = this.videoRef.current; // This is the <video> element, taken from the DOM via React
          video.onloadedmetadata = (event) => {
            this.canvasRef.current.width = event.srcElement.videoWidth;
            this.canvasRef.current.height = event.srcElement.videoHeight;
          };
          video.srcObject = stream;

          const onFrame = () => {
            window.requestAnimationFrame(onFrame);
            const canvasContext = this.canvasRef.current.getContext('2d');
            canvasContext.drawImage(video, 0, 0);
          };
          onFrame();
        })
        .catch((error) => {
          console.log(error);
        });
    }

[hthetiot]

@gabrielenosso looking into it one moment.

[gabrielenosso]

To complete the information: I am using version 6.0.15.

Changing the line:
drawImage.apply(context, args);
To:
context.drawImage(img, 0 0);
seems to work fine.

EDIT:
I think I found the issue:

In your comment the line is:
args.splice(0, 1, img); // this works fine

But in the repository code the line is:
args.splice(0, 1, img.src); // error

[hthetiot]

@gabrielenosso First, this issue is originaly closed.
Second, yes, I did pushed the fix on master already of img vs img.src.

Please comment on #582 not this issue.
Thank you.

Note: Regarding the background thread warning, I have a fix but it also create another error and also requestAnimationFrame will not be efficient with the current implementation.

[hthetiot]

#582 will be closed on 6.0.16 with the img fix. its on master already.

[hthetiot]

Note: 6.0.21 fix memory leak on Capture video.