Semantic way to declare a video as an image

[mounirlamouri]

Videos are often used as animated images and design elements on the Web on Desktop. On Mobile, websites will provide GIF or use <canvas> instead because of autoplay restrictions. This is wasting more bandwidth for the users who need it the most and use more energy on devices able to hardware decode the most popular video formats.

I was wondering if it would make sense to have a declarative way to mark a video as an image so it can be used for GIF and design elements (like Uber, Apple or NYT do on their websites). Having an attribute for this would help feature detection and allow the UA to have different behaviour like the controls UI or the Media Session integration. Some browsers like Safari on iPhone could use the information to know that the video can be played inline for example.

An alternative to this would be to make <video autoplay muted> play automatically in mobile browsers and block muted change unless it has a user gesture. It has a few downsides: not semantically defined and slightly magic behaviour but has the benefit of being simple.

Chrome would be interested to push something to the spec, whether a new attribute or make a different muted behaviour allowed by the spec. It would be great to hear from other vendors on this :)

[tabatkins]

We already have the right semantic - <img src=video.mp4>. Just need to support it. ^_^

[mounirlamouri]

The main concern with this is that we might need to expose a few events and some APIs (like loop, play, pause, seek...) which might end up re-defininging most of what HTMLMediaElement already does.

[tabatkins]

So add a superclass and move some methods to there?

[domenic]

I like the muted idea.

[domenic]

Although that will still not meet the needs of web apps like vine.co, for which sound autoplay is a crucial part of the experience. Presumably this will force them into hacks involving using web audio to autoplay their sound.

[tigt]

Hanging this behavior on <video> feels to me like it would be less trouble in the long run:

• Possible new attributes would have to be included on both <img> and <video>.
• Would this work with <picture> and srcset? Would that work with media and sizes to offer more functionality than <video> currently does?
• Would this work with CSS images? (On the other hand, people are going to use video backgrounds anyway, so might as well explicitly label them as backgrounds for better download prioritization/possible runtime optimizations. Maybe that should be a separate thing, like video().)
• Would this work with SVG's <image>?
• This appears to overload element behavior like <object type> and <input type>, which I think Hixie cited awhile ago as being a pain in the butt.
• There will be an annoying gap for a while where technology built on the assumption that <img> won't contain video files won't work:
  • "Don't play video or I'll get a seizure" settings in extensions/assistive technology.
  • Context menus to Play/Pause, Save [Video? Image?] As..., and so on.
  • Sites that would prefer not to display moving images, having already blacklisted <video> and animated GIFs.
• Lookahead preparsers in current browsers that are aggressive with <img> will make spurious requests for files they can't use.
• How will the request headers predict they're dealing with a video file? Particularly Accept.
• How would this interact with <link rel="preload">'s as attribute?
• How would this work with Resource Fragments?

I think <img src="gif.mp4"> is really elegant, but the <img> tag has so much intertwingled with it and with many assumptions over the years, that something like a new attribute on <video> or browser heuristics (autoplay loop muted as mentioned, or small Content-Length server responses, etc.) would prove to be simpler over time.

[Yay295]

Alternatively, you could use an image format everyone forgot about a week after it was introduced: WebP. Of course, it's currently only supported on Chrome and Opera, but it IS supported on Chrome and Opera, unlike the current proposals here.

[foolip]

Although that will still not meet the needs of web apps like vine.co, for which sound autoplay is a crucial part of the experience. Presumably this will force them into hacks involving using web audio to autoplay their sound.

I think we should treat autoplay with sound as a separate concern, it's going to be a lot more challenging to get that behavior in all browsers than this smaller of step of allowing autoplay if muted.

The simplest thing one could do here, I think, is to allow autoplay if there is no audio track, and have a new attribute that causes audio tracks to be ignored. I wouldn't be possible to unmute.

However, that fails to handle media resources where there is an audio track, but playing it muted is still meaningful. This is now fairly common in ads and I see this behavior on Twitter too I think.

Something around the muted IDL attribute looks fairly promising to me. If muted is true when play() is called without a user gesture or when the autoplaying steps are reached, playback would be allowed. Setting muted to true while playing would do nothing if there is no user gesture. The risk of both these changes can be measured in advance of attempting to make the change. One might also consider a promise-returning unmute() that can be rejected if it's too subtle to have a setter that does nothing. Would make feature detection slightly easier, too.

Edit: To be clear, the setter would still have to do nothing, unmute() would strictly speaking be redundant.

[domenic]

I think we should treat autoplay with sound as a separate concern, it's going to be a lot more challenging to get that behavior in all browsers than this smaller of step of allowing autoplay if muted.

I'd like to push back on this a little bit. In the end it's up to implementers what they're willing to do, but I think we shouldn't sink too much time into overdesigning this feature in service of a use case which is unclear.

To me, the end goal should be to allow the <video> element to do all the things you can already do with the platform (i.e., moving images and audio both autoplaying). Today you can do this with canvas (or other hacks, like manually seeking every 16 ms) + web audio. I see no reason <video> should be restricted from doing these things; that just forces the other hacks to come into play, at the cost of performance and battery life.

So it's not clear to me what the use case of all the design work you outline is. It doesn't give authors any new capabilities that they can't already get through hacks, and it doesn't make things significantly easier to use, since the hacks are necessary if you want sound anyway. I guess you could argue that we want to make it easier for authors to play videos without sound easily, but I have serious doubts this is something any authors are asking for. The (in my estimation) very small number of authors for which this is appropriate can just use hacks, and don't need new design work, spec surface area, and implementation work done for them.

It's also noteworthy that Firefox mobile already autoplays videos (including audio) by default.

[mounirlamouri]

To me, the end goal should be to allow the

You are assuming that all mobile browsers allow autoplay for Web Audio which isn't true for Safari iOS.

It wouldn't be surprising if other browsers like Chrome Android were to add the same kind of restrictions.

It's also noteworthy that Firefox mobile already autoplays videos (including audio) by default.

Firefox Android has a flag to disable autoplay which is bound to a UI toggle. It is really surprising that autoplay isn't disabled by default for them. Hopefully someone at Mozilla can give some context.

[kornelski]

I support looped autoplaying of video files with no audio tracks in <img src=webm>. I wrote in detail about reasons to support it.

• GIF has preferential treatment in <img> probably only due to a historical accident. Now that browsers support other video codecs natively it seems that they should allow other video (not audio) codecs in all places where GIF is allowed.
• Browsers already support autoplay of video — using the GIF codec. It's absurd that in order to limit autoplay of videos (presumably to save users' bandwith) browsers actually leave no option to websites than to use video files that are 10 times larger than they need to be. Autoplay options should apply to GIF the same way they apply to better codecs (or even situation should be reversed - autoplay of GIF should be penalized for its inefficiency, not privileged).
• GIF is widely used because it just works, despite being primitive and bandwidth-inefficient. And yet, all GIF-killers (APNG, WebP) focus on copying GIF's bandwidth wastefulness and inadequacy for encoding video, as if being technically backwards and producing files 10× larger than 10-year-old H.264 was a virtue. WebP is especially terrible example of this, as it has inherited complexity of VP8, but with changes that made it as bandwith-wasteful as GIF.
• Low-motion backgrounds encoded using VP9 or H.265 video use less bandwidth than a static image in WebP, which is stuck with previous generation of the codec.

The last point is a bit of a risk/downside, because it'll mean that 1-frame WebM files with VP9/VP10 will be the best still image codec we have available on the web.

[domenic]

You are assuming that all mobile browsers allow autoplay for Web Audio which isn't true for Safari iOS.

That's a good point. But my point about it not being worth investing the feature/spec/implementation work on this, given that very few authors want it, still remains to be addressed. Is there data to the contrary, e.g. a large partner that would like video autoplay despite it having no sound?

[Yay295]

Interestingly, I just found this from Google:
https://developers.google.com/speed/webp/faq#why_should_i_use_animated_webp
Scroll down to the third subsection, "Why not simply support WebM in <img>?".

[kornelski]

I'm entirely unconvinced by WebP vs WebM FAQ (it ignores cost of bandwidth wasted by WebP, and puts too much weight on Chrome-specific implementation problems) edited to shorten offtopicing rant

[domenic]

This seems off-topic?

[Yay295]

@pornel - "Chrome-specific implementation problems" are important considering this is tagged with "needs implementor interest".

[foolip]

Is there data to the contrary, e.g. a large partner that would like video autoplay despite it having no sound?

Paging @ojanvafai.

I think this would mostly be ads, videos as background images and Twitter-style videos where playback starts when you scroll by, but is muted by default. This happens on Twitter's Android app as well, so it's not a desktop-centric design choice.

[foolip]

@domenic, I'd also like the autoplay restrictions to be more lax than they are now, but see no reason to think that audio playback without user interaction will become universally allowed any time soon to make this a non-problem. At the very least you'd have off-by-default settings for disabling autoplaying audio and things like do-not-disturb modes on phones.

[domenic]

OK, so, I don't want to be obstructionist here, if I'm in the minority. Let's reset this conversation, with me just logging an objection that I really think we should allow autoplay of sound as well, and understanding that I'm overruled :).

I'd like to understand exactly which use cases we're serving here, as I think it changes the shape of the solution a bit. In particular:

1. Is the idea mostly to replace "animated gif" type of scenarios? This would lean toward the declarative video-as-image idea. In this case I think we'd just throw away any audio tracks, most likely.
2. Or is the idea to allow video-centric apps to have the autoplay-by-default experience, while requiring a user gesture to un-mute? In that case the somewhat-magical muted behavior seems better.

I guess there's nothing to say we can't do both. (2) would be useful for sites like Twitter, Facebook, or Vine, I imagine, whereas (1) would be useful for animated-gif replacement (ads, background images, transcoded gifs). For example, Twitter transcodes gifs, but also allows posting videos: maybe they would use (1) for uploaded transcoded gifs, and (2) for uploaded videos.

I think doing both would be compelling mainly if browsers had ideas on how to use the additional semantic information: as you said, in things like media session or in different controls or menus.

Anyway, as @mounirlamouri said, it would be good to get input from other vendors on this. Does anyone know the relevant people to ping?

[foolip]

@padenot for Gecko, @jernoble @eric-carlson for WebKit

FWIW, "autoplay-by-default experience, while requiring a user gesture to un-mute" (2) is what I'm hoping for, but I have no data, I've just seen that behavior in the wild.

[foolip]

Also, I don't see this as an issue of semantics, rather just a matter of enabling the behaviors we want to enable. If there happens to be a semantic interpretation that's fine, of course.

[mounirlamouri]

I think my main question is whether we can try to catch two birds with one stone and do (1) and (2) with the same solution. (1) can be see as a subset of (2) if switching from video-as-image to regular video is possible and allow then the video to be played unmuted.

@foolip when you say that we should allow muted video to autoplay, do you mean <video autoplay muted> specifically or a video that doesn't make sound in general regardless of the solution? I'm a bit concerned about the compat issues that video.muted = false; not working would imply.

[kornelski]

My use case is a CDN that uses HTTP content negotiation (or UA sniffing) to automagically transcode GIF to WebM to offer 10× reduction in bandwidth (conversion to WebP or APNG is not worth the hassle). That will be possible only when <img src=webm> and background:url(webm) are supported.

[jernoble]

Overall, I really like this idea. Perhaps because Mobile Safari is the most restrictive browser when it comes to automatic playback of media, we get a lot of requests to loosen our restrictions. They fall broadly into three sets of use cases:

1. GIF replacement. Some sites are already doing this (e.g., imgur's .gifv format), and other big media sites with .gif-heavy pages really want to be able to use a looping .mp4 or .webm for bandwidth savings and image quality improvement.
2. Video-as-page-element. Many sites use large, silent video elements on their landing pages, or want dynamic video integrated with the rest of their page. This use case differs from 1. in that the authors want control over the play state, maybe the timeline, and maybe the loop state. Examples include "play on hover", "synchronize playback position with scroll position", "loop N times", etc.
3. Silent until click. The canonical "Facebook feed" or "Twitter timeline", where videos begin playing silently once they scroll on-screen, pause when scrolling off-screen, and only play sound with an explicit user gesture.

@domenic asked "[is there any] large partner that would like video autoplay despite it having no sound?" Yes, each of these use cases have very large clients requesting support for these features in Mobile Safari.

Each of these use cases would be solved with a different proposal which has been listed above. (1) (GIF replacement) would be solved by simply implementing support for .mp4 in an image element. (2) would be solved by (1), with the addition of simple playback controls to image elements. (Note that those additions would be useful even in the absence of (1), as it would allow page control of .gifs as well as .mp4s or .webms.) (3) could only be solved by a video element, with all the trappings and API that provides.

[jernoble]

@foolip said:

If muted is true when play() is called without a user gesture or when the autoplaying steps are reached, playback would be allowed. Setting muted to true while playing would do nothing if there is no user gesture.

Rather than add a user-gesture requirement to the muted property itself (you have to do the same for VideoTrack.enabled, e.g.), it seems simpler to just stop automatic playback (i.e., pause()) if the video became un-muted without a user gesture.

[jernoble]

@domenic said:

I think doing both would be compelling mainly if browsers had ideas on how to use the additional semantic information: as you said, in things like media session or in different controls or menus.

I think Media Session support is going to be very important. Authors using a video as a dynamic page element are probably not going to want, e.g., their videos to automatically play to an Apple TV.

[foolip]

OK, so it sounds like the idea of allowing autoplay while muted and doing something special when trying to unmute without a user gesture is worth exploring more.

Candidates for something are doing nothing or pausing. When doing nothing, there's the risk that the page has no button for the user to unmute and thus the video just plays silently, while if pausing there's the risk that there's no way for the user to resume playback, if it was intended that the video be controlled only by scripts.

@jernoble, is your main concern with "do nothing" that it would break existing content, or that it's confusing to users or developers? Either solution seems simple enough implementation-wise I think.

Would it help at all to measure how often scripts try to change muted from false to true with and without a user gesture?

[mounirlamouri]

I agree that pausing might be better than doing nothing. Though, I think it would help to measure. We are also considering experimenting with this behaviour.

[foolip]

Is there anything we could measure to help decide between pausing or doing nothing? Both seem kind of surprising to me, but I do lean towards doing nothing because at least it doesn't change another bit of state than was poked at.

[mounirlamouri]

I can't think of anything that will help us. We could add a lot of metrics to find out if play/pause was called from a user gesture to find out if pausing would actually break but I don't think it will help given that it would mostly be wild guesses (UI could be removed, the element could be re-used). Finding whether the situation is common sounds more important given that it being uncommon means that we can focus on what's the best solution instead of backward compatibility.

Note that I prefer pausing because it's something that should be smoother. Not un-muting when muted = false; is called sounds like something developers wouldn't expect and the value of muted after such call seams unclear. Should muted returns false or true? Both solutions have downsides. The other solution introduces side effects which isn't ideal but they can be easily observed with the pause event and even the paused property that could be switched synchronously by running the internal pause steps.

[foolip]

I think it the value should remain unchanged, and one can just check the attribute's value. (This is the same way that one can currently tell that a play() call did nothing.) But in #976 (comment) I also said "One might also consider a promise-returning unmute() that can be rejected if it's too subtle to have a setter that does nothing. Would make feature detection slightly easier, too."

[mounirlamouri]

I think adding a unmute() is moot (no pun intended :)) because websites will likely keep using muted for a while and we will need a comprehensive behaviour when muted is used instead of unmute() anyway.

[foolip]

I agree, it would be somewhat silly, the main point being feature detection really. I think this is the feature detection one would use otherwise, which seems fine:

var v = document.createElement('video');
v.muted = true;
v.play();
var supportsGesturelessPlayWhileMuted = !v.paused;

Let's see if I understand both options fully.

Option 1
play(): if muted is false and there is no user gesture, leave paused unchanged and return a rejected promise.
muted setter: if paused is false, the new value is false and there is no user gesture, leave muted unchanged.

Option 2
play(): as in option 1.
muted setter: if paused is false, the new value is false and there is no user gesture, invoke the internal pause steps and set muted to false.

I do like the symmetry of option 1, and it seems like a plus that it leaves the element in a state that we know is tolerable, because it was already in that state.

[jernoble]

I prefer Option 2. Sites should already be responsive to "paused" events, since the UA may make affordances for the user to pause() the video outside of the site-provided custom playback controls. Having a property setter silently fail (Option 1) is much weirder.

[foolip]

Well, OK, let's go with that then. Just to make sure, the hoped-for behavior is exactly what I wrote, with no extra state bit tracking if it has previously played audibly, has been previously unmuted by a user gesture, etc? The rules for play() are subtly different, and I hope we can have something very simple and interoperable here at least.

[foolip]

Based on Intent to Implement and Ship: autoplay muted videos on Android on blink-dev, I think the necessary spec changes here are:

• Change play() and the muted setter as per option 2 in Semantic way to declare a video as an image #976 (comment)
• Change the existing point where the autoplaying flag takes effect to optionally skip it if muted is false.
• Add a new point where the autoplaying flag can take effect based on element visibility.

I'll assign to @mounirlamouri to write a PR or figure out who should :)

[mounirlamouri]

I've started a thread in WICG/interventions:
WICG/interventions#23

Regarding autoplay attribute when muted is false, the spec already allows to simply ignore the attribute. Should muted be underlined as a specific case?

@xxyzzzq and @avayvod will help.

[milesthedisch]

Is there a good way to feature detect autoplay inline-video in Android? Been digging around and I can't find anything. Right now i'm just using the UA string but that's is sort of unreliable.

[foolip]

@milesthedisch, do you mean to detect whether <video autoplay muted src="..."> will work? If so, I think this will do it:

function autoplayMutedSupported() {
  var v = document.createElement('video');
  v.muted = true;
  v.play();
  return !v.paused;
}

(A media element doesn't actually need a src to try playing. It's all pretty weird.)

[mounirlamouri]

I'm going to close this as this has been solved AFAICT.

[alejandroiglesias]

@foolip I am trying to detect if autoplay un-muted is possible, so I modified your function a bit by just removing v.muted = true, so:

function autoplayUnmutedSupported() {
  var v = document.createElement('video');
  v.play();
  return !v.paused;
}

Strangely enough I get true on Chrome 57 for Android while at the same time is not autoplaying a YouTube embed with the autoplay param. The function correctly returns false on iOS Safari and true on desktop Chrome, Safari and Firefox, so I assume it's correct.

Any ideas on why the unexpected result on Chrome 57 for Android?

[avayvod]

@alejandroiglesias Are you trying this in Chrome Dev Tools by any chance? Using DevTools might be recognized by Chrome for Android as a user gesture (for ease of debugging I guess) - that would explain the different result.

https://jsfiddle.net/v09tq5sv/ shows false for me on Chrome for Android if opened without opening DevTools (from chrome://inspect works fine).

[alejandroiglesias]

@avayvod yes, using Chrome Devtools. Thanks a lot for chiming in and adding that info. As you say, it works fine on the fiddle in Chrome for Android. It's possible that running the function by hitting the Enter key after typing its name is also detected as a user gesture.
Thanks again.