Feed a Response to existing APIs

[annevk]

If you manage all network activity yourself you still need a way to inject the results into existing APIs that consume them. There's two approaches here that I know of:

<img>.srcObject = responseInstance
<img>.src = convertResponseToURL(responseInstance)

The first is more elegant, the second invokes Fetch twice (but the convertResponseToURL() URL should be a way shorter trip). If we go with the second, which scales better due to the many APIs in existence, we need to avoid the flaws from blob URLs and make sure the URL can be used only once and consumes the underlying stream (you cannot use responseInstance again, if you wanted to do that you'd have to clone it first).

[domenic]

There is an additional paradigm where you pipe the response body stream to a writable stream. I am not 100% sure on how this would all work but you could imagine e.g.

const ws = new StreamingImageDecoder();
responseInstance.body.pipeTo(ws).then(() => {
  // use ws.something... maybe ws.bitmap returns an ImageBitmap?
  // ws.element returns a HTMLImageElement?
});

In this scenario you could probably access ws.something before the pipe finishes to see the in-progress image. (Remember the 90s where you could see images stream visually over slow connections?)

Another version (possibly more useful for continually-updating things) is

responseInstance.body.pipeTo(videoEl.writable);

[wanderview]

There is an additional paradigm where you pipe the response body stream to a writable stream.

Sorry to raise the readable-vs-writable thing again, but it seems this:

.srcObject = responseInstance

Could accept a ReadableStream object or anything that can be unwrapped to a ReadableStream (like Response). This would keep the processing loop in the C++ code where it currently lives today for these APIs. You also get the backpressure for free, of course.

This doesn't preclude having a WritableStream mechanism as well, though. I expect the C++ code would just create a pipe, expose the writer end, and just do its current read loop internally. So in effect it would probably just be a convenience to avoid manually creating a pipe.

Another way to describe what I'm saying, is things with a .src attribute are designed to consume sources. A ReadableStream is a source. A WriteableStream is a sink. Converting APIs from consume-a-source to expose-a-sink is an extra amount of work.

In the end it would probably be nice to have both.

[annevk]

Note that a stream doesn't have the metadata that a response does have. Most APIs want (some of) that metadata.

[wanderview]

Note that a stream doesn't have the metadata that a response does have. Most APIs want (some of) that metadata.

You mean the http headers or something else?

I have to say I prefer setting the Response object to creating some kind of URL. I like the concept that a Response is essentially an evaluated URL.

It seems setting a img.src = new Request(...) with an implied fetch() would be closer to what assigning a URL currently does.

It would be also nice if we could set img.src to a Promise. So img.src = cache.match(req).

Unless there is some time pressure forcing a short term solution, it would be nice to go the object route instead of creating a complex new URL type to intermediate.

[annevk]

Headers, yes. The problem with doing things object-based is that there's dozens of things in CSS that take a URL, but have no good API. And outside CSS there's dozens of APIs too. But maybe we should just start rolling them out one-by-one and try to get there.

[domenic]

Re-discussed at F2F 2015-07-22:

• Adapting srcObject to allow Response or ReadableStream (or ReadableByteStream??) is a good path forward.
• Cannot allow e.g. Promise<Response> since promises are not distinguishable from other types
• Can do HTMLImageElement, HTMLMediaElement, maybe HTMLScriptElement and HTMLStyleElement.

[annevk]

(One thing I found while looking at this today is that only Safari seems to implement srcObject = blob for HTMLMediaElement. Everyone else throws, which seems appropriate. There are also no tests for that scenario in web-platform-tests.)

[domenic]

Coming back to this years later.

First off, I think we should design our solution to accept:

• Request
• Promise<Response>
• Maybe any BodyInit, including ReadableStream, if we can figure out some default headers. (Note that Blob is already accepted by HTMLMediaElement.)

Cannot allow e.g. Promise<Response> since promises are not distinguishable from other types

This note was a bit unclear to me. I think it was talking about existing srcObject properties: i.e., we couldn't do (MediaStream or Media or Blob or Promise<Response>) for HTMLMediaElement's srcObject. This is true, especially because of the getter, which currently synchronously returns the value it was set to.

I see a few solutions:

• Change IDL to support separate getter/setter types, and change the setter to accept Promise<... all the things ...> while the getter returns null while the promise is resolving and returns the actual object once it's given
• Choose a different property name, e.g. srcResponse, and use that everywhere.
• Choose a different API shape, e.g. one without a troublesome getter, like load() or loadFromResponse() or similar.

I lean toward the last one, to be honest. Any thoughts?

[wanderview]

@domenic Do you have thoughts on how to handle browser UX that exposes element URLs to the user or developer? For example, if we feed a new Response(body) to an <img> what would the user get when they right-click and open in new tab?

I really want to see this feature, but I'm not sure how to handle this issue.

[domenic]

Great question. How do <audio> and <video> currently handle it?

[wanderview]

@jakearchibald I seem to recall you suggested once we should allow the creation of a blob URL from a Response which might address this situation. Is that correct?

[mkruisselbrink]

@jakearchibald I seem to recall you suggested once we should allow the creation of a blob URL from a Response which might address this situation. Is that correct?

That seems like a terrible idea to me. This feature should let us get rid of blob URLs, we don't want to encourage more blob URL usage.

[wanderview]

I'm probably not remembering correctly or misunderstood.

[mkruisselbrink]

(i.e. we've been pushing back against being able to create blob URLs for responses with the argument that you should just be able to use the Response directly like this issue is suggesting)

[annevk]

I'm concerned with opening up the request side as well. The way <img> fetches is extremely poorly defined. We'd have to get that into shape before extending it.

[domenic]

I was envisioning that feeding a Request is the same as feeding the result of fetch(thatRequest); it's just sugar.

[annevk]

It seems weird to me if it bypasses the image fetching pipeline and caches defined therefore. I'd prefer sticking to const response = await fetch(...); img.srcObject = response initially.

[kinu]

Just learned about this, seems neat.

#49 (comment) only mentions images, media, scripts and styles, but I wonder if this could be also used for nested frames (i.e. iframes).

One obviously tricky case is cross-origin cases (esp. with service workers, something that's discussed in w3c/ServiceWorker#590), but what if we allow this only for same-origin or sandboxed cases (without same-origin-access)? Then one can put fine control over when to load embeddable content, which feels convenient.

[annevk]

It seems doable, but given policies and the general complexity of navigation it might be a lot more involved. I would suggest we start with a "simpler" variant and work our way up from there.

[kinu]

@annevk yep it makes sense and I agree. Wanted to exercise myself a bit to see how this idea can be extended.