Transfer huge binary data buffer between service worker and content script (more efficiently)

[tophf]

Currently it's impossible to transfer 1GB ArrayBuffer/Blob/Uint8Array and the like between an extension background script (service worker) and the extension's content script both instantly (~1ms for Blob, 1sec for buffers) and directly.

Current workarounds

1. The fastest workaround is to create a web_accessible_resources iframe -> then use navigator.serviceWorker messaging and secure message passing with the parent window, all the while using the last parameter of postMessage to enable instant transfers. This workaround is fragile because the web page can delete the iframe, even if we hide it inside shadow DOM. It also adds a considerable overhead to create and initialize the iframe, which is a waste of time (at least 50ms), CPU, and memory.

2. Sending binary data via extension sendMessage API via structured clone algorithm. Chrome can't do it yet, Firefox can. This is quite slow and blocks both processes (extension and the web page) for the duration of the internal serialization/deserialization, which can be long for a huge binary data, thus introducing janks and lags.

Better solutions wanted

1. URL.createObjectURL + asynchronous fetch in the content script doesn't block the page. Currently it is disabled in service workers due to concerns with the lifetime of the blob. The web doesn't suffer from this restriction because a web SW is only used by the same origin pages/workers which all can use postMessage or self.onfetch + Response API. The extension's SW usage patterns have nothing in common with the web SW in 99.9% of cases, so we need this restriction lifted - just for the duration of SW lifetime.

2. Add transfer parameter to chrome.tabs.sendMessage or at least to long-lived messaging via port.postMessage, that transfers ownership of the data instantly same as in the web postMessage's transferables.

[Rob--W]

We've discussed this issue, not the solutions as phrased, but more the general capability request.

The overall sentiment is that the browser vendors are in favor of the better-designed capabilities from the web platform. I'm tentatively marking this as follow-up: follow-up, so that @dotproto can synchronize with his team to confirm whether his personal opinion is also shared by the team.

Note: the current extension messaging APIs have broadcast semantics, which means that there may potentially be more than one receiver. Therefore, if there is a desire to have transferable semantics, then the API needs to modified be such that the message targets on receiver only (e.g. tabId + the existing frameId, or documentId from #294).

[dotproto]

The Chrome Extensions is supportive of exposing URL.createObjectURL() in extension service workers.

Unfortunately I didn't have a chance to check in on the possibility of extending the Chrome's extension messaging system to support transerables. Based on previous conversations in the past, my impression is that this would be a significant amount of work and it may be preferable to simply replace the extensions-specific implementation of messaging with something based on web platform primitives. Given the amount of work this would likely require, I suspect it will likely be at least a year before we're able to seriously evaluate this change. I'm mostly thinking out loud here, but maybe we should tentatively consider redesigning extension messaging in the next manifest version bump?

[dotproto]

@Rob--W and @xeenon, we currently have supportive tags for Firefox and Safari. Can you clarify what specific aspect of the concerns raised or ideas suggested you are each supportive of? I think you were both expressing support for the idea of transferables in the extension messaging APIs, but I'd prefer to have you confirm rather than run with assumptions.

[Rob--W]

@Rob--W and @xeenon, we currently have supportive tags for Firefox and Safari. Can you clarify what specific aspect of the concerns raised or ideas suggested you are each supportive of? I think you were both expressing support for the idea of transferables in the extension messaging APIs, but I'd prefer to have you confirm rather than run with assumptions.

We discussed the feature request in general terms as noted in #293 (comment).

Here is my train of thoughts on the translation from the capability request to an actual API:

• The real request here is for a performant way to send large blobs of data from one side of the extension to another, potentially across processes. Transferables are a concept in window.postMessage. Before we can consider the use of the "Transferable" primitive, we need to rework the API. The reworking of it is much more likely to affect the performance than the last step to transferables.
• runtime.sendMessage and tabs.sendMessage currently have broadcast semantics (similarly for the runtime.connect and tabs.connect). There may be more than one recipient. The actual implementation in Firefox and Chrome is to serialize the message, send it to the parent process, broadcast to all recipients, and deserialize in the recipients. runtime.sendMessage delivers to a single process (extension), tabs.sendMessage to multiple.
• Serialization format: In Chrome, these APIs use some sort of JSON serialization (base::Value to be precise. Not exactly JSON, but similar). In Firefox, structured cloning is used, which is preferable. Chrome has started the work on structured cloning semantics, but it is behind a flag: https://crbug.com/248548#c44 . The author of that project is no longer working there, so I guess that it accidentally fell off the radar.
• If we define the serialization in terms of structured cloning, and make an API change to ensure a 1:1 sender-recipient relation (no btoadcast), then we are able to optimize the implementation by switching to the same implementation as postMessage internally (including opt-in transferable semantics). For comparison, conceptually the optimized implementation could look like this: sender process asks parent process for pipe to recipient. Sender streams the serialization over the pipe to the recipient, who deserializes it. Compared to the current implementation, the memory usage is bounded and the number of process hops for IPC is reduced.
  • The request for targetting one specific recipient has been requested before at Proposal: one-time message passing addressed to a specific documentId or contextId #294.

Currently a unique frame can be targeted (tabId + frameId/documentId) but not extension recipient (e.g. background, devtools, action popup, options_ui, ...). Let's cover that in #294.

Note: window.postMessage does not provide a way to determine whether the message has fully been sent. It may be useful to default to not expecting a response and allowing an opt-in to avoid unnecessary ping-pong.

[rdcronin]

I share @Rob--W 's desire to see if we can leverage existing web APIs for this, and have lots of interest in how we can incorporate this into messaging. I think this will take a good amount of looking into, but in principle, we're supportive.

[dotproto]

I renamed the issue because it's not possible to do this instantly. Regardless of how this is implemented data will have to be copied across processes. While we cannot make this instant, we are open to exploring ways to efficiently move data between a service worker and content script.

[tophf]

Transferring a Blob is effectively instant because only the handle is transferred. Sent, to be precise, not transferred.

[schickling]

Also very interested in this topic as I'm facing the same limitation. Excited to hear that the Chrome team is positive about getting a possible solution implemented.

In the meanwhile I'd be interested in giving the 1. suggested workaround by @tophf a try. @tophf do you happen to know whether there is any public example of the workaround pattern you've described I could study as a reference implementation? Thank you!

[tophf]

@schickling, see "Web messaging (two-way MessagePort)" in https://stackoverflow.com/a/68689866

[schickling]

Thanks a lot. I was able to make it work with that great explanation but it seems it doesn't work within Incognito tabs. Can you confirm this limitation @tophf?

[tophf]

You'll have to use "incognito": "split" in manifest.json.

[schickling]

Works. Thanks a lot! Appreciate your help with this a lot. There are only very few learning resources about this topic. 🙏

[kyr0]

Hey all,

@schickling There's still a few small glitches in all of this, which is, that if you follow the stackoverflow example code, the Promise that is awaited in order for the first MessageChannel "tunneling" response to be resolved, will never fulfill, if the onmessage callback is not assigned first - at least it behaves like that in most recent Chrome versions. Also, the <div> isn't dissolved, but can be. GC won't collect the MessageChannel object and ownership is transferred as well.

Goal: We will establish a bi-directional communication between the content script of an extension and the worker / background script. It will pass all kind of data structures around using transferable objects instantly and the page won't be able to intercept with this easily.

Here is a fully working, battle-tested, TypeScript and documented example (I'm passing a Machine Learning model around):

1. Make sure the necessary permissions are granted

// manifest.json: on top-level, add

  "incognito": "split",
  "web_accessible_resources": [
    {
      "resources": ["tunnel.html", "tunnel.js"],
      "matches": ["<all_urls>"],
      "use_dynamic_url": true
    }
  ]

2. Create the tunnel code files (only used for a splitsecond, then dissolved)

// create tunnel.html

<!-- we are allowed to inject this script, as it is registered in manifest.json's web_accessible_resources -->
<script src=tunnel.js></script>

// create tunnel.js

// one-time postMessage to the service worker
// this callback is executed once; the MessageChannel object
// passed down from the content script is passed to the service worker
// to establish a DIRECT two-way communication channel between the content script and the service worker
window.onmessage = e => {
  if (e.data === new URLSearchParams(location.search).get('secret')) {
    // and that's why we free the event listener instantly
    window.onmessage = null;
    // once the self.onmessage event listener is set up in the service worker
    // we pass it the MessagePort object from the content script
    navigator.serviceWorker.ready.then(swr => {
      swr.active.postMessage('port', [e.ports[0]]);
    });
  }
};

3. Add the injection code to your content script file

// change your content-script.ts

// A: content script
// B: injected iframe and script
// C: background script/worker

// A (content script) cannot pass transferables (e.g. Blobs with GB of data) to C (background script/worker) directly
// So we create a tunnel by passing a MessageChannel to B (injected iframe/script) that passes it to C (background script)
// via the navigator.serviceWorker.messageChannel API which isn't available in content scripts
// the injected iframe/script dissolves itself after the first message is received
// once the tunnel is established, A can talk to C directly using transferabbles by using the standard MessageChannel API
async function makeTunnel(
  path: string,
  onMessage: (e: MessageEvent) => void,
) {
  // we need a new secret for each tunnel to become unique, non-cached
  const secret = Math.random().toString(36);
  const url = new URL(chrome.runtime.getURL(path));
  // this is why we need to set use_dynamic_url in manifest.json's web_accessible_resources entry
  url.searchParams.set("secret", secret);

  const el = document.createElement("div");
  // we attach the element to the shadow DOM to prevent it from bleeding
  const root = el.attachShadow({ mode: "closed" });
  const iframe = document.createElement("iframe");
  iframe.hidden = true;
  root.appendChild(iframe);
  (document.body || document.documentElement).appendChild(el);

  // wait for the iframe to be loaded
  await new Promise((resolve, reject) => {
    iframe.onload = resolve;
    iframe.onerror = reject;
    iframe.contentWindow!.location.href = url.toString();
  });

  // once the iframe is loaded, we send the MessageChannel object to the iframe
  // by reference (transferable); this only happens once
  const mc = new MessageChannel();
  iframe.contentWindow!.postMessage(secret, "*", [mc.port2]);

  // we need to wait for the iframe to respond with its port
  // and assign onMessage to the port first so that addEventListener
  // would be called (new behavior in Chrome)
  await new Promise((cb) => {
    mc.port1.onmessage = onMessage;
    // fulfill the promise after the first message (port is ready, bi-directionally)
    mc.port1.addEventListener("message", cb, { once: true });
  });

  // we can safely remove the injected element and it's iframe now
  if (el.parentNode) {
    el.parentNode.removeChild(el);
  }
  // we return the port to the caller as well (`port.postMessage(...)`)
  return mc.port1 as MessagePort;
}

You should call this function once.

// change your content-script.ts

const port = await makeTunnel(
  "/tunnel.html",
  (e: MessageEvent) => {
    console.log("received from worker:", e.data);
  },
);

// for demonstration, let's pass down some transferables...
console.log("port", port);
console.log("postMesaages...");
port.postMessage(123);
port.postMessage({ foo: "bar" });
port.postMessage(new Blob(["foo"]));

4. Add the following code to your worker script:

// change your worker.ts

const addTunnelListener = (
  onContentMessage: (port: MessagePort, e: MessageEvent) => void,
) => {
  self.onmessage = (connEstablishedEvt) => {
    if (connEstablishedEvt.data === "port") {
      // as we use the reference to the MessagePort here
      // the callback assignment will last as long as the MessagePort
      // so we can use it to communicate with the content script
      connEstablishedEvt.ports[0].onmessage = (messageEvent) =>
        onContentMessage(connEstablishedEvt.ports[0], messageEvent);

      // initial ack/resolve, as we were receiving the port via the tunnel script
      // and it needs to be passed back to the content script, for the last step's
      // Promise to resolve
      connEstablishedEvt.ports[0].postMessage(null);
    }
  };
};

// example implementation; addTunnelListener should be called *once*
addTunnelListener((port: MessagePort, e: MessageEvent) => {
  // prints both in the background console and in the worker script console
  console.log("from content script:", e.data);
  // example to simply echo all data, so we can demonstrate that Blobs are passed back to the content script as they have been passed down here
  port.postMessage(e.data);
});

I hope you'll enjoy this solution and that it works well for ya'all :) Have fun!

And please, after all these years... can't we simply have a spec that won't stand in the way of developers ;)) 🤗 ? I mean, it's fun to hack stuff, but.. this one has been interesting mental gymnastics =)

[kyr0]

Here's a more decent implementation that also supports multiple calls, doesn't collide with an existing Worker's onmessage callback, provides a clean API, alongside generic typing and better naming: kyr0/redaktool@4d06444

[Edea1992]

大家好，

在所有这些中仍然存在一些小故障，即，如果您遵循 stackoverflow 示例代码，如果没有首先分配回调，则为解决第一个“隧道”响应而等待的 Promise 将永远不会实现 - 至少它在最新的 Chrome 版本中的行为是这样的。此外，不会溶解，但可以溶解。GC 不会收集对象，所有权也会转移。MessageChannel``onmessage``<div>``MessageChannel

目标：我们将在扩展的内容脚本和工作器/后台脚本之间建立双向通信。它将立即使用可转移对象传递所有类型的数据结构**，并且**页面将无法轻易地拦截。

这是一个完全运行的、经过实战测试的 TypeScript 和记录的示例（我正在传递一个机器学习模型）：

1. 确保已授予必要的权限

// manifest.json: on top-level, add

  "incognito": "split",
  "web_accessible_resources": [
    {
      "resources": ["tunnel.html", "tunnel.js"],
      "matches": ["<all_urls>"],
      "use_dynamic_url": true
    }
  ]

2. 创建隧道代码文件（仅在瞬间使用，然后解散）

// create tunnel.html

<!-- we are allowed to inject this script, as it is registered in manifest.json's web_accessible_resources -->
<script src=tunnel.js></script>

// create tunnel.js

// one-time postMessage to the service worker
// this callback is executed once; the MessageChannel object
// passed down from the content script is passed to the service worker
// to establish a DIRECT two-way communication channel between the content script and the service worker
window.onmessage = e => {
  if (e.data === new URLSearchParams(location.search).get('secret')) {
    // and that's why we free the event listener instantly
    window.onmessage = null;
    // once the self.onmessage event listener is set up in the service worker
    // we pass it the MessagePort object from the content script
    navigator.serviceWorker.ready.then(swr => {
      swr.active.postMessage('port', [e.ports[0]]);
    });
  }
};

3. 将注入代码添加到内容脚本文件中

// change your content-script.ts

// A: content script
// B: injected iframe and script
// C: background script/worker

// A (content script) cannot pass transferables (e.g. Blobs with GB of data) to C (background script/worker) directly
// So we create a tunnel by passing a MessageChannel to B (injected iframe/script) that passes it to C (background script)
// via the navigator.serviceWorker.messageChannel API which isn't available in content scripts
// the injected iframe/script dissolves itself after the first message is received
// once the tunnel is established, A can talk to C directly using transferabbles by using the standard MessageChannel API
async function makeTunnel(
  path: string,
  onMessage: (e: MessageEvent) => void,
) {
  // we need a new secret for each tunnel to become unique, non-cached
  const secret = Math.random().toString(36);
  const url = new URL(chrome.runtime.getURL(path));
  // this is why we need to set use_dynamic_url in manifest.json's web_accessible_resources entry
  url.searchParams.set("secret", secret);

  const el = document.createElement("div");
  // we attach the element to the shadow DOM to prevent it from bleeding
  const root = el.attachShadow({ mode: "closed" });
  const iframe = document.createElement("iframe");
  iframe.hidden = true;
  root.appendChild(iframe);
  (document.body || document.documentElement).appendChild(el);

  // wait for the iframe to be loaded
  await new Promise((resolve, reject) => {
    iframe.onload = resolve;
    iframe.onerror = reject;
    iframe.contentWindow!.location.href = url.toString();
  });

  // once the iframe is loaded, we send the MessageChannel object to the iframe
  // by reference (transferable); this only happens once
  const mc = new MessageChannel();
  iframe.contentWindow!.postMessage(secret, "*", [mc.port2]);

  // we need to wait for the iframe to respond with its port
  // and assign onMessage to the port first so that addEventListener
  // would be called (new behavior in Chrome)
  await new Promise((cb) => {
    mc.port1.onmessage = onMessage;
    // fulfill the promise after the first message (port is ready, bi-directionally)
    mc.port1.addEventListener("message", cb, { once: true });
  });

  // we can safely remove the injected element and it's iframe now
  if (el.parentNode) {
    el.parentNode.removeChild(el);
  }
  // we return the port to the caller as well (`port.postMessage(...)`)
  return mc.port1 as MessagePort;
}

应调用此函数一次。

// change your content-script.ts

const port = await makeTunnel(
  "/tunnel.html",
  (e: MessageEvent) => {
    console.log("received from worker:", e.data);
  },
);

// for demonstration, let's pass down some transferables...
console.log("port", port);
console.log("postMesaages...");
port.postMessage(123);
port.postMessage({ foo: "bar" });
port.postMessage(new Blob(["foo"]));

4. 将以下代码添加到辅助角色脚本中：

// change your worker.ts

const addTunnelListener = (
  onContentMessage: (port: MessagePort, e: MessageEvent) => void,
) => {
  self.onmessage = (connEstablishedEvt) => {
    if (connEstablishedEvt.data === "port") {
      // as we use the reference to the MessagePort here
      // the callback assignment will last as long as the MessagePort
      // so we can use it to communicate with the content script
      connEstablishedEvt.ports[0].onmessage = (messageEvent) =>
        onContentMessage(connEstablishedEvt.ports[0], messageEvent);

      // initial ack/resolve, as we were receiving the port via the tunnel script
      // and it needs to be passed back to the content script, for the last step's
      // Promise to resolve
      connEstablishedEvt.ports[0].postMessage(null);
    }
  };
};

// example implementation; addTunnelListener should be called *once*
addTunnelListener((port: MessagePort, e: MessageEvent) => {
  // prints both in the background console and in the worker script console
  console.log("from content script:", e.data);
  // example to simply echo all data, so we can demonstrate that Blobs are passed back to the content script as they have been passed down here
  port.postMessage(e.data);
});

我希望您会喜欢这个解决方案，并且它对你们所有:)玩得愉快！

拜托，这么多年过去了......我们不能简单地有一个不会妨碍开发人员的规范吗 ;)） 🤗 ？我的意思是，破解东西很有趣，但是..这个一直很有趣的心理体操=）

Could you please let me know if your solution is compatible with Safari on iOS?

[kyr0]

大家好，
在所有这些中仍然存在一些小故障，即，如果您遵循 stackoverflow 示例代码，如果没有首先分配回调，则为解决第一个“隧道”响应而等待的 Promise 将永远不会实现 - 至少它在最新的 Chrome 版本中的行为是这样的。此外，不会溶解，但可以溶解。GC 不会收集对象，所有权也会转移。 MessageChannelonmessage<div>MessageChannel ``
目标：我们将在扩展的内容脚本和工作器/后台脚本之间建立双向通信。它将立即使用可转移对象传递所有类型的数据结构**，并且**页面将无法轻易地拦截。
这是一个完全运行的、经过实战测试的 TypeScript 和记录的示例（我正在传递一个机器学习模型）：

1. 确保已授予必要的权限

// manifest.json: on top-level, add

  "incognito": "split",
  "web_accessible_resources": [
    {
      "resources": ["tunnel.html", "tunnel.js"],
      "matches": ["<all_urls>"],
      "use_dynamic_url": true
    }
  ]

2. 创建隧道代码文件（仅在瞬间使用，然后解散）

// create tunnel.html

<!-- we are allowed to inject this script, as it is registered in manifest.json's web_accessible_resources -->
<script src=tunnel.js></script>

// create tunnel.js

// one-time postMessage to the service worker
// this callback is executed once; the MessageChannel object
// passed down from the content script is passed to the service worker
// to establish a DIRECT two-way communication channel between the content script and the service worker
window.onmessage = e => {
  if (e.data === new URLSearchParams(location.search).get('secret')) {
    // and that's why we free the event listener instantly
    window.onmessage = null;
    // once the self.onmessage event listener is set up in the service worker
    // we pass it the MessagePort object from the content script
    navigator.serviceWorker.ready.then(swr => {
      swr.active.postMessage('port', [e.ports[0]]);
    });
  }
};

3. 将注入代码添加到内容脚本文件中

// change your content-script.ts

// A: content script
// B: injected iframe and script
// C: background script/worker

// A (content script) cannot pass transferables (e.g. Blobs with GB of data) to C (background script/worker) directly
// So we create a tunnel by passing a MessageChannel to B (injected iframe/script) that passes it to C (background script)
// via the navigator.serviceWorker.messageChannel API which isn't available in content scripts
// the injected iframe/script dissolves itself after the first message is received
// once the tunnel is established, A can talk to C directly using transferabbles by using the standard MessageChannel API
async function makeTunnel(
  path: string,
  onMessage: (e: MessageEvent) => void,
) {
  // we need a new secret for each tunnel to become unique, non-cached
  const secret = Math.random().toString(36);
  const url = new URL(chrome.runtime.getURL(path));
  // this is why we need to set use_dynamic_url in manifest.json's web_accessible_resources entry
  url.searchParams.set("secret", secret);

  const el = document.createElement("div");
  // we attach the element to the shadow DOM to prevent it from bleeding
  const root = el.attachShadow({ mode: "closed" });
  const iframe = document.createElement("iframe");
  iframe.hidden = true;
  root.appendChild(iframe);
  (document.body || document.documentElement).appendChild(el);

  // wait for the iframe to be loaded
  await new Promise((resolve, reject) => {
    iframe.onload = resolve;
    iframe.onerror = reject;
    iframe.contentWindow!.location.href = url.toString();
  });

  // once the iframe is loaded, we send the MessageChannel object to the iframe
  // by reference (transferable); this only happens once
  const mc = new MessageChannel();
  iframe.contentWindow!.postMessage(secret, "*", [mc.port2]);

  // we need to wait for the iframe to respond with its port
  // and assign onMessage to the port first so that addEventListener
  // would be called (new behavior in Chrome)
  await new Promise((cb) => {
    mc.port1.onmessage = onMessage;
    // fulfill the promise after the first message (port is ready, bi-directionally)
    mc.port1.addEventListener("message", cb, { once: true });
  });

  // we can safely remove the injected element and it's iframe now
  if (el.parentNode) {
    el.parentNode.removeChild(el);
  }
  // we return the port to the caller as well (`port.postMessage(...)`)
  return mc.port1 as MessagePort;
}

应调用此函数一次。
// change your content-script.ts

const port = await makeTunnel(
  "/tunnel.html",
  (e: MessageEvent) => {
    console.log("received from worker:", e.data);
  },
);

// for demonstration, let's pass down some transferables...
console.log("port", port);
console.log("postMesaages...");
port.postMessage(123);
port.postMessage({ foo: "bar" });
port.postMessage(new Blob(["foo"]));

4. 将以下代码添加到辅助角色脚本中：

// change your worker.ts

const addTunnelListener = (
  onContentMessage: (port: MessagePort, e: MessageEvent) => void,
) => {
  self.onmessage = (connEstablishedEvt) => {
    if (connEstablishedEvt.data === "port") {
      // as we use the reference to the MessagePort here
      // the callback assignment will last as long as the MessagePort
      // so we can use it to communicate with the content script
      connEstablishedEvt.ports[0].onmessage = (messageEvent) =>
        onContentMessage(connEstablishedEvt.ports[0], messageEvent);

      // initial ack/resolve, as we were receiving the port via the tunnel script
      // and it needs to be passed back to the content script, for the last step's
      // Promise to resolve
      connEstablishedEvt.ports[0].postMessage(null);
    }
  };
};

// example implementation; addTunnelListener should be called *once*
addTunnelListener((port: MessagePort, e: MessageEvent) => {
  // prints both in the background console and in the worker script console
  console.log("from content script:", e.data);
  // example to simply echo all data, so we can demonstrate that Blobs are passed back to the content script as they have been passed down here
  port.postMessage(e.data);
});

我希望您会喜欢这个解决方案，并且它对你们所有:)玩得愉快！
拜托，这么多年过去了......我们不能简单地有一个不会妨碍开发人员的规范吗 ;)） 🤗 ？我的意思是，破解东西很有趣，但是..这个一直很有趣的心理体操=）

Could you please let me know if your solution is compatible with Safari on iOS?

I'm sorry, I haven't checked it. My extension isn't compatible with mobile devices (it's a desktop only UI in a business setting), so there was no need on my side. It would be amazing, if you could try it out an report back wether it works as others might be interested in that as well. Thank you in advance!