RangeError: offset is out of bounds

[spencekim]

I am running the flan-t5 model for text2text-generation (e.g. await pipeline("text2text-generation", "flan-t5-base");) in a service worker. Inference runs without an issue for the first several runs, but eventually I get this error:

RangeError: offset is out of bounds
    at Uint8Array.set (<anonymous>)
    at e.createSessionAllocate (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:450663)
    at Object.e.createSession (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:451415)
    at e.createSession (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:443678)
    at e.OnnxruntimeWebAssemblySessionHandler.loadModel (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:446572)
    at Object.createSessionHandler (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:156408)
    at Function.create (webpack-internal:///../../node_modules/onnxruntime-common/dist/lib/inference-session-impl.js:176:39)
    at async constructSession (webpack-internal:///../../node_modules/@xenova/transformers/src/models.js:20:19)
    at async Promise.all (index 2)
    at async Function.from_pretrained (webpack-internal:///../../node_modules/@xenova/transformers/src/models.js:102:75)

I ran with the same input params 5 times in a row and still get the error, so it doesn't seem like it's an issue with an invalid input value.

[xenova]

Could you provide the text you used? It may be too long for the context window (and I am not doing truncation of inputs at the moment).

Edit: Nevermind, the error looks to be when you create the session. The error looks to be coming from onnxruntime-web. Could you perhaps provide more steps to see if I can reproduce it on my side?

Or, could you provide the full code you are using? It's quite difficult to determine the root cause from the above error message. The only thing I can think of right now is you are loading the model multiple times, leading to an out of memory error?

[spencekim]

Here is my minimal reproducible example within a typescript web worker:

import { pipeline, env } from "@xenova/transformers";

env.remoteModels = false;
env.localURL = "/models";

onmessage = async (event: MessageEvent) => {
  const pipe = await pipeline("text2text-generation", "flan-t5-base");

  const completion: string[] = await pipe(
    "Summarize: The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
  );

  console.log(completion);

  return completion[0];
};

Here are my logs after running 5 times in a row:

['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
['The Eiffel Tower is the tallest structure in France, and the tallest structure in the world.']
Uncaught (in promise) RangeError: offset is out of bounds
    at Uint8Array.set (<anonymous>)
    at e.createSessionAllocate (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:450663)
    at Object.e.createSession (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:451415)
    at e.createSession (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:443678)
    at e.OnnxruntimeWebAssemblySessionHandler.loadModel (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:446572)
    at Object.createSessionHandler (webpack-internal:///../../node_modules/onnxruntime-web/dist/ort-web.min.js:7:156408)
    at Function.create (webpack-internal:///../../node_modules/onnxruntime-common/dist/lib/inference-session-impl.js:176:39)
    at async constructSession (webpack-internal:///../../node_modules/@xenova/transformers/src/models.js:20:19)
    at async Promise.all (index 2)
    at async Function.from_pretrained (webpack-internal:///../../node_modules/@xenova/transformers/src/models.js:102:75)
e.createSessionAllocate @ ort-web.min.js?d8cf:6
e.createSession @ ort-web.min.js?d8cf:6
e.createSession @ ort-web.min.js?d8cf:6
loadModel @ ort-web.min.js?d8cf:6
createSessionHandler @ ort-web.min.js?d8cf:6
create @ inference-session-impl.js?f23d:170
Promise.then (async)
asyncGeneratorStep @ index.ts?9da8:168
_next @ index.ts?9da8:168
eval @ index.ts?9da8:168
eval @ index.ts?9da8:168
eval @ index.ts?9da8:173

[xenova]

Okay yes, it looks like you are creating multiple versions of the model, leading to an out of memory error. The current Caching API caches the model, but does not force you to only create one instance of the model. If you move your
const pipe = await pipeline("text2text-generation", "flan-t5-base"); to the same level as env.localURL, that should solve your problem.

Alternatively, you can follow what I did in my worker example (https://github.com/xenova/transformers.js/blob/main/assets/js/worker.js), I use a singleton class to ensure that only one pipeline is created.

---

Something like this should work:

import { pipeline, env } from "@xenova/transformers";

env.remoteModels = false;
env.localURL = "/models";

const pipe = await pipeline("text2text-generation", "flan-t5-base");

onmessage = async (event: MessageEvent) => {

  const completion: string[] = await pipe(
    "Summarize: The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."
  );

  console.log(completion);

  return completion[0];
};

[spencekim]

Ah, that makes a lot of sense. Closing this one. Thank you!

[spencekim]

@xenova the above solution worked great.

However, with this same pipeline, I seem to be seeing this error come up fairly often:

TypeError: Cannot convert undefined to a BigInt
    at BigInt (<anonymous>)
    at eval (models.js?a626:289:1)
    at Array.map (<anonymous>)
    at Function.toI64Tensor (models.js?a626:289:1)
    at Function.prepare_inputs (models.js?a626:303:1)
    at Function.forward (models.js?a626:557:1)
    at Function.runBeam (models.js?a626:544:1)
    at Function.generate_single (models.js?a626:366:1)
    at eval (models.js?a626:326:1)
    at Array.map (<anonymous>)

Any ideas? I don't think it has to do with context window size, since my input is fairly small and has worked on bigger.

[xenova]

This occurs when the tokenizer produces undefined tokens (most likely due to unknown characters). Could you send the input you used? Also, could you open a new issue with that? Since it is probably a real bug.

[spencekim]

Yes I'll open a new issue, thanks.