Local Edge functions: event loop error after a few minutes #206

ChuckJonas · 2023-11-08T22:37:25Z

Describe the bug

It seems that after my edge function has been running locally for some time (perhaps a few minutes) I start to get these messages in the logs:

wall clock duration warning. isolate: df3c6dc8-dcc9-4348-a278-e0e1ff102fc3
wall clock duration warning. isolate: 93b745a9-cdb0-49a0-ae88-514a87391cd6
wall clock duration warning. isolate: ce00ba14-d7f7-4e27-8b6e-92191f485a81
wall clock duration warning. isolate: 39ebddfd-8b3e-4af6-a966-178f821877b6

And then eventually I see this:

event loop error: TypeError: Cannot convert undefined or null to object
    at slice (<anonymous>)
    at new DOMException (ext:deno_web/01_dom_exception.js:126:14)
    at [[[signalAbort]]] (ext:deno_web/03_abort_signal.js:83:14)
    at AbortController.abort (ext:deno_web/03_abort_signal.js:172:30)
    at file:///home/deno/main/index.ts:147:31
    at Object.action (ext:deno_web/02_timers.js:151:11)
    at handleTimerMacrotask (ext:deno_web/02_timers.js:65:10)
    at eventLoopTick (ext:core/01_core.js:189:21)

After that error happens, all my requests fail with this:

Error in worker connection: connection closed before message completed
request failed (uri: "/my-function" reason: hyper::Error(Canceled, "connection was not ready"))

The function is as simple as this:

Deno.serve(async (req) => {
  return new Response(
    JSON.stringify({ publicToken: "123" }),
    {
      headers: { "Content-Type": "application/json" },
    }
  );
});

To Reproduce
Steps to reproduce the behavior:

Create a new function with the above code
run supabase functions serve
Call the function at least once (not sure if this is actually necessary)
Wait a few minutes... You should eventually see the above event loop error
Try to call the function again. You will get "connection was not ready" error

Expected behavior
Local functions should not randomly crash after only a few minutes :)

Desktop (please complete the following information):

OS: OSX
Version of CLI: 1.110.1

Additional context

At least one other person has reported experiencing this problem in the supabase discord (The above report is using their description).

The text was updated successfully, but these errors were encountered:

ChuckJonas · 2023-11-08T23:44:45Z

I rolled back to 1.106.1 and the issue seem to go away. I still see the wall clock duration warning. isolate: 88c59c58-1271-4712-ae7e-863ecb2449fa errors, but it no longer crashes (so maybe those aren't related to the crash)

sweatybridge · 2023-11-09T03:53:35Z

Transferring because it sounds like an issue with edge runtime container.

scottweinert · 2023-11-09T05:26:05Z

I can also verify that I'm experiencing the same issue.

confusedmatrix · 2023-11-10T13:26:08Z

Seeing the same issue also

evelant · 2023-11-10T18:32:36Z

I'm also seeing this issue, can't update supabase beyond 1.106.1 or edge fns break every few minutes.

rbkayz · 2023-11-15T06:13:26Z

yep. same

d-e-h-i-o · 2023-11-15T16:28:19Z

I'm also experiencing this issue with CLI version 1.110.1

hyldmo · 2023-11-16T15:21:17Z

I had this is issue too on my M2 mac until i upgraded Docker Desktop (it was very much out of date) as well as supabase. Now running without a single crash on supabase 1.112.2 and Docker Desktop 4.25.1 (128006). Still seeing the wall clock duration warning though, but no crashes.

d-e-h-i-o · 2023-11-16T15:40:30Z

I had this is issue too on my M2 mac until i upgraded Docker Desktop (it was very much out of date) as well as supabase. Now running without a single crash on supabase 1.112.2 and Docker Desktop 4.25.1 (128006). Still seeing the wall clock duration warning though, but no crashes.

My Docker Desktop version is 4.19.0 (106363

scottweinert · 2023-11-17T14:05:46Z

Thanks @hyldmo. Updating Docker Desktop and my supabase CLI resolved the crashing issue.

rothnic · 2023-11-29T18:20:32Z

Any comments on whether the wall clock duration reached. isolate: 11455258470253939329 logs are a problem? It kind of seems like the connection isn't being closed out or something, but the response it getting returned from the endpoint. Is there something else we should be doing to tell supabase that the request is complete?

all-mute · 2024-02-11T08:05:49Z

same

SebasScript · 2024-02-23T09:14:56Z

same issue here, i don't get an error, but all cloud functions become very slow or irresponsive after a couple requests. Seems like the workers don't exit execution. If i restart the runtime it works again for a couple of request

CalvinWilkinson · 2024-03-15T14:39:37Z

I as well am getting related issues.

I get wall clock log msgs like these:

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

I also am having an issue where I cannot use the existsSync() and Deno.readDirSync() functions.
When running these, I get the following error:

runtime has escaped from the event loop unexpectedly: event loop error: TypeError: Deno.statSync is not a function
    at existsSync (https://deno.land/[email protected]/fs/exists.ts:121:23)
    at EnvService.dirDoesNotExist (file:///home/deno/functions/instructor-ui/services/env-service.ts:100:13)
    at EnvService.validateInitEnvFile (file:///home/deno/functions/instructor-ui/services/env-service.ts:65:14)
    at EnvService.loadEnv (file:///home/deno/functions/instructor-ui/services/env-service.ts:22:48)
    at file:///home/deno/functions/instructor-ui/index.ts:5:12
failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

I have tried to search for any open or closed issues on this to see if I can get some more info but have come up dry.

nyannyacha · 2024-03-18T22:31:16Z

hello @CalvinWilkinson 😁

This runtime provides a limited File API compared to Deno ¹, and that's a relatively recent change. From a security perspective, allowing file access can be problematic.

As for the error message... that's something that needs to be documented for self-hosted users 🙄

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

Do you mind if I ask what version of the runtime you're getting this message from? I remember this is something I've already fixed.

failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

This error message is a trivial runtime message caused by a script error, so you don't have to care about it, but now that I think about it, it's a bit verbose for a user 😅

https://supabase.com/docs/guides/functions/debugging#limitations ↩

CalvinWilkinson · 2024-03-19T00:07:14Z

hello @CalvinWilkinson 😁

This runtime provides a limited File API compared to Deno 1, and that's a relatively recent change. From a security perspective, allowing file access can be problematic.

As for the error message... that's something that needs to be documented for self-hosted users 🙄
wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped
Do you mind if I ask what version of the runtime you're getting this message from? I remember this is something I've already fixed.
failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}
This error message is a trivial runtime message caused by a script error, so you don't have to care about it, but now that I think about it, it's a bit verbose for a user 😅

Footnotes

https://supabase.com/docs/guides/functions/debugging#limitations ↩

That is good to know. Thanks for the update!!

As for the runtime, here is what I have:

Deno - v1.41.3
Supabase - 1.145.4

I hope that is what you need. If you need anything else. let me know.

nyannyacha · 2024-03-19T01:50:52Z

Your version of Deno is different from the version of Deno that edge-runtime is using internally.
I assume the version of Deno you are mentioning is a standalone.

The actual version of edge-runtime included in the 1.145.4 you mentioned is v1.36.1¹² The version of Deno it is using is v1.40.3³.

It's pretty complicated, isn't it?😅 I think this needs to be documented at some point.
And it looks to me like you're using a version prior to the one that fixed the bug (that should be fixed at edge-runtime v1.38.0 and above).

You can upgrade the CLI version to beta to bring edge-runtime up to date⁴.

CalvinWilkinson · 2024-03-20T18:37:36Z

Your version of Deno is different from the version of Deno that edge-runtime is using internally. I assume the version of Deno you are mentioning is a standalone.

The actual version of edge-runtime included in the 1.145.4 you mentioned is v1.36.11 2 The version of Deno it is using is v1.40.33.

It's pretty complicated, isn't it?😅 I think this needs to be documented at some point. And it looks to me like you're using a version prior to the one that fixed the bug (that should be fixed at edge-runtime v1.38.0 and above).

You can upgrade the CLI version to beta to bring edge-runtime up to date4.

Footnotes

https://github.com/supabase/cli/blob/v1.145.4/internal/utils/misc.go#L37 ↩

https://github.com/supabase/edge-runtime/releases/tag/v1.36.1 ↩

https://github.com/supabase/edge-runtime/blob/v1.36.1/crates/cli/.env.build ↩

https://github.com/supabase/cli/releases ↩

That makes sense to me!!

Indeed I am running a separately installed version of deno due to other projects.
Thanks for the update!!

ManoloTonto1 · 2024-12-12T08:25:45Z

I am also running into the same Issue using transformers.js

import { layer_norm, pipeline } from "npm:@huggingface/transformers";

export async function createEmbeddings(content: string): Promise<number[]> {
    const pipe = await pipeline(
        "feature-extraction",
        "nomic-ai/nomic-embed-text-v1.5",
        {
            // quantized: false
        }
    );
    const tensors = await pipe(content, { pooling: "mean" });
    const matryoshka_dim = 512;
    const embeddings = layer_norm(tensors, [tensors.dims[1]])
        .slice(null, [0, matryoshka_dim])
        .normalize(2, -1);
    return embeddings.tolist() as number[];
}


Deno.serve(async (req) => {
 ....
 
     for (const row of rows) {
        const { id, [contentColumn]: content } = row;

        if (!content) {
            console.error(`No content available in column '${contentColumn}'`);
            continue;
        }

        const output = await createEmbeddings(content);
        const embedding = JSON.stringify(output);

        const { error } = await supabase
            .from(table)
            .update({
                [embeddingColumn]: embedding,
            })
            .eq("id", id);

        if (error) {
            console.error(
                `Failed to save embedding on '${table}' table with id ${id}`,
                error
            );
            continue;
        }
    }
 })

sweatybridge transferred this issue from supabase/cli Nov 9, 2023

andreespirela self-assigned this Jan 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local Edge functions: event loop error after a few minutes #206

Local Edge functions: event loop error after a few minutes #206

ChuckJonas commented Nov 8, 2023 •

edited

Loading

ChuckJonas commented Nov 8, 2023 •

edited

Loading

sweatybridge commented Nov 9, 2023

scottweinert commented Nov 9, 2023

confusedmatrix commented Nov 10, 2023

evelant commented Nov 10, 2023

rbkayz commented Nov 15, 2023

d-e-h-i-o commented Nov 15, 2023

hyldmo commented Nov 16, 2023

d-e-h-i-o commented Nov 16, 2023

scottweinert commented Nov 17, 2023

rothnic commented Nov 29, 2023

all-mute commented Feb 11, 2024

SebasScript commented Feb 23, 2024

CalvinWilkinson commented Mar 15, 2024

nyannyacha commented Mar 18, 2024 •

edited

Loading

CalvinWilkinson commented Mar 19, 2024

Footnotes

nyannyacha commented Mar 19, 2024

CalvinWilkinson commented Mar 20, 2024

Footnotes

ManoloTonto1 commented Dec 12, 2024

Local Edge functions: event loop error after a few minutes #206

Local Edge functions: event loop error after a few minutes #206

Comments

ChuckJonas commented Nov 8, 2023 • edited Loading

ChuckJonas commented Nov 8, 2023 • edited Loading

sweatybridge commented Nov 9, 2023

scottweinert commented Nov 9, 2023

confusedmatrix commented Nov 10, 2023

evelant commented Nov 10, 2023

rbkayz commented Nov 15, 2023

d-e-h-i-o commented Nov 15, 2023

hyldmo commented Nov 16, 2023

d-e-h-i-o commented Nov 16, 2023

scottweinert commented Nov 17, 2023

rothnic commented Nov 29, 2023

all-mute commented Feb 11, 2024

SebasScript commented Feb 23, 2024

CalvinWilkinson commented Mar 15, 2024

nyannyacha commented Mar 18, 2024 • edited Loading

Footnotes

CalvinWilkinson commented Mar 19, 2024

Footnotes

nyannyacha commented Mar 19, 2024

Footnotes

CalvinWilkinson commented Mar 20, 2024

Footnotes

ManoloTonto1 commented Dec 12, 2024

ChuckJonas commented Nov 8, 2023 •

edited

Loading

ChuckJonas commented Nov 8, 2023 •

edited

Loading

nyannyacha commented Mar 18, 2024 •

edited

Loading