-
-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FunctionsFetchError: Failed to send a request to the Edge Function - effecting 1% of invocations #263
Comments
I have exactly the same issue, around 2% of the requests fail with this error. I've been talking with the support for a month but they haven't managed to resolve the issue |
There's another related thread here which I don't understand why it was closed. I'm also facing this error and wondering what a possible solution would be. I recently started using backoff on the client side to retry failed calls, but only helped to some degree. Still facing errors after all retries have been consumed. L.E. I was able to reproduce this locally by shutting down the docker container for the edge runtime:
Also seeing errors when fetching DB object (so not via edge functions, but via the normal DB REST API): Seriously considering moving away from Supabase. I really love the product, but if it's this unreliable in a production environment then it's not really usable. |
We try to figure out the causes for these intermittent failures. But these request failures can happen due to multiple reasons:
I think it's best to have some error handling and retrying mechanism implemented in the client calling the edge function to reduce the chances of them erroring out (we'll consider building some of this logic into |
I retry for 3 times with backoff of 2, 4, 8 seconds. Most of the time the issue goes away, but around 0.5% of the requests still fail. |
@laktek as mentioned already using backoff retries, but still seeing issues. And IMO that's a terrible solution to suggest for users (hey, you should just retry requests). Because failures can still happen and products are loosing customers because of them. |
@laktek here is another type of error I was able to catch recently. It's the html code of a cloudflare page. I copy pasted it into an html file and it looks like this:
I think it's pretty clear from this that the issue is on Supabase's end. Whatever server you have running the edge functions runtime is crashing. |
@laktek, thanks for your response. The biggest issue here is that the error codes don't give any information on which one of the options you listed was the reason for the error. From our perspective
We already have retries and error handling built into our site, but that doesn't fix the problem, and it doesn't seem like there is a path forward on my end to solve the problem. I hope you can find how incredible frustrating this is on our end. We really love supabase, and would really love to stay on supabase edge functions, but I'm starting to think we have to move off unless you are able to indicate to us that this is something that you can solve within the next month or so. |
Another note to add is, it is definitely something to do with the cloudflare CDN. Every invocation 502 error we have has cloudflare listed in the response metadata. { |
Perhaps some sort of request ID, or OpenTracing header, could be added, to make it easier to find logs corresponding to the failing requests. Similar to e.g. the |
I had the same issue. 2-4% of all requests just failed for no apparent reason at all. I ended up switching to Bun hosted on fly.io. It was really easy and the experience is way better. It just works, no hassles with broken Deno tooling (monorepos are practically impossible), bugs in Deno, missing features in Deno, random failures, etc. |
Are y'all still experiencing random errors? We've made some stability improvements in platform which we believe should help reduce the random 502 errors. |
Hi Im getting this (similar) error when doing password reset on my webapp. |
@cspace001 password reset using Edge Functions? |
@laktek I still seem to get this error more often than I'd like in my pg_cron --> pg_net --> edge function invocation flow. |
I've improved the example in pg_cron for calling edge functions so you raise the timeout value: select
cron.schedule(
'invoke-function-every-half-minute',
'30 seconds',
$$
select
net.http_post(
url:='https://project-ref.supabase.co/functions/v1/function-name',
headers:=jsonb_build_object('Content-Type','application/json', 'Authorization', 'Bearer ' || 'YOUR_ANON_KEY'),
body:=jsonb_build_object('time', now() ),
timeout_milliseconds:=5000
) as request_id;
$$
);
|
@alexbriannaughton Can you try increasing the timeouts in pg_net requests as mentioned in above example from @mansueli ? |
@laktek I had actually already set the timeout_milliseconds to 4500 to troubleshoot before posting here and opening a support request. I will note that I haven't had the issue for the last 24 hours, but I'm not sure what I did on my end to make it stop! Edit: had another one the following day. |
We've been dealing with this for about a year now. It's hard to tell the exact percentage of failing requests but it seems like 2% is probably accurate. We've built in backoff on the client side all over the place but that's more annoying to do on function-to-function calls and is also a terrible end user experience. These days the errors seem to be mostly 502, 503, and 520. Just today we got this one: in our payment processing code. There are no other logs I can look at, this all seems to be squarely a Supabase issue and it hasn't really seemed to improve much over time |
@RedChops, can you open a support ticket (via https://supabase.help) with the name of the project ID and functions that are receiving these errors? I'll investigate internally to find causes for these errors. |
We've been seeing the same for multiple weeks (we're seeing ~5% 502s), opened a support ticket there, ticket ID We only get |
Bug report
Describe the bug
We are getting intermittent "Failed to send a request to the Edge Function" errors. This bug is quite hard to reproduce. I have never been able to reproduce it locally, and 98.5% of function invocations go through just fine. For context, we've seen this error 1.3k times out of 72,474 invocations. I've had users report that they are getting this error, but then when they retry it goes through fine the next time.
My big questions are:
The stack trace we see in sentry is this:
To Reproduce
I can't even reproduce this locally, so I can't provide reproduction steps.
Additional Context
We are running supabase CLI 1.142.1 and
"@supabase/supabase-js": "2.39.1",
Please let me know if there's anything else I can provide that would be helpful.
The text was updated successfully, but these errors were encountered: