onSnapshot exceeds retries?

[LordMerc]

• Operating System version: 1809___
• Firebase SDK version: 8.2.0_
• Library version: _8.2.0
• Firebase Product: database_ (auth, database, storage, etc)

[REQUIRED] Step 3: Describe the problem

After I'd say 2-3 hours, it'll error that I exceeded maximum retries. I've searched for any relevant encounters with this to try to fix but couldn't find any.

Steps to reproduce:

I'd honestly say just wait 2-3 hours and it should occur on it's own.

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[schmidt-sebastian]

@DamnMerc Thanks for reporting this. Can you send us logs from your client? You can enable logging by invoking

admin.firestore.setLogFunction(console.log);

Thanks!

[matej-svejda]

Im not the original reporter, but Im experiencing the same issue and I managed to reproduce it with logging enabled (basically I just have to wait a few hours). Here are the logs:
firestore_max_retries.txt

[LordMerc]

Im not the original reporter, but Im experiencing the same issue and I managed to reproduce it with logging enabled (basically I just have to wait a few hours). Here are the logs:
firestore_max_retries.txt

Thanks for this! Everytime this happened I wasn't at home, so I couldn't catch the errors without it already passing in console.

[schmidt-sebastian]

I'm not 100% sure I found the root cause for your issue, but the logs do show that we are performing extra retries for each error. The retries can then again cause more retries, which could explain why you are quickly exceeding the retry limit of ten. Once googleapis/nodejs-firestore#713 we can try and see if the behavior improves.

[schmidt-sebastian]

@DamnMerc / @matej-svejda Can you update to v2.2.4 to see if this improves the behavior?

[matej-svejda]

@schmidt-sebastian I upgraded, lets see what happens :)

Do you have any ideas what is causing the connection to get lost in the first place? Im running this on an AWS server, so my guess is connectivity shouldnt be an issue ;)

[schmidt-sebastian]

My theory is that you are either seeing a random connection drops, which can happen even over a strong network link, or you are running into our authentication deadline, in which case our backend terminates your connection (it should however not return GRPC Error Code 13) and we should retry seamlessly.

From your logs, however, it seems like we did not handle these connectivity issues correctly. For every connection drop, it looks like we tried to re-establish the connection twice. This then quickly ballooned, as the second connection drop started 4 retry attempts. Then we have 8 and finally 16. Since our limit is 10, you saw the "Exceeded maximum retries" error when we hit the fourth network drop.

[matej-svejda]

If I understand you correctly, this means that for this error to have happend, connecting should have failed multiple times in short succession (too short in this case). Can this then even be random connection drops or authentification deadlines?

Can I provide you with any more info to find out why the Errror Code 13 happened in the first place?

[schmidt-sebastian]

Unfortunately, there is no metadata for the internal error you are seeing. If these errors occur repeatedly, you can file a Google Support Ticket with your Project ID and get more information from the backend team.