Crash not reported: 413 response code when uploading with large/many breadcrumbs

[edenman]

Platform:

• Android -> compile 29 target 29, device is running 28
• Java -> 1.8/1.8
• Kotlin -> 1.3.72, jvmTarget 1.8

IDE:

• Android Studio -> 4.1 Canary 8

Build system:

• Gradle -> 6.3-all

Android Gradle Plugin:

• Yes -> 4.1.0-alpha08

Sentry Android Gradle Plugin:

• Yes -> If yes, which version?
• No

Proguard/R8:

• Disabled

sentry-android installed with:

• JCenter

The version of sentry-android:
2.0.0

---

I have the following issue:

The description goes here ...

Steps to reproduce:

• Force the app to crash

Actual result:

• No crash is uploaded to Sentry.

Expected result:

• Crash is uploaded to Sentry.

This doesn't happen for all of our crashes, afaict, just some. The error in question is coming from a kotlin-multiplatform library, if that's relevant.

I've got the debugger hooked up, is there any way to gather additional information about why this isn't able to be written to disk quickly enough?

[edenman]

Auto-init: false
Retrieving auto-init from AndroidManifest.xml
Initializing SDK with DSN: 'MY DSN HERE'
GlobalHubMode: 'true'
Scheduled sending cached files from /data/user/0/chat.quill.android/cache/sentry
SendCachedEventFireAndForgetIntegration installed.
Processing 0 items from cache dir /data/user/0/chat.quill.android/cache/sentry
Scheduled sending cached files from /data/user/0/chat.quill.android/cache/sentry
SendCachedEventFireAndForgetIntegration installed.
default UncaughtExceptionHandler class='com.android.internal.os.RuntimeInit$KillApplicationHandler'
'/data/user/0/chat.quill.android/cache/sentry/outbox' is not a file.
UncaughtExceptionHandlerIntegration installed.
Registering EnvelopeFileObserverIntegration for path: /data/user/0/chat.quill.android/cache/sentry/outbox
EnvelopeFileObserverIntegration installed.
NdkIntegration enabled: true 
Processing 0 items from cache dir /data/user/0/chat.quill.android/cache/sentry/outbox
Finished processing cached files from /data/user/0/chat.quill.android/cache/sentry
Finished processing cached files from /data/user/0/chat.quill.android/cache/sentry
NdkIntegration installed.
AnrIntegration enabled: true 
ANR timeout in milliseconds: 4000
AnrIntegration installed.
Uncaught exception received. 
Capturing event: 2338af2f77d3465fb9e607825adb0347
Not possible to read external files directory
Adding Event to offline storage: /data/user/0/chat.quill.android/cache/sentry/2338af2f77d3465fb9e607825adb0347.sentry-event
Invoking inner uncaught exception handler.
Disk flush event fired: 2338af2f77d3465fb9e607825adb0347```

[edenman]

Ooh, fun!

05-19 08:59:08.738 D/Sentry  (17579): 413 Request Entity Too Large
05-19 08:59:08.739 D/Sentry  (17579): The transport failed to send the event with response code 413. Retrying in 60000ms.
05-19 08:59:08.742 E/Sentry  (17579): Event submission failed: db64221962ef47a7990e580c4a867b09
05-19 08:59:08.742 E/Sentry  (17579): java.lang.IllegalStateException: The transport failed to send the event with response code 413. Retrying in 60000ms.
05-19 08:59:08.742 E/Sentry  (17579): 	at io.sentry.core.transport.AsyncConnection$EventSender.flush(AsyncConnection.java:184)
05-19 08:59:08.742 E/Sentry  (17579): 	at io.sentry.core.transport.AsyncConnection$EventSender.run(AsyncConnection.java:139)
05-19 08:59:08.742 E/Sentry  (17579): 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:458)
05-19 08:59:08.742 E/Sentry  (17579): 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
05-19 08:59:08.742 E/Sentry  (17579): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
05-19 08:59:08.742 E/Sentry  (17579): 	at io.sentry.core.transport.RetryingThreadPoolExecutor$AttemptedRunnable.run(RetryingThreadPoolExecutor.java:232)
05-19 08:59:08.742 E/Sentry  (17579): 	at java.util.concurrent.ThreadPoolExecutor.processTask(ThreadPoolExecutor.java:1187)
05-19 08:59:08.742 E/Sentry  (17579): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
05-19 08:59:08.742 E/Sentry  (17579): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
05-19 08:59:08.742 E/Sentry  (17579): 	at java.lang.Thread.run(Thread.java:784)```

[edenman]

Wasn't actually able to repro the "timed out waiting to flush event" issue without the debugger hooked up. However, the 413 issue is 100% repeatable. I can work around it by turning down the maxBreadcrumbs to a uselessly small number! It would sure be great if this SDK was aware of the upload size limit and did something other than silently failing to upload the crash when a 413 is encountered.

[marandaneto]

hey @edenman thanks for raising this.

See here a bit of more context about 413: https://docs.sentry.io/enriching-error-data/additional-data/?platform=python#context-size-limits

So the upload didn't fail, the payload was just too large and the server discarded it.
What would you suggest? ideas for improvements? we'd be glad to hear.

Also, Ignore the logging Retrying in 60000ms, this has been removed from newer versions, it won't be retried as this will be taken care of by the server.

About the timed out waiting to flush event, I'd recommend upgrading the SDK to the latest version (2.1.2 right now), we've increased our SentryOptions.flushTimeoutMillis which works better with larger payloads.

[edenman]

Thanks for the quick response! I would like for this sentence to be true: Sentry will try its best to accommodate the data you send, but Sentry will trim large context payloads or truncate the payloads entirely.

Rather than just failing to record the event, it would be awfully great if Sentry could truncate/trim the breadcrumbs if they are too large. Failing to record an event and not telling me about it is literally the worst thing this SDK can do (aside from crashing my app, I guess).

[marandaneto]

Thanks for the quick response! I would like for this sentence to be true: Sentry will try its best to accommodate the data you send, but Sentry will trim large context payloads or truncate the payloads entirely.

@edenman got your point.
well, this is arguable, as or truncate the payloads entirely. is correct, right.

Failing to record an event and not telling me about it is literally the worst thing this SDK can do (aside from crashing my app, I guess).

Docs mention this and also you've got logs in the SDK, not sure what would be the proper channel for telling you the payload is too large?

Trimming contents have pros and cons, if you trim a JSON, it's gonna be broken.

You may have a single breadcrumb which is 1MB payload, so reducing the number of breadcrumbs is also not a solution, but a best-effort.

Another problem is, to be able to trim something you need to load in memory and that's why we have a hard limit and discard the whole thing if it exceeds that limit, which is 200kb.

I'm not telling we should not do something about it, we are gonna think and see if there's a solution altogether, we are open for ideas, thanks.

[edenman]

Ah, ok. So, a few things:

• truncate the payloads to me indicates that the non-essential parts of the payload would be removed and the issue would still be created. This should probably be clarified in the documentation. Maybe "ignore the payloads entirely" would be more clear?
• Logs in the SDK don't help me at all unless I'm the one encountering the scenario where the app crashes. Lots of crashes happen in the wild and the circumstances where this 413 are encountered might not be easily reproducible.
• The proper channel for telling me the payload is too large, in descending order of goodness:

1. Have the client SDK measure the payload size and since it knows the data format, it can go through and remove breadcrumbs until it's under the limit.
2. Still create an issue, removing the breadcrumbs from the json to get it down under your payload limit. To me it seems like you'd be able to load the first 200kb of the message and then close any quotes/brackets necessary to get it back to being valid json. This seems a little gross but should be workable.
3. Don't do any json mangling, create a dummy "payload was too large" issue
4. In case creating a dummy issue is too difficult, just send me an email. Literally anything is better than dropping this on the floor.

[edenman]

(i am assuming that breadcrumbs are the largest variable as far as payload size so it makes sense to only focus on trimming them. strategies could be adjusted based on whether that's actually true or not)

[marandaneto]

@edenman we are gonna discuss this internally and come up with something.
I'll keep this open, thanks for the feedback :)

[mrmitew]

(i am assuming that breadcrumbs are the largest variable as far as payload size so it makes sense to only focus on trimming them. strategies could be adjusted based on whether that's actually true or not)

@edenman , Actually, attaching all threads and their stack traces contribute also quite a bit to the size of the json. Surely, you can also have a breadcrumb with a ton of information. For now, I guess you could try disabling the thread info.

[marandaneto]

SentryOptions.attachThreads is indeed enabled by default, but attachStacktrace is disabled by default on sentry-android >= 2.0.1 See getsentry/sentry-android#267
so unless you have tons of running threads, breadcrumbs are still the largest part, depending on your usage, of course.

[edenman]

Any movement here? Even after removing some of my larger breadcrumbs, I'm still seeing this sporadically, which makes it impossible to reliably tell when my app is actually encountering errors.

[marandaneto]

hey @edenman we did have some discussions but still in an early stage to define the next steps.

Actually we've decided 2 things already that's worthy to share.

1st is to better document this feature, when it gets truncated or dropped, also about its size limits.
2nd is to not truncate the payload at all on the client-side, this has several downsides.

are you adding breadcrumbs with full payloads or responses? because 200kb data is quite a lot, we'd like to understand the nature of your events, so we'll have a better understanding of what to do next, could you share a few examples?

Thanks.

[edenman]

I'm building a new app, so yeah, I'm in "log as much as possible" mode since I don't have a sense yet of where my crashes/bugs will be coming from. I turned off http response body logging when this first happened. I'm still logging messages that come in over the websocket we're using, which I'm guessing is the cause of the bloat, but that's extremely useful information for me right now.

Can you elaborate on "not truncate the payload at all on the client side" and what the downsides are?