Adding endpoint to upload android map files #161252

LikeTheSalad · 2023-07-05T12:22:46Z

Summary

We need a way to decode Android crash's stacktraces so that they can provide meaningful insights to our customers, this is because, due to security reasons, android apps tend to obfuscate their code before publishing it online, making crash reports contain obfuscated names, which don't make any sense before mapping them to the actual source code names.

In order to help our customers deobfuscate their stacktraces, we need to allow them to provide us with an R8 map file, which is generated by the code obfuscation tool (R8) at compile time. This map file is needed to later do the deobfuscation process.

So these code changes take care of adding a new endpoint that our customers can use to upload their map files, similarly to what's currently available to RUM Sourcemaps, the Android map files will be uploaded to ES, using the same index as the one currently used to store RUM Sourcemaps.

There's a couple of reasons why a new endpoint to upload android maps is needed instead of re-using the existing RUM Sourcemaps one:

The Sourcemaps upload endpoint has validations in place to check the sourcemap format, which must be a JSON with some expected keys available. Android map files don't have a JSON format, so they are rejected by the sourcemaps endpoint.
Android map files tend to be large in size, just as an example, the map file generated for our sample app has a size of ~7 MB, so for real apps this number can be larger, which would also cause issues with the RUM upload endpoint since it has a max file limit size of 1 MB.
The RUM upload endpoint contains a parameter (bundle_filepath ) that doesn't have an equivalent for the android map use case.

This PR depends on #161152

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

ghost · 2023-07-05T12:23:01Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…-fix'

x-pack/plugins/apm/server/routes/source_maps/route.ts

Co-authored-by: Søren Louv-Jansen <[email protected]>

…/kibana into uploading-android-map-files

…-ref HEAD~1..HEAD --fix'

…/kibana into uploading-android-map-files

LikeTheSalad · 2023-07-06T09:51:39Z

@elasticmachine merge upstream

elasticmachine · 2023-07-06T09:51:54Z

Pinging @elastic/apm-ui (Team:APM)

x-pack/plugins/apm/server/routes/source_maps/route.ts

LikeTheSalad · 2023-07-07T12:54:22Z

@elasticmachine merge upstream

gbamparop · 2023-07-10T09:59:08Z

x-pack/plugins/apm/server/routes/source_maps/route.ts

  },
 });

+const uploadAndroidMapRoute = createApmServerRoute({


Could you please add docs for this one? (Example: https://www.elastic.co/guide/en/kibana/current/apm-api.html)

Thanks @gbamparop - I've just added docs for the overall Android source map API. There are lots of similarities with the RUM one, which is why I took a lot of the parts involved in that part of the doc file. I'm not sure if my approach is the right one, so please have a look just in case there might be a better way to handle it. I'm also looping @bmorelli25 in, in case he's got some inputs as well.

Co-authored-by: Brandon Morelli <[email protected]>

bmorelli25 · 2023-07-11T16:59:42Z

@elasticmachine, run elasticsearch-ci/docs

LikeTheSalad · 2023-07-24T12:08:58Z

@elasticmachine merge upstream

LikeTheSalad · 2023-07-24T14:15:59Z

@elasticmachine merge upstream

kpatticha · 2023-07-31T07:26:30Z

x-pack/plugins/apm/server/routes/source_maps/route.ts

  options: {
    tags: ['access:apm', 'access:apm_write'],
-    body: { accepts: ['multipart/form-data'] },
+    body: { accepts: ['multipart/form-data'], maxBytes: 100 * 1024 * 1024 },


Does it override the option https://www.elastic.co/guide/en/kibana/current/settings.html#server-maxPayload?

According to the core team, this option sets the limit at the route level but there is no option for the user to increase the value. We need to introduce a new config setting for the endpoint.

I think we can proceed with the current approach and handle it in a follow-up work if is needed

x-pack/plugins/apm/server/routes/source_maps/route.ts

kpatticha · 2023-07-31T09:26:49Z

x-pack/plugins/apm/server/routes/source_maps/route.ts

+          fleetId: artifact.id,
+          created: artifact.created,
+          mapContent: sourceMapContent,
+          bundleFilepath: bundleFilePath.toString(),


bundleFilePath is a string. Is this needed?

You're right, I think it's not needed. I've just removed it 👍

kpatticha

LGTM just left 2 comments

Co-authored-by: Katerina Patticha <[email protected]>

LikeTheSalad · 2023-07-31T12:00:36Z

@elasticmachine merge upstream

…/kibana into uploading-android-map-files

…-ref HEAD~1..HEAD --fix'

kibana-ci · 2023-07-31T16:18:19Z

💚 Build Succeeded

Buildkite Build
Commit: 4d6160c

Metrics [docs]

✅ unchanged

History

💔 Build #145990 failed 083ed70
💔 Build #145921 failed 9bccc1d
💛 Build #144329 was flaky 7c34a1a
💔 Build #144274 failed 04dae25
💛 Build #141245 was flaky db098dd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

## Summary We need a way to decode Android crash's stacktraces so that they can provide meaningful insights to our customers, this is because, due to security reasons, android apps tend to obfuscate their code before publishing it online, making crash reports contain obfuscated names, which don't make any sense before mapping them to the actual source code names. In order to help our customers deobfuscate their stacktraces, we need to allow them to provide us with an R8 map file, which is generated by the code obfuscation tool (R8) at compile time. This map file is needed to later do the deobfuscation process. So these code changes take care of adding a new endpoint that our customers can use to upload their map files, similarly to what's currently available to [RUM Sourcemaps](https://www.elastic.co/guide/en/apm/guide/current/source-map-how-to.html#source-map-rum-upload), the Android map files will be uploaded to ES, using the same index as the one currently used to store RUM Sourcemaps. There's a couple of reasons why a new endpoint to upload android maps is needed instead of re-using the existing RUM Sourcemaps one: * The Sourcemaps upload endpoint has validations in place to check the sourcemap format, which must be a JSON with some expected keys available. Android map files don't have a JSON format, so they are rejected by the sourcemaps endpoint. * Android map files tend to be large in size, just as an example, the map file generated for our [sample app](https://github.com/elastic/opbeans-android) has a size of ~7 MB, so for real apps this number can be larger, which would also cause issues with the RUM upload endpoint since it has a max file limit size of 1 MB. * The RUM upload endpoint contains a parameter (`bundle_filepath `) that doesn't have an equivalent for the android map use case. This PR depends on elastic#161152 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Søren Louv-Jansen <[email protected]> Co-authored-by: Brandon Morelli <[email protected]> Co-authored-by: Katerina Patticha <[email protected]>

gbamparop · 2023-08-31T10:15:23Z

Tested on edge-lite by uploading an android sourcemap.

LikeTheSalad added 3 commits July 5, 2023 11:00

Adding android upload map validation

7e1ebf3

Adding same service name and version android map files

fb4002b

Setting android upload map endpoint body max size to 30 MB

a995b4f

kibanamachine and others added 4 commits July 5, 2023 13:01

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

f670716

…-fix'

Fixing lint issues

6ea4d39

Merge branch 'main' into uploading-android-map-files

91e0b8b

Fixing type issues

271b757

sorenlouv reviewed Jul 5, 2023

View reviewed changes

x-pack/plugins/apm/server/routes/source_maps/route.ts Show resolved Hide resolved

LikeTheSalad and others added 6 commits July 6, 2023 09:32

Adding | undefined to test return type

e05b441

Update x-pack/plugins/apm/server/routes/source_maps/route.ts

4366939

Co-authored-by: Søren Louv-Jansen <[email protected]>

Merge branch 'uploading-android-map-files' of github.com:LikeTheSalad…

5dceb1c

…/kibana into uploading-android-map-files

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

1eaef79

…-ref HEAD~1..HEAD --fix'

Fixing maxBytes value for sourcemaps upload endpoint

a593a12

Merge branch 'uploading-android-map-files' of github.com:LikeTheSalad…

03bbf95

…/kibana into uploading-android-map-files

LikeTheSalad marked this pull request as ready for review July 6, 2023 09:33

LikeTheSalad requested a review from a team as a code owner July 6, 2023 09:33

LikeTheSalad requested a review from AlexanderWert July 6, 2023 09:34

LikeTheSalad added v8.9.0 v8.10.0 and removed v8.9.0 labels Jul 6, 2023

Merge branch 'main' into uploading-android-map-files

588fa1a

botelastic bot added the Team:APM - DEPRECATED Use Team:obs-ux-infra_services. label Jul 6, 2023

LikeTheSalad added the release_note:enhancement label Jul 6, 2023

sorenlouv reviewed Jul 6, 2023

View reviewed changes

x-pack/plugins/apm/server/routes/source_maps/route.ts Outdated Show resolved Hide resolved

Merge branch 'main' into uploading-android-map-files

20a0e32

gbamparop reviewed Jul 10, 2023

View reviewed changes

LikeTheSalad and others added 3 commits July 11, 2023 10:42

Update docs/apm/api.asciidoc

7dc2d44

Co-authored-by: Brandon Morelli <[email protected]>

Update docs/apm/api.asciidoc

04ad479

Co-authored-by: Brandon Morelli <[email protected]>

Update docs/apm/api.asciidoc

673eb18

Co-authored-by: Brandon Morelli <[email protected]>

bmorelli25 requested a review from a team July 11, 2023 16:59

Merge branch 'main' into uploading-android-map-files

04dae25

Merge branch 'main' into uploading-android-map-files

7c34a1a

LikeTheSalad mentioned this pull request Jul 26, 2023

Adding R8 deobfuscation to otlp batch processors elastic/apm-server#11005

Closed

3 tasks

kpatticha reviewed Jul 31, 2023

View reviewed changes

x-pack/plugins/apm/server/routes/source_maps/route.ts Outdated Show resolved Hide resolved

kpatticha reviewed Jul 31, 2023

View reviewed changes

kpatticha approved these changes Jul 31, 2023

View reviewed changes

LikeTheSalad and others added 2 commits July 31, 2023 13:57

Update x-pack/plugins/apm/server/routes/source_maps/route.ts

c774db8

Co-authored-by: Katerina Patticha <[email protected]>

Removing unnecessary string conversion

292b9ec

kibanamachine and others added 5 commits July 31, 2023 08:00

Merge branch 'main' into uploading-android-map-files

9bccc1d

Fixing typo

0676b19

Merge branch 'uploading-android-map-files' of github.com:LikeTheSalad…

083ed70

…/kibana into uploading-android-map-files

Renaming var to match type's name

37c7143

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

4d6160c

…-ref HEAD~1..HEAD --fix'

LikeTheSalad merged commit 6751324 into elastic:main Aug 1, 2023

kibanamachine added the backport:skip This PR does not require backporting label Aug 1, 2023

gbamparop added the apm:test-plan-8.10.0 label Aug 7, 2023

gbamparop added the apm:test-plan-done Pull request that was successfully tested during the test plan label Aug 31, 2023

sorenlouv mentioned this pull request Aug 31, 2023

[APM] Disable android source map API and remove docs #165393

Closed

Adding endpoint to upload android map files #161252

Adding endpoint to upload android map files #161252

Uh oh!

Conversation

LikeTheSalad commented Jul 5, 2023

Summary

Checklist

For maintainers

Uh oh!

ghost commented Jul 5, 2023

🤖 GitHub comments

Uh oh!

Uh oh!

LikeTheSalad commented Jul 6, 2023

Uh oh!

elasticmachine commented Jul 6, 2023

Uh oh!

Uh oh!

LikeTheSalad commented Jul 7, 2023

Uh oh!

gbamparop Jul 10, 2023

Choose a reason for hiding this comment

Uh oh!

LikeTheSalad Jul 10, 2023

Choose a reason for hiding this comment

Uh oh!

bmorelli25 commented Jul 11, 2023

Uh oh!

LikeTheSalad commented Jul 24, 2023

Uh oh!

LikeTheSalad commented Jul 24, 2023

Uh oh!

kpatticha Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

kpatticha Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kpatticha Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

LikeTheSalad Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

kpatticha left a comment

Choose a reason for hiding this comment

Uh oh!

LikeTheSalad commented Jul 31, 2023

Uh oh!

kibana-ci commented Jul 31, 2023

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

gbamparop commented Aug 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants