feat: support --obfuscate on iOS

[psatler]

Description

Running shorebird patch ios-alpha on an app that uses flavors fails with the following message

Failed to link AOT files: Exception: Failed to link: base and patch snapshots have differing VM sections

I created a repo and manually added flavors/schemes to iOS (development and production). The repo I used to reproduce the issue can be found here.
I was also able to reproduce this issue using the very good CLI which automatically sets up an app with flavors from the get-go when running very_good create flutter_app <app-name>

Android release and patch are working fine.

Steps To Reproduce

On a project that uses flavors/schemes

1. Trigger a release for a given flavor, let's say development:

shorebird release ios-alpha \
        --target lib/main_development.dart \
        --flavor development -- \
        --obfuscate --split-debug-info=./build/app/outputs/symbols

2. After that release is published, trigger a patch release this time

shorebird patch ios-alpha \
        --target lib/main_development.dart \
        --flavor development -- \
        --obfuscate --split-debug-info=./build/app/outputs/symbols

3. Verify that the patch fails

Again, a reproducible repo was created here.

Expected Behavior

To be able to publish a patch release for the respective flavor/target.

Screenshots

• initializing shorebird
  

• triggering a release
  

• triggering a patch
  

Additional Context

• shorebird doctor:

Shorebird 0.21.1 • [email protected]:shorebirdtech/shorebird.git
Flutter 3.16.5 • revision 4e8a7c746ae6f10951f3e676f10b82b21d7300a5
Engine • revision 001b60083b642c134c62e7c10bc34f188b0a2869

✓ Shorebird is up-to-date (0.6s)
✓ Flutter install is correct (0.5s)
✓ AndroidManifest.xml files contain INTERNET permission (35ms)
✓ Has access to storage.googleapis.com (0.2s)

No issues detected!

• flutter doctor

❯ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.16.5, on macOS 14.2.1 23C71 darwin-arm64, locale en-BR)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1)
[✓] Xcode - develop for iOS and macOS (Xcode 15.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.3)
[✓] VS Code (version 1.85.1)
[✓] Connected device (4 available)
[✓] Network resources

• No issues found!

[bryanoltman]

Reproduced this locally. The issue seems to be that our linker does not currently work with obfuscated builds.

[eseidel]

We could also just disable the linker for obfuscated builds for now?

[psatler]

interesting @bryanoltman , thanks for the feedback.

I am not sure if it's worth mentioning, but I believe I was able to successfully execute the patch ios-alpha command along with the obfuscation flags when I used a previous version of flutter, more precisely, version 3.10.6. See this link here, please. This is a repo I created a while ago when I was first experimenting Shorebird.

• lines where I use the obfuscation flags: patch and release

I hope this helps to figure out the issue.

[eseidel]

Yes, obfuscation is expected to work on Flutter 3.16.4 and before. We broke it (accidentally) in 3.16.5 (Shorebird 0.20.0, end of Dec 2023) with our new iOS runtime + linking.

[eseidel]

I suspect that this will have a similar fix to what @felangel is working on the dart compiler now. He's making the dart compiler (gen_snapshot) aware of the existence of the "release" build when building the patch. Making obfuscation work with our linker will probably be similar.

However the way this is failing (vm sections differing) may just suggest that obfuscation needs to be disabled for the vm section of the snapshot, since the vm section is the same for all apps compiled with the same Dart compiler, there isn't a lot of purpose in obfuscating that code.

However obfuscation will probably break our linker in other ways, since it renames methods and variables which likely will cause their heap indices to change, which will likely cause all code which references those names to fail to link (for similar reasons to what Felix is working on right now).

This will probably take us a few weeks to fix and will be after we fix #1665

[ZahidTekbas]

I'm having the same issue here.

[eseidel]

We're aware that the --obfuscate flag is broken for iOS builds at the moment. Android works without issue. This issue tracks fixing --obfuscate on iOS.

[lukemmtt]

+1 for this!

[eseidel]

@felangel is finishing the last patch for #1892 and then plans to work on this.

Work here involves trying shorebird release ios --obfuscate, shorebird patch ios --obfuscate and see what breaks.
Assuming that --obfuscate is deterministic (same inputs produce same outputs) then it should already "just work" with shorebird patch and we'd need to investigate why it doesn't.
If --obfuscate is not deterministic, then we probably need to change Dart's built-in obfuscate to be deterministic, which may involve having it take a "seed" value for whatever random/hashing it uses for the plain_name->obfuscated_name mappings.

It would be OK (as a v1) for --obfuscate to "work" but just cause very low link percentages (because the obfuscation might confuse how we link code between the patch and the release). We can later come back with a "v2" which teaches our linker to be aware of obfuscated selectors, etc.

[wemgl]

Following along for updates on this. I'm temporarily disabling the --obfuscate flag so I use use Shorebird, but would love to re-enable it at some point in the future.

[eseidel]

Wanted to give an update that we've not forgotten about this, just been busy fixing other areas of the product that block more people.

We recognize that obfuscation is a regulatory requirement for some businesses, and thus blocks Shorebird adoption for those companies. We intend to resolve this issue within the next couple months. Thank you for your continued patience.

[eseidel]

We recently resolved one of the underlying blockers here, and I expect we'll pick this up soon and finish the work necessary to resolve this.

[tienngang]

I'm also waiting for this feature. I hope Shorebird will fix and get it working again soon. Thanks for the update and I'll continue to follow up on this.

[derekpitts28]

Bump. This is definitely a problem to be solved before we adopt Shorebird but overall it looks like a promising solution, keep up the good work!!

[eseidel]

Thanks. We know how to solve it, it's just not yet risen to the top of the list. Lots of customers adopting Shorebird every month has kept us quite busy supporting them. We'll get back to this, just probably a few months away still.

[eseidel]

In terms of more detail. What we will do:

There is an obfuscator inside Dart:
https://github.com/dart-lang/sdk/blob/3.6.0/runtime/vm/compiler/aot/precompiler.cc#L3799
Which does record some state information. But we'll probably need to modify it to record a bit more, so that we can use the exact same obfuscation names when compiling the patch that were used during compilation of the release. That's important to our solution since one of the things we have to do as part of preparing a patch on iOS is to map function-to-function so we can decide which functions we can execute from the "patch" vs. which can be used from the "release" (compiled/signed code inside the IPA).

In any case, we've built similar systems for other parts of the Dart compiler, so we have a good idea of what we will do here, we just need to finish other things on our list to get to this one.

Thank you all for your patience. We look forward to supporting you as users of Shorebird in the future.