issue with Linking AOT files... take forever for iOS patch

[rashedswen]

App ID: 218d4c39-68d5-4c6d-9df1-7ba35ed338f9

Description

when try to patch IOS I stuck on Linking AOT files which may take a long time (more than 3000 seconds and not finished yet) ,I try Flutter Clean, reinstall Shorebird and all other ways to solve this, and still not solved

Steps To Reproduce

run shorebird patch --platforms=ios --flavor production -t lib/main_production.dart
stuck on

Expected Behavior

finish patching the app with short linking time as always

Screenshots

If applicable, add screenshots to help explain your problem.

Additional Context

stuck here

[Process.run] /Users/user/.shorebird/bin/cache/flutter/6d047e401941206daf11740c523ffa3364a2385b/bin/dart run /Users/user/.shorebird/bin/cache/artifacts/aot-tools/c600950be82dcdad6a46da46bcc6e73773ca3907/aot-tools.dill link --base=/var/folders/72/rbh1p4l15xd0mqwcll7kj7f80000gn/T/umDzle/Products/Applications/Runner.app/Frameworks/App.framework/App --patch=/Users/user/FlutterProjects/app/build/out.aot --analyze-snapshot=/Users/user/.shorebird/bin/cache/flutter/6d047e401941206daf11740c523ffa3364a2385b/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --output=/Users/user/FlutterProjects/app/build/out.vmcode --gen-snapshot=/Users/user/.shorebird/bin/cache/flutter/6d047e401941206daf11740c523ffa3364a2385b/bin/cache/artifacts/engine/ios-release/gen_snapshot_arm64 --kernel=/Users/user/FlutterProjects/app/.dart_tool/flutter_build/fae5ab9b2216fe82dd7d9111b10667eb/app.dill --reporter=json --redirect-to=/Users/user/FlutterProjects/app/build/link.jsonl --dump-debug-info=/var/folders/72/rbh1p4l15xd0mqwcll7kj7f80000gn/T/fDviHI (in /Users/user/FlutterProjects/app/build)

[eseidel]

I seem to recall Felix or Bryan mentioning there was a possible hang in the linker when a subcommand fails?

[rashedswen]

I don't know i just run shorebird patch with verbose it just hanging there for more than 3000 seconds and not work ever

[eseidel]

Thanks. @erickzanardo is investigating as we speak. We're not yet sure where it's hanging.

[rashedswen]

Yesterday it worked with me just one time after deleting everything related to IOS and after that time it won't work again even when repeating same steps

Maybe this will be helpful

[erickzanardo]

@rashedswen can you verify if there is a link.jsonl file in your build folder?

This file can contain informations that might help us debug this.

[rashedswen]

I can't locate it, can you give me possible path?

[erickzanardo]

Hey @rashedswen, I just realized that you will not get those files since the command is hanging before that would move the debug files into the build folder.

I've just landed a PR that will log the location where the debug files will be stored before they are consolidate.

If it is not much of a hassle, could you try running you patch from shorebird in the main channel and verifying if you can grab the debug files with the instruction of the log?

To get in shorebird's main channel should be simple, you just need to enter in your shorebird installation folder (which will be ~/.shorebird) if you used the installation script in our docs, then just do a git checkout main.

Then, when running the patch with -v, keep an eye for a log like this: Dumping link debug info to SOME_FOLDER.

If you can zip the contents of that folder and send it us, we would really appreciate, and we hope that with that information we may be able to find some clues on what is happening.

PS: Don't forget to checkout back to the stable branch afterwards.

Thanks!

[rashedswen]

Ok @erickzanardo i will do it and keep you updated

[rashedswen]

After switching to main and try to patch, and try it several time, still can't see terminal dumping any files related to debug info

[erickzanardo]

After switching to main and try to patch, and try it several time, still can't see terminal dumping any files related to debug info

Thanks for giving it a try, sorry for any trouble, I am not sure why you don't see the log. Going to put more thought in this and will come back to here.

[rashedswen]

Hi, is there any update on this?

[eseidel]

We looked again just now, and don't really have any good theories for why it's hanging. @erickzanardo is investigating adding more debug logging from our side, but it might be quicker to work on it with you if you're able to reproduce?

Would you be interested/willing to debug with us? Or if your project is open source, we're happy to repro from your project directly.

https://calendly.com/eseidel can schedule directly with my calendar, or if you just ping us on Discord: https://discord.gg/shorebird, we're happy to debug interactively with you there (either via text or video call).

[rashedswen]

sorry format response, I will try to make patch today if that doesn't work I will jump into debug session with you, is that okay?
@eseidel @erickzanardo

[eseidel]

We're happy to work with you. https://discord.gg/shorebird is the fastest way to reach us.

[eseidel]

We believe this is resolved. @erickzanardo did work here to better log. We've not heard any reports of this since. Please let us know if that's not the case!

[akwa-peter]

Please I started having this same issue yesterday while trying to push an iOS patch, I have tried all suggestions online but could not resolve it till now, and I am using the latest Shorebird version. I have attached the log file and the error I am getting below

1739524265048_shorebird.log

✗ Failed to link AOT files: aot_tools link --base=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/mObt1H/Products/Applications/Runner.app/Frameworks/App.framework/App --patch=/Users/uceemfb/StudioProjects/Mobile%20App/build/out.aot --analyze-snapshot=/Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --output=/Users/uceemfb/StudioProjects/Mobile%20App/build/out.vmcode --verbose --gen-snapshot=/Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/gen_snapshot_arm64 --kernel=/Users/uceemfb/StudioProjects/Mobile%20App/build/app.dill --reporter=json --redirect-to=/Users/uceemfb/StudioProjects/Mobile%20App/build/link.jsonl --dump-debug-info=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/O6WyUq failed with exit code 255
stdout: [Process] /Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --dump_dispatch_table_link_data=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/mObt1H/Products/Applications/Runner.app/Frameworks/App.framework/App.dt.link --dump_object_pool_link_data=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/mObt1H/Products/Applications/Runner.app/Frameworks/App.framework/App.op.link /var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/mObt1H/Products/Applications/Runner.app/Frameworks/App.framework/App
[Process] /Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --shorebird --out=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/O6WyUq/App.analyze_snapshot.json /var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/mObt1H/Products/Applications/Runner.app/Frameworks/App.framework/App
[Process] /Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --dump_pool /var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/mObt1H/Products/Applications/Runner.app/Frameworks/App.framework/App
[Process] /Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --dump_class_table --out=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/O6WyUq/App.snapshot.class_table.json /var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/mObt1H/Products/Applications/Runner.app/Frameworks/App.framework/App
[Process] /Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --shorebird --out=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/O6WyUq/out.analyze_snapshot.json /Users/uceemfb/StudioProjects/Mobile%20App/build/out.aot

stderr: /Users/uceemfb/.shorebird/bin/cache/flutter/634e1f4587f660b4e74b87e5f05bc8d0bf3a9453/bin/cache/artifacts/engine/ios-release/analyze_snapshot_arm64 --shorebird --out=/var/folders/9x/t5lx9zcs12b3kgjjsq7_qjw00000gn/T/O6WyUq/out.analyze_snapshot.json /Users/uceemfb/StudioProjects/Mobile%20App/build/out.aot

Failure reading snapshot

Unhandled exception:
Exception: analyze_snapshot failed with exit code 255
stdout:
stderr: Failure reading snapshot

#0 SnapshotAnalyzer._run (package:aot_tools/src/snapshot_analyzer.dart:59:7)
#1 SnapshotAnalyzer.dumpSnapshotAnalysis (package:aot_tools/src/snapshot_analyzer.dart:81:5)
#2 SnapshotAnalyzer.dumpSnapshotDebugInfo (package:aot_tools/src/snapshot_analyzer.dart:188:5)
#3 LinkCommand._generateOptimizedPatchSnapshot (package:aot_tools/src/commands/link_command.dart:299:11)
#4 LinkCommand.run (package:aot_tools/src/commands/link_command.dart:140:40)
#5 CommandRunner.runCommand (package:args/command_runner.dart:212:27)
#6 CommandRunner.run. (package:args/command_runner.dart:122:25)
#7 new Future.sync (dart:async/future.dart:313:27)
#8 CommandRunner.run (package:args/command_runner.dart:122:14)
#9 main. (file:///home/gha/.engine_checkout/engine/src/flutter/third_party/dart/pkg/aot_tools/bin/aot_tools.dart:9:57)
#10 _rootRun (dart:async/zone.dart:1414:13)
#11 _CustomZone.run (dart:async/zone.dart:1317:19)
#12 _runZoned (dart:async/zone.dart:1838:10)
#13 runZoned (dart:async/zone.dart:1777:10)
#14 LoggerOverrides.runZoned (package:aot_tools/src/logger.dart:28:18)
#15 runWithLogger (package:aot_tools/src/logger.dart:18:26)
#16 main (file:///home/gha/.engine_checkout/engine/src/flutter/third_party/dart/pkg/aot_tools/bin/aot_tools.dart:9:26)
#17 _delayEntrypointInvocation. (dart:isolate-patch/isolate_patch.dart:295:33)
#18 _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:184:12)
(47.5s)

If you aren't sure why this command failed, re-run with the --verbose flag to see more information.

You can also file an issue if you think this is a bug. Please include the following log file in your report:
/Users/uceemfb/Library/Application Support/shorebird/logs/1739524265048_shorebird.log

@eseidel @erickzanardo