Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inter dynamic library dependencies #7327

Closed
vsarunas opened this issue Feb 8, 2024 · 1 comment
Closed

Inter dynamic library dependencies #7327

vsarunas opened this issue Feb 8, 2024 · 1 comment
Labels
bug

Comments

@vsarunas
Copy link

vsarunas commented Feb 8, 2024 •
edited by MaxDesiatov
Loading

Description

With adoption of Macros, one of the dependencies required to be added is swift-syntax, or other larger projects. Unfortunately this increases build time significantly and there is no simple way to distribute pre-built binaries of larger more complicated projects like swift-syntax.

The one way working way involves creating an xcodeproj using Swift 5.7 via swift package generate-xcodeproj (as it was removed in #5748) and from there the dynamic libraries linking is preserved if using xcodebuild properly (as was done here); but there is no native SwiftPM way of creating libraries/frameworks that achieves the correct dynamically linked libraries that I have found.

The generated xcodeproj preserves the dependencies and linking structure:

xcode_link_deps

Currently there is a feature gap of creating and using pre-built binaries: #7035

Expected behavior

A dynamic library which depends on another library, which too is dynamic should link to it rather than fully embedding it.

The frameworks that are build by Xcode have the following linking between frameworks (from https://github.com/johnno1962/InstantSyntax):

otool -L InstantSyntax/509.1.1/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/SwiftSyntaxMacros 
InstantSyntax/509.1.1/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/SwiftSyntaxMacros (architecture x86_64):
	@rpath/SwiftSyntaxMacros.framework/Versions/A/SwiftSyntaxMacros (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftSyntaxBuilder.framework/Versions/A/SwiftSyntaxBuilder (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftParserDiagnostics.framework/Versions/A/SwiftParserDiagnostics (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftBasicFormat.framework/Versions/A/SwiftBasicFormat (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftParser.framework/Versions/A/SwiftParser (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftDiagnostics.framework/Versions/A/SwiftDiagnostics (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftSyntax.framework/Versions/A/SwiftSyntax (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftSyntax509.framework/Versions/A/SwiftSyntax509 (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 120.100.0, weak)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
	/usr/lib/swift/libswiftDispatch.dylib (compatibility version 1.0.0, current version 34.0.2, weak)
	/usr/lib/swift/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	/usr/lib/swift/libswiftOSLog.dylib (compatibility version 1.0.0, current version 4.0.0, weak)
	/usr/lib/swift/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 8.0.0, weak)
	/usr/lib/swift/libswiftXPC.dylib (compatibility version 1.0.0, current version 29.0.2, weak)
	/usr/lib/swift/libswiftos.dylib (compatibility version 1.0.0, current version 1040.0.0, weak)
	/usr/lib/swift/libswiftFoundation.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
InstantSyntax/509.1.1/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/SwiftSyntaxMacros (architecture arm64):
	@rpath/SwiftSyntaxMacros.framework/Versions/A/SwiftSyntaxMacros (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftSyntaxBuilder.framework/Versions/A/SwiftSyntaxBuilder (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftParserDiagnostics.framework/Versions/A/SwiftParserDiagnostics (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftBasicFormat.framework/Versions/A/SwiftBasicFormat (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftParser.framework/Versions/A/SwiftParser (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftDiagnostics.framework/Versions/A/SwiftDiagnostics (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftSyntax.framework/Versions/A/SwiftSyntax (compatibility version 0.0.0, current version 0.0.0)
	@rpath/SwiftSyntax509.framework/Versions/A/SwiftSyntax509 (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 120.100.0, weak)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
	/usr/lib/swift/libswiftDispatch.dylib (compatibility version 1.0.0, current version 34.0.2, weak)
	/usr/lib/swift/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	/usr/lib/swift/libswiftOSLog.dylib (compatibility version 1.0.0, current version 4.0.0, weak)
	/usr/lib/swift/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 8.0.0, weak)
	/usr/lib/swift/libswiftXPC.dylib (compatibility version 1.0.0, current version 29.0.2, weak)
	/usr/lib/swift/libswiftos.dylib (compatibility version 1.0.0, current version 1040.0.0, weak)
	/usr/lib/swift/libswiftFoundation.dylib (compatibility version 1.0.0, current version 1.0.0, weak)

Actual behavior

A dynamic library which has a dependency on dynamic library embeds all dependencies:

$ otool -L .build/debug/libSwiftSyntaxMacros.dylib 
.build/debug/libSwiftSyntaxMacros.dylib:
	@rpath/libSwiftSyntaxMacros.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 120.100.0, weak)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
	/usr/lib/swift/libswiftDispatch.dylib (compatibility version 1.0.0, current version 34.0.2, weak)
	/usr/lib/swift/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	/usr/lib/swift/libswiftOSLog.dylib (compatibility version 1.0.0, current version 4.0.0, weak)
	/usr/lib/swift/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 8.0.0, weak)
	/usr/lib/swift/libswiftXPC.dylib (compatibility version 1.0.0, current version 29.0.2, weak)
	/usr/lib/swift/libswiftos.dylib (compatibility version 1.0.0, current version 1040.0.0)
	/usr/lib/swift/libswiftFoundation.dylib (compatibility version 1.0.0, current version 1.0.0, weak)

This results in large binary size:

$ du -sh .build/release/libSwiftSyntaxMacros.dylib
 18M	.build/release/libSwiftSyntaxMacros.dylib

While xcodebuild dynamically linked results in:

$ du -sh InstantSyntax/509.1.1/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/SwiftSyntaxMacros 
324K	InstantSyntax/509.1.1/SwiftSyntaxMacros.xcframework/macos-arm64_x86_64/SwiftSyntaxMacros.framework/Versions/A/SwiftSyntaxMacros

Steps to reproduce

  1. git clone -b 509.1.1-dynamic https://github.com/vsarunas/swift-syntax.git
  2. cd swift-syntax
  3. swift build --product SwiftSyntaxMacros
  4. What is built is a dynamic library:
$ file .build/debug/libSwiftSyntaxMacros.dylib 
.build/debug/libSwiftSyntaxMacros.dylib: Mach-O 64-bit dynamically linked shared library arm64
  1. But the dependencies are embedded rather than linked against:
    .target(
      name: "SwiftSyntaxMacros",
      dependencies: ["SwiftDiagnostics", "SwiftParser", "SwiftSyntax", "SwiftSyntaxBuilder"],
      exclude: ["CMakeLists.txt"]
    ),

$ otool -L .build/debug/libSwiftSyntaxMacros.dylib 
.build/debug/libSwiftSyntaxMacros.dylib:
	@rpath/libSwiftSyntaxMacros.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.157.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.9.2)
	/usr/lib/swift/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 120.100.0, weak)
	/usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)
	/usr/lib/swift/libswiftDispatch.dylib (compatibility version 1.0.0, current version 34.0.2, weak)
	/usr/lib/swift/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
	/usr/lib/swift/libswiftOSLog.dylib (compatibility version 1.0.0, current version 4.0.0, weak)
	/usr/lib/swift/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 8.0.0, weak)
	/usr/lib/swift/libswiftXPC.dylib (compatibility version 1.0.0, current version 29.0.2, weak)
	/usr/lib/swift/libswiftos.dylib (compatibility version 1.0.0, current version 1040.0.0)
	/usr/lib/swift/libswiftFoundation.dylib (compatibility version 1.0.0, current version 1.0.0, weak)

Additionally, when using xcodebuild archive to create the library/framework, the linking dependencies are lost too:

xcodebuild archive -scheme SwiftSyntaxMacros -quiet -configuration Release -destination 'generic/platform=macOS' -archivePath /private/tmp/swift-syntax/SwiftSyntaxMacros-macos.xcarchive -derivedDataPath /private/tmp/swift-syntax/SwiftSyntaxMacros-macos.derived SKIP_INSTALL=NO BUILD_LIBRARY_FOR_DISTRIBUTION=YES SWIFT_SERIALIZE_DEBUGGING_OPTIONS=NO

This may be #4781 but it is not clear.

Swift Package Manager version/commit hash

5.9.0

Swift & OS version (output of swift --version ; uname -a)

$ swift --version
swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-5.9.2.2.56 clang-1500.1.0.2.5)
Target: arm64-apple-macosx14.0
@vsarunas vsarunas added the bug label Feb 8, 2024
@neonichu
Copy link
Contributor

neonichu commented Feb 8, 2024

This is #4951, you cannot depends on products in your own package and targets are always linked statically by definition.

@neonichu neonichu closed this as completed Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@neonichu @vsarunas