Qt5 spring cleaning

[K900]

Description of changes

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[vcunat]

error: undefined variable 'apple_sdk_11_0'
at pkgs/top-level/all-packages.nix:22493:40

(on x86_64-darwin)

[vcunat]

qt5.qtbase evals now for me but won't build:

In file included from /private/tmp/nix-build-qtbase-5.15.8.drv-0/qtbase-9cf586d/qmake/generators/mac/pbuilder_pbx.cpp:45:
In file included from /nix/store/a2sd4amfsljdf4pxnx5i1alb1mgg1px4-apple-framework-ApplicationServices-11.0.0/Library/Frameworks/ApplicationServices.framework/Headers/ApplicationServices.h:23:
In file included from /nix/store/1wzdsghj5nk25xbrf715izgz6z62dlsp-apple-framework-CoreServices-11.0.0/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:19:
In file included from /nix/store/pijg4pkqizi263fp9l023k5a91245qf6-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:60:
In file included from /nix/store/pijg4pkqizi263fp9l023k5a91245qf6-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFPropertyList.h:17:
In file included from /nix/store/pijg4pkqizi263fp9l023k5a91245qf6-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFStream.h:16:
/nix/store/pijg4pkqizi263fp9l023k5a91245qf6-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1189:83: error: expected ','
    kCFURLBookmarkCreationWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0))  API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 11 ), // Mac OS X 10.7.3 and later, include information in the bookmark data which al>
                                                                                  ^
/nix/store/pijg4pkqizi263fp9l023k5a91245qf6-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1190:98: error: expected ','
    kCFURLBookmarkCreationSecurityScopeAllowOnlyReadAccess API_AVAILABLE(macos(10.7), macCatalyst(13.0)) API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 12 ), // Mac OS X 10.7.3 and later, if used with kCFURLBookmarkCreation>
                                                                                                 ^
/nix/store/pijg4pkqizi263fp9l023k5a91245qf6-apple-framework-CoreFoundation-11.0.0/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:1200:85: error: expected ','
    kCFURLBookmarkResolutionWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0)) API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 10 ), // Mac OS X 10.7.3 and later, use the secure information included at creation >
                                                                                    ^

[K900]

Progress: it now fails in identical ways on x86_64-darwin and aarch64-darwin, which means we should be good on the SDK upgrade. The bad news is, it fails because qkvgen segfaults, and I have no idea where to even start debugging it.

[vcunat]

And I suppose it's not easy to avoid the cleaning and just fix the build? (which regressed on darwin on the qt5 patch update)

[K900]

We'd have to rebase all the Darwin patches again, and I really don't want to do that tbh. Upstream is expecting at least 10.13, we can't just keep patching things out forever.

[vcunat]

Let's exercise ofBorg at least a bit:
@ofborg build qt5.qtbase

[K900]

Current status:

• x86_64-linux builds and runs up to Plasma bits
• aarch64-linux builds up to qtbase, rest TBD, but these changes shouldn't affect it much
• Darwin (both flavors) builds up to qttools + qqc2, qtwebengine currently blocked on Mesa, waiting for @winterqt to run-test qttools

[tricktron]

I cherry-picked your three commits and the qt5: 5.15.7 -> 5.15.8 update commit onto master so that I don't have to rebuild 500 package to build qtbase.

However on my aarch64-darwin machine I always run into the following issue:

WARNING: Failure to find: /private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui/qt5gui_metatypes.json
make[2]: Entering directory '/private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui'
/private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui/qvkgen_wrapper.sh vulkan/vk.xml /private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/header.LGPL vulkan/qvulkanfunctions
make[2]: *** [Makefile:2749: vulkan/qvulkanfunctions.h] Segmentation fault: 11
make[2]: *** Deleting file 'vulkan/qvulkanfunctions.h'
make[2]: Leaving directory '/private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui'
make[1]: *** [Makefile:533: sub-gui-make_first] Error 2
make[1]: Leaving directory '/private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src'
make: *** [Makefile:54: sub-src-make_first] Error 2
note: keeping build directory '/private/tmp/nix-build-qtbase-5.15.8.drv-3'
error: builder for '/nix/store/s124k9ap6zsvyb98hbfh8d6d71g5mh5c-qtbase-5.15.8.drv' failed with exit code 2;
       last 10 log lines:
       > cd gui/ && ( test -e Makefile || /private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/bin/qmake -o Makefile /private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui/gui.pro ) && make -f Makefile
       > WARNING: Failure to find: /private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui/qt5gui_metatypes.json
       > make[2]: Entering directory '/private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui'
       > /private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui/qvkgen_wrapper.sh vulkan/vk.xml /private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/header.LGPL vulkan/qvulkanfunctions
       > make[2]: *** [Makefile:2749: vulkan/qvulkanfunctions.h] Segmentation fault: 11
       > make[2]: *** Deleting file 'vulkan/qvulkanfunctions.h'
       > make[2]: Leaving directory '/private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src/gui'
       > make[1]: *** [Makefile:533: sub-gui-make_first] Error 2
       > make[1]: Leaving directory '/private/tmp/nix-build-qtbase-5.15.8.drv-3/qtbase-9cf586d/src'
       > make: *** [Makefile:54: sub-src-make_first] Error 2
       For full logs, run 'nix log /nix/store/s124k9ap6zsvyb98hbfh8d6d71g5mh5c-qtbase-5.15.8.drv'.

[K900]

Disable the sandbox.

[K900]

If you're building on native x86_64, that may make libepoxy work, and yes, that error looks like you'll need an SDK upgrade.

[risicle]

I thought so too, but it doesn't appear to solve it 🤷 . I'm just working on the security bump #210742 so addressing it isn't my highest priority.

[ilya-fedin]

kCFURLBookmarkCreationWithSecurityScope API_AVAILABLE(macos(10.7), macCatalyst(13.0)) API_UNAVAILABLE(ios, watchos, tvos) = ( 1UL << 11 ), // Mac OS X 10.7.3 and later, include information in the bookmark data which allows the same sandboxed process to access the resource after being relaunched

I seem to have the same errors (with other package though) on GitHub builders when building my NUR packages for binary cache
https://github.com/ilya-fedin/nur-repository/actions/runs/3954209355/jobs/6771316602

[K900]

Try switching it to SDK 11?

[ilya-fedin]

Isn't that what this PR does? I don't quite understand what should I do to switch otherwise as paths in the errors look like SDK 11 is already used.

[K900]

Those errors usually show up when you mix SDK versions, so maybe you're linking against SDK 10 frameworks or something. Got a link to the expression? Or is it just the default nixpkgs one?

[ilya-fedin]

Never mind, got it

[ilya-fedin]

Do I understand right that various Qt applications are broken on Darwin now? As the package I get errors for is also present in nixpkgs, so it should've been start failing in nixpkgs as well as you haven't changed it in your PR and I'm its maintainer.

[K900]

I've fixed everything I noticed during the staging cycle, but there have been some regressions unfortunately :( It specifically affects the applications that either don't use libsForQt5.callPackage or pass in their Darwin frameworks directly.

[ilya-fedin]

or pass in their Darwin frameworks directly

Is there other way to pass Darwin frameworks?

[K900]

You should be able to just take darwin and then access individual attributes on that, but I'm not sure if this is a good idea. Ideally we'd probably have some other way of handling that...

[ilya-fedin]

You should be able to just take darwin and then access individual attributes on that

But it would need to specify sdk version anyway as the frameworks are in darwin.(apple_sdk|apple_sdk_11_0).frameworks, just inside the derivation?

[K900]

Sorry, apple_sdk, not darwin.

[ilya-fedin]

Sorry, apple_sdk, not darwin.

And then libsForQt5.callPackage will give apple_sdk_11_0 instead?

[K900]

Yep. I believe it already does.

[ilya-fedin]

That's interesting, but it seems only a few derivations in nixpkgs are doing like that and none of them are Qt derivations...

[ilya-fedin]

Something seem to be wrong with SDK 11 as it throws at me errors about various system types and constants being not defined

/tmp/nix-build-kotatogram-desktop-1.4.9.drv-0/source/Telegram/lib_webrtc/webrtc/mac/webrtc_media_devices_mac.mm:25:41: error: use of undeclared identifier 'AudioObjectPropertyAddress'
auto AudioOutputDevicePropertyAddress = AudioObjectPropertyAddress{
                                        ^

 /tmp/nix-build-kotatogram-desktop-1.4.9.drv-0/source/Telegram/lib_webrtc/webrtc/mac/webrtc_media_devices_mac.mm:59:18: error: use of undeclared identifier 'kAudioDeviceUnknown'
        if (deviceId == kAudioDeviceUnknown) {
                        ^

Isn't it possible to choose to stay on old SDK for a package?

[K900]

Qt upstream requires a newer SDK version now. We used to patch it to make it build with 10.12, but the recent Qt updates have raised the requirements again, and trying to backport stuff forever just isn't sustainable.

[ilya-fedin]

nixpkgs honestly has a weird workflow with the SDKs, most projects use latest SDK with MACOSX_DEPLOYMENT_TARGET being set to the needed minimum supported macOS version.

I wonder why it throws those errors at me thoguh, the code builds just fine outside of nixpkgs, using Xcode cmake generator and AppleClang. Is it possible to build using AppleClang within nixpkgs?

[K900]

I'm not really familiar with the details of Darwin stuff in nixpkgs, so I don't have the full full picture, but the reason we don't have the latest SDKs is because they're a pain to package in a way that works with the Nix sandbox.

[ilya-fedin]

Do you know where I can find someone who familiar? As I'm out of ideas how to make it build now...

[K900]

Maybe try #macos:nixos.org on Matrix. I've linked this thread there too.

[wegank]

Something seem to be wrong with SDK 11 as it throws at me errors about various system types and constants being not defined

/tmp/nix-build-kotatogram-desktop-1.4.9.drv-0/source/Telegram/lib_webrtc/webrtc/mac/webrtc_media_devices_mac.mm:25:41: error: use of undeclared identifier 'AudioObjectPropertyAddress'
auto AudioOutputDevicePropertyAddress = AudioObjectPropertyAddress{
                                        ^

/tmp/nix-build-kotatogram-desktop-1.4.9.drv-0/source/Telegram/lib_webrtc/webrtc/mac/webrtc_media_devices_mac.mm:59:18: error: use of undeclared identifier 'kAudioDeviceUnknown'
       if (deviceId == kAudioDeviceUnknown) {
                       ^

Isn't it possible to choose to stay on old SDK for a package?

sed -i "13i#import <CoreAudio/CoreAudio.h>" Telegram/lib_webrtc/webrtc/mac/webrtc_media_devices_mac.mm

[ilya-fedin]

sed -i "13i#import <CoreAudio/CoreAudio.h>" Telegram/lib_webrtc/webrtc/mac/webrtc_media_devices_mac.mm

🤔 I wonder how this worked with old SDK and how this works with Xcode-based build