Skip to content

python312Packages.kaleido: init at 0.2.1#339136

Merged
natsukium merged 5 commits intoNixOS:masterfrom
Pandapip1:init-python3packages-kaleido
Sep 5, 2024
Merged

python312Packages.kaleido: init at 0.2.1#339136
natsukium merged 5 commits intoNixOS:masterfrom
Pandapip1:init-python3packages-kaleido

Conversation

@Pandapip1
Copy link
Member

@Pandapip1 Pandapip1 commented Sep 3, 2024

Description of changes

I'm sorry, I literally couldn't figure out how to do a source build. If someone would like to figure it out for themselves, uh, good luck!

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@Pandapip1 Pandapip1 requested a review from natsukium as a code owner September 3, 2024 00:56
@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Sep 3, 2024
@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Sep 3, 2024
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the third attempt, as far as I know, but no one has been able to build it from the source.😢

@Pandapip1 Pandapip1 force-pushed the init-python3packages-kaleido branch 4 times, most recently from 4ce530b to f30a433 Compare September 3, 2024 04:48
@Pandapip1 Pandapip1 changed the title kaleido: init at 0.2.1.post1 kaleido: init at 0.2.1 Sep 3, 2024
@Pandapip1 Pandapip1 force-pushed the init-python3packages-kaleido branch from 6135aa7 to 8cf971e Compare September 3, 2024 05:07
@Pandapip1 Pandapip1 force-pushed the init-python3packages-kaleido branch 3 times, most recently from 9add00d to a411299 Compare September 3, 2024 05:12
@ofborg ofborg bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. and removed 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Sep 3, 2024
@eclairevoyant eclairevoyant changed the title kaleido: init at 0.2.1 python312Packages.kaleido: init at 0.2.1 Sep 3, 2024
@eclairevoyant
Copy link
Contributor

Also please use the full attrname in commits so that borgo knows what to build (i.e. python312Packages.kaleido and same with plotly)

@Pandapip1 Pandapip1 force-pushed the init-python3packages-kaleido branch from a411299 to 78d34e5 Compare September 3, 2024 14:17
@Pandapip1
Copy link
Member Author

Just noticed there weren't any maintainers for plotly, so this PR also has me adopting it.

@Pandapip1 Pandapip1 added the 0.kind: package adoption Requests or PRs for adopting packages that have no maintainers label Sep 3, 2024
@eclairevoyant
Copy link
Contributor

eclairevoyant commented Sep 3, 2024

please use python312Packages, not python3Packages, in the commit message.

@natsukium
Copy link
Member

@ofborg build python312Packages.kaleido python311Packages.kaleido

@natsukium
Copy link
Member

Tests fail on darwin with the following log.
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Critical error: required built-in appearance SystemAppearance not found'

I suspect CoreFoundation, AppKit, or the matplotlib environment variable MPLBACKEND, but I can't fix it.
Maybe someone at @NixOS/darwin-maintainers knows how to make it work.

Details
kaleido-tests-aarch64-darwin> building '/nix/store/r6dib0z8rc7pbjhfqrh184a6vh9k9a8h-kaleido-tests.drv'
kaleido-tests-aarch64-darwin> Traceback (most recent call last):
kaleido-tests-aarch64-darwin>   File "/nix/store/i36nyn014clgrvrgmv60vg58wl0ifmvn-tests.py", line 11, in <module>
kaleido-tests-aarch64-darwin>     fig.write_image(outfile, engine="kaleido")
kaleido-tests-aarch64-darwin>   File "/nix/store/k4hlqc8dcdw3890fvghzl3sppwfcs78p-python3.12-plotly-5.23.0/lib/python3.12/site-packages/plotly/basedatatypes.py", line 3835, in write_image
kaleido-tests-aarch64-darwin>     return pio.write_image(self, *args, **kwargs)
kaleido-tests-aarch64-darwin>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/k4hlqc8dcdw3890fvghzl3sppwfcs78p-python3.12-plotly-5.23.0/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 266, in write_image
kaleido-tests-aarch64-darwin>     img_data = to_image(
kaleido-tests-aarch64-darwin>                ^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/k4hlqc8dcdw3890fvghzl3sppwfcs78p-python3.12-plotly-5.23.0/lib/python3.12/site-packages/plotly/io/_kaleido.py", line 143, in to_image
kaleido-tests-aarch64-darwin>     img_bytes = scope.transform(
kaleido-tests-aarch64-darwin>                 ^^^^^^^^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/scopes/plotly.py", line 153, in transform
kaleido-tests-aarch64-darwin>     response = self._perform_transform(
kaleido-tests-aarch64-darwin>                ^^^^^^^^^^^^^^^^^^^^^^^^
kaleido-tests-aarch64-darwin>   File "/nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/scopes/base.py", line 293, in _perform_transform
kaleido-tests-aarch64-darwin>     self._ensure_kaleido()
kaleido-tests-aarch64-darwin>   File "/nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/scopes/base.py", line 198, in _ensure_kaleido
kaleido-tests-aarch64-darwin>     raise ValueError(message)
kaleido-tests-aarch64-darwin> ValueError: Failed to start Kaleido subprocess. Error stream:
kaleido-tests-aarch64-darwin> 
kaleido-tests-aarch64-darwin> [0905/000022.334097:WARNING:resource_bundle.cc(431)] locale_file_path.empty() for locale en-US
kaleido-tests-aarch64-darwin> *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Critical error: required built-in appearance SystemAppearance not found'
kaleido-tests-aarch64-darwin> *** First throw call stack:
kaleido-tests-aarch64-darwin> (
kaleido-tests-aarch64-darwin> 	0   CoreFoundation                      0x0000000186e662ec __exceptionPreprocess + 176
kaleido-tests-aarch64-darwin> 	1   libobjc.A.dylib                     0x000000018694a158 objc_exception_throw + 60
kaleido-tests-aarch64-darwin> 	2   CoreFoundation                      0x0000000186e661dc +[NSException exceptionWithName:reason:userInfo:] + 0
kaleido-tests-aarch64-darwin> 	3   AppKit                              0x000000018a61e214 __33+[NSAppearance _initializeCoreUI]_block_invoke + 88
kaleido-tests-aarch64-darwin> 	4   libdispatch.dylib                   0x0000000186b613e8 _dispatch_client_callout + 20
kaleido-tests-aarch64-darwin> 	5   libdispatch.dylib                   0x0000000186b62c68 _dispatch_once_callout + 32
kaleido-tests-aarch64-darwin> 	6   AppKit                              0x000000018a63e5cc +[NSAppearance _aquaAppearance] + 60
kaleido-tests-aarch64-darwin> 	7   AppKit                              0x000000018a61d98c +[NSAppearance appearanceNamed:] + 32
kaleido-tests-aarch64-darwin> 	8   AppKit                              0x000000018a61d124 -[NSSystemAppearanceProxy init] + 124
kaleido-tests-aarch64-darwin> 	9   AppKit                              0x000000018a61d098 __38+[NSSystemAppearanceProxy systemProxy]_block_invoke + 24
kaleido-tests-aarch64-darwin> 	10  libdispatch.dylib                   0x0000000186b613e8 _dispatch_client_callout + 20
kaleido-tests-aarch64-darwin> 	11  libdispatch.dylib                   0x0000000186b62c68 _dispatch_once_callout + 32
kaleido-tests-aarch64-darwin> 	12  AppKit                              0x000000018a61d07c +[NSSystemAppearanceProxy systemProxy] + 64
kaleido-tests-aarch64-darwin> 	13  AppKit                              0x000000018a61d008 -[NSApplication(NSApplicationAppearance_Internal) _registerForAppearanceNotifications] + 32
kaleido-tests-aarch64-darwin> 	14  AppKit                              0x000000018a61acec -[NSApplication init] + 904
kaleido-tests-aarch64-darwin> 	15  AppKit                              0x000000018a61a798 +[NSApplication sharedApplication] + 128
kaleido-tests-aarch64-darwin> 	16  kaleido                             0x000000010908fb74 _ZN8headless27HeadlessContentMainDelegate24PreCreateMainMessageLoopEv + 36
kaleido-tests-aarch64-darwin> 	17  kaleido                             0x0000000106fb9748 _ZN7content21ContentMainRunnerImpl17RunServiceManagerERNS_18MainFunctionParamsEb + 188
kaleido-tests-aarch64-darwin> 	18  kaleido                             0x0000000106fb9660 _ZN7content21ContentMainRunnerImpl3RunEb + 304
kaleido-tests-aarch64-darwin> 	19  kaleido                             0x0000000106fb7e40 _ZN7content17RunContentProcessERKNS_17ContentMainParamsEPNS_17ContentMainRunnerE + 552
kaleido-tests-aarch64-darwin> 	20  kaleido                             0x0000000106fb8520 _ZN7content11ContentMainERKNS_17ContentMainParamsE + 48
kaleido-tests-aarch64-darwin> 	21  kaleido                             0x0000000107041240 _ZN8headless12_GLOBAL__N_114RunContentMainENS_15HeadlessBrowser7OptionsEN4base12OnceCallbackIFvPS1_EEE + 260
kaleido-tests-aarch64-darwin> 	22  kaleido                             0x0000000107041104 _ZN8headless19HeadlessBrowserMainENS_15HeadlessBrowser7OptionsEN4base12OnceCallbackIFvPS0_EEE + 56
kaleido-tests-aarch64-darwin> 	23  kaleido                             0x000000010444b8e4 main + 180
kaleido-tests-aarch64-darwin> 	24  dyld                                0x0000000186987154 start + 2476
kaleido-tests-aarch64-darwin> )
kaleido-tests-aarch64-darwin> /nix/store/y1fsgdcc12c2jyb1pal6ds0icrz7x8x2-python3.12-kaleido-0.2.1/lib/python3.12/site-packages/kaleido/executable/kaleido: line 5:  3017 Trace/BPT trap: 5       ./bin/kaleido $@

@Pandapip1
Copy link
Member Author

@natsukium in the meantime, I've marked the package as broken on darwin. Does that address that concern for the time being?

@natsukium
Copy link
Member

I think it looks good. The darwin fix can be done in another PR.

@natsukium
Copy link
Member

cc @GaetanLepage, @Sigmanificient if you're interested.

@Pandapip1 Pandapip1 added the 12.approvals: 1 This PR was reviewed and approved by one person. label Sep 5, 2024
@Pandapip1
Copy link
Member Author

Pandapip1 commented Sep 5, 2024

Result of nixpkgs-review pr 339136 run on x86_64-linux 1

10 packages marked as broken and skipped:
  • picosnitch
  • picosnitch.dist
  • python311Packages.pymatgen
  • python311Packages.pymatgen.dist
  • python311Packages.sumo
  • python311Packages.sumo.dist
  • python312Packages.pymatgen
  • python312Packages.pymatgen.dist
  • python312Packages.sumo
  • python312Packages.sumo.dist
2 packages failed to build:
  • theharvester
  • theharvester.dist
120 packages built:
  • apkleaks
  • apkleaks.dist
  • checkov
  • checkov.dist
  • cve-bin-tool
  • cve-bin-tool.dist
  • das
  • das.dist
  • deeptools
  • deeptools.dist
  • hp2p
  • jadx
  • python311Packages.ax-platform
  • python311Packages.ax-platform.dist
  • python311Packages.catboost
  • python311Packages.catboost.dist
  • python311Packages.chart-studio
  • python311Packages.chart-studio.dist
  • python311Packages.compressai
  • python311Packages.compressai.dist
  • python311Packages.cufflinks
  • python311Packages.cufflinks.dist
  • python311Packages.dash
  • python311Packages.dash-bootstrap-components
  • python311Packages.dash-bootstrap-components.dist
  • python311Packages.dash.dist
  • python311Packages.experiment-utilities
  • python311Packages.experiment-utilities.dist
  • python311Packages.explorerscript
  • python311Packages.explorerscript.dist
  • python311Packages.igraph
  • python311Packages.igraph.dist
  • python311Packages.kaleido
  • python311Packages.kaleido.dist
  • python311Packages.kmapper
  • python311Packages.kmapper.dist
  • python311Packages.leidenalg
  • python311Packages.leidenalg.dist
  • python311Packages.meshtastic
  • python311Packages.meshtastic.dist
  • python311Packages.optuna
  • python311Packages.optuna.dist
  • python311Packages.orange3
  • python311Packages.orange3.dist
  • python311Packages.plotly
  • python311Packages.plotly.dist
  • python311Packages.pyannote-audio
  • python311Packages.pyannote-audio.dist
  • python311Packages.pyannote-pipeline
  • python311Packages.pyannote-pipeline.dist
  • python311Packages.reflex
  • python311Packages.reflex.dist
  • python311Packages.scikit-tda
  • python311Packages.scikit-tda.dist
  • python311Packages.skytemple-dtef
  • python311Packages.skytemple-dtef.dist
  • python311Packages.skytemple-files
  • python311Packages.skytemple-files.dist
  • python311Packages.skytemple-ssb-debugger
  • python311Packages.skytemple-ssb-debugger.dist
  • python311Packages.synergy
  • python311Packages.synergy.dist
  • python311Packages.textnets
  • python311Packages.textnets.dist
  • python312Packages.ax-platform
  • python312Packages.ax-platform.dist
  • python312Packages.catboost
  • python312Packages.catboost.dist
  • python312Packages.chart-studio
  • python312Packages.chart-studio.dist
  • python312Packages.compressai
  • python312Packages.compressai.dist
  • python312Packages.cufflinks
  • python312Packages.cufflinks.dist
  • python312Packages.dash
  • python312Packages.dash-bootstrap-components
  • python312Packages.dash-bootstrap-components.dist
  • python312Packages.dash.dist
  • python312Packages.experiment-utilities
  • python312Packages.experiment-utilities.dist
  • python312Packages.explorerscript
  • python312Packages.explorerscript.dist
  • python312Packages.igraph
  • python312Packages.igraph.dist
  • python312Packages.kaleido
  • python312Packages.kaleido.dist
  • python312Packages.kmapper
  • python312Packages.kmapper.dist
  • python312Packages.leidenalg
  • python312Packages.leidenalg.dist
  • python312Packages.meshtastic
  • python312Packages.meshtastic.dist
  • python312Packages.optuna
  • python312Packages.optuna.dist
  • python312Packages.orange3
  • python312Packages.orange3.dist
  • python312Packages.plotly
  • python312Packages.plotly.dist
  • python312Packages.pyannote-audio
  • python312Packages.pyannote-audio.dist
  • python312Packages.pyannote-pipeline
  • python312Packages.pyannote-pipeline.dist
  • python312Packages.reflex
  • python312Packages.reflex.dist
  • python312Packages.scikit-tda
  • python312Packages.scikit-tda.dist
  • python312Packages.skytemple-dtef
  • python312Packages.skytemple-dtef.dist
  • python312Packages.skytemple-files
  • python312Packages.skytemple-files.dist
  • python312Packages.skytemple-ssb-debugger
  • python312Packages.skytemple-ssb-debugger.dist
  • python312Packages.synergy
  • python312Packages.synergy.dist
  • quark-engine
  • quark-engine.dist
  • skytemple
  • skytemple.dist
  • whisper-ctranslate2
  • whisper-ctranslate2.dist

aiomultiprocess failed to build, causing theharvester to fail. It's also failing on master. See my fix here: #339823

@natsukium natsukium merged commit 41120cb into NixOS:master Sep 5, 2024
@Pandapip1 Pandapip1 deleted the init-python3packages-kaleido branch September 5, 2024 16:18
@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2024

Backport failed for release-24.05, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-24.05
git worktree add -d .worktree/backport-339136-to-release-24.05 origin/release-24.05
cd .worktree/backport-339136-to-release-24.05
git switch --create backport-339136-to-release-24.05
git cherry-pick -x 25ad2d11ec45cbc6c908b78de63ad49aba401f9f 534c90d223c32bf88b7aba3399fa69936c25fbf6 46c1ce890a749f6c17b07ce87237e7ebccf5af42 6c9867637668f00d0e7f8dac0f3312c525500c9d 7edc3b600d3e35e422bed873b9956797aeec801b

@Pandapip1
Copy link
Member Author

Manually backported: #339839

@sarahec
Copy link
Contributor

sarahec commented Sep 5, 2024

Here's a weirdness: it builds fine on Darwin when run as a stand-alone build. The error you saw (essentially "we don't know how to find the system UI configuration") may be due to the building machine being headless.

Recommendation: disable the failing test. It passes when run on a desktop/laptop.

@emilazy
Copy link
Member

emilazy commented Sep 5, 2024

It’s just the Nix sandbox forbidding access to OS stuff. Happens a lot.

@Pandapip1
Copy link
Member Author

Unfortunately I don't have access to a darwin machine to debug this. I suspect it might be libGL-related?

@sarahec
Copy link
Contributor

sarahec commented Sep 5, 2024

Unfortunately I don't have access to a darwin machine to debug this. I suspect it might be libGL-related?

I think @emilazy has identified the issue. I'm attempting to verify it on a Darwin machine so we can disable the appropriate tests.

@Pandapip1
Copy link
Member Author

I think @emilazy has identified the issue.

Yea, I was just noting that libGL is one of the few impurities left in nixpkgs.

@emilazy
Copy link
Member

emilazy commented Sep 5, 2024

I’m not sure what you mean; libGL isn’t impure. On Darwin, it’s the same as darwin.apple_sdk.frameworks.OpenGL.

@eclairevoyant
Copy link
Contributor

Loading libGL is done impurely in NixOS, because nvidia.
(Not sure if the error is related since I can't find it atm.)

@emilazy
Copy link
Member

emilazy commented Sep 5, 2024

I wouldn’t quite call that an impurity. libGL itself is libglvnd, a perfectly pure, standard, Nix‐packaged library. That library happens to delegate to a system OpenGL driver at runtime, which I would say makes it late‐bound, rather than impure per se; the properties of the Nix build aren’t being violated at any point.

Anyway, that’s not too relevant to macOS, where every system framework is built inside Nix against purely‐packaged stubs listing the available symbols, and which is then linked against the OS‐level framework code at runtime by the dynamic loader. (So in a sense, every system library is like libGL on Darwin.)

@Pandapip1
Copy link
Member Author

I wouldn’t quite call that an impurity. libGL itself is libglvnd, a perfectly pure, standard, Nix‐packaged library. That library happens to delegate to a system OpenGL driver at runtime, which I would say makes it late‐bound, rather than impure per se; the properties of the Nix build aren’t being violated at any point.

libglvnd's .sos have RPATHs that contain paths outside of /nix/store.

@sarahec
Copy link
Contributor

sarahec commented Sep 5, 2024

I think @emilazy has identified the issue. I'm attempting to verify it on a Darwin machine so we can disable the appropriate tests.

I can't get the build to break for me on Darwin, even with sandboxing enabled. Should I file a PR that conditionally disables the passthrough test on Darwin and removes the broken flag?

@Pandapip1
Copy link
Member Author

I can't get the build to break for me on Darwin, even with sandboxing enabled. Should I file a PR that conditionally disables the passthrough test on Darwin and removes the broken flag?

Go right ahead!

9999years added a commit to 9999years/nixpkgs that referenced this pull request Oct 27, 2025
This package was introduced in NixOS#339136, which attracted quite a bit of
conversation. I don't have the energy to go through the conversations
but it looks like some low-hanging fruit got overlooked. This package
depends on `sbcl` (!!) in order to get `fontconfig` files for `dejavu`
(!!!), which is ... already in the default `pkgs.fontconfig` config
files.

This is a stripped-down version of NixOS#454818 to (hopefully) expedite its
merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

0.kind: package adoption Requests or PRs for adopting packages that have no maintainers 6.topic: python Python is a high-level, general-purpose programming language. 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 11.by: package-maintainer This PR was created by a maintainer of all the package it changes. 12.approvals: 1 This PR was reviewed and approved by one person.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants