Skip to content

python3Packages.chromadb: fixes#412528

Merged
GaetanLepage merged 6 commits intoNixOS:masterfrom
sarahec:bump-chromadb
Jun 8, 2025
Merged

python3Packages.chromadb: fixes#412528
GaetanLepage merged 6 commits intoNixOS:masterfrom
sarahec:bump-chromadb

Conversation

@sarahec
Copy link
Contributor

@sarahec sarahec commented May 31, 2025

chromadb: updates and backwards compatibility

  1. Cleanup: Organize imports, fix with lib in meta block
  2. Add self as maintainer
  3. Update chromadb to 0.6.3 and package as python3Packages.chromadb_0. Many packages in the AI ecosystem aren't ready for 1.0 and can use chromadb_0.
  4. Update to version 1.0.12. release notes: https://github.com/chroma-core/chroma/releases/tag/1.0.12

Dependents

  1. Update python3Packages.langchain-chroma to 0.2.4 which needs chromadb >= 1.0.9
  2. Update vectorcode to use chroma_0

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • Nixpkgs 25.11 Release Notes (or backporting 24.11 and 25.05 Nixpkgs Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
  • NixOS 25.11 Release Notes (or backporting 24.11 and 25.05 NixOS Release notes)
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@sarahec sarahec marked this pull request as draft May 31, 2025 00:37
@github-actions github-actions bot added 6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels May 31, 2025
@nix-owners nix-owners bot requested a review from fabaff May 31, 2025 00:43
@sarahec sarahec marked this pull request as ready for review May 31, 2025 15:49
@sarahec sarahec marked this pull request as draft June 1, 2025 21:59
@nix-owners nix-owners bot requested a review from natsukium June 1, 2025 22:00
@github-actions github-actions bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. and removed 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Jun 1, 2025
@sarahec sarahec force-pushed the bump-chromadb branch 3 times, most recently from 14d2ff3 to 4d37579 Compare June 2, 2025 01:01
@sarahec sarahec changed the title python3Packages.chromadb: 0.5.20 -> 1.0.10 python3Packages.chromadb: fixes Jun 2, 2025
@github-actions github-actions bot added the 8.has: documentation This PR adds or changes documentation label Jun 2, 2025
@sarahec sarahec marked this pull request as ready for review June 2, 2025 01:53
@sarahec sarahec marked this pull request as draft June 2, 2025 01:55
@sarahec sarahec marked this pull request as ready for review June 2, 2025 02:54
@sarahec
Copy link
Contributor Author

sarahec commented Jun 2, 2025

@fabaff Please take a look.

@sarahec sarahec requested a review from GaetanLepage June 2, 2025 17:31
@nix-owners nix-owners bot requested a review from samuela June 2, 2025 17:57
@GaetanLepage
Copy link
Contributor

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 412528
Commit: 1b5eeb616930f78ae1473f0cc966d875d0f33f6b


x86_64-linux

⏩ 1 package blacklisted:
  • tests.nixos-functions.nixos-test
✅ 38 packages built:
  • nixpkgs-manual
  • open-webui
  • open-webui.dist
  • private-gpt
  • private-gpt.dist
  • python312Packages.chromadb
  • python312Packages.chromadb.dist
  • python312Packages.chromadb_0
  • python312Packages.chromadb_0.dist
  • python312Packages.langchain-chroma
  • python312Packages.langchain-chroma.dist
  • python312Packages.llama-index
  • python312Packages.llama-index-cli
  • python312Packages.llama-index-cli.dist
  • python312Packages.llama-index-vector-stores-chroma
  • python312Packages.llama-index-vector-stores-chroma.dist
  • python312Packages.llama-index.dist
  • python312Packages.private-gpt
  • python312Packages.private-gpt.dist
  • python313Packages.chromadb
  • python313Packages.chromadb.dist
  • python313Packages.chromadb_0
  • python313Packages.chromadb_0.dist
  • python313Packages.langchain-chroma
  • python313Packages.langchain-chroma.dist
  • python313Packages.llama-index
  • python313Packages.llama-index-cli
  • python313Packages.llama-index-cli.dist
  • python313Packages.llama-index-vector-stores-chroma
  • python313Packages.llama-index-vector-stores-chroma.dist
  • python313Packages.llama-index.dist
  • python313Packages.private-gpt
  • python313Packages.private-gpt.dist
  • seagoat
  • seagoat.dist
  • vectorcode
  • vectorcode.dist
  • vimPlugins.vectorcode-nvim

aarch64-linux

⏩ 1 package blacklisted:
  • tests.nixos-functions.nixos-test
❌ 3 packages failed to build:
  • vectorcode
  • vectorcode.dist
  • vimPlugins.vectorcode-nvim
✅ 33 packages built:
  • nixpkgs-manual
  • private-gpt
  • private-gpt.dist
  • python312Packages.chromadb
  • python312Packages.chromadb.dist
  • python312Packages.chromadb_0
  • python312Packages.chromadb_0.dist
  • python312Packages.langchain-chroma
  • python312Packages.langchain-chroma.dist
  • python312Packages.llama-index
  • python312Packages.llama-index-cli
  • python312Packages.llama-index-cli.dist
  • python312Packages.llama-index-vector-stores-chroma
  • python312Packages.llama-index-vector-stores-chroma.dist
  • python312Packages.llama-index.dist
  • python312Packages.private-gpt
  • python312Packages.private-gpt.dist
  • python313Packages.chromadb
  • python313Packages.chromadb.dist
  • python313Packages.chromadb_0
  • python313Packages.chromadb_0.dist
  • python313Packages.langchain-chroma
  • python313Packages.langchain-chroma.dist
  • python313Packages.llama-index
  • python313Packages.llama-index-cli
  • python313Packages.llama-index-cli.dist
  • python313Packages.llama-index-vector-stores-chroma
  • python313Packages.llama-index-vector-stores-chroma.dist
  • python313Packages.llama-index.dist
  • python313Packages.private-gpt
  • python313Packages.private-gpt.dist
  • seagoat
  • seagoat.dist

x86_64-darwin

✅ 30 packages built:
  • nixpkgs-manual
  • python312Packages.chromadb
  • python312Packages.chromadb.dist
  • python312Packages.chromadb_0
  • python312Packages.chromadb_0.dist
  • python312Packages.langchain-chroma
  • python312Packages.langchain-chroma.dist
  • python312Packages.llama-index
  • python312Packages.llama-index-cli
  • python312Packages.llama-index-cli.dist
  • python312Packages.llama-index-vector-stores-chroma
  • python312Packages.llama-index-vector-stores-chroma.dist
  • python312Packages.llama-index.dist
  • python313Packages.chromadb
  • python313Packages.chromadb.dist
  • python313Packages.chromadb_0
  • python313Packages.chromadb_0.dist
  • python313Packages.langchain-chroma
  • python313Packages.langchain-chroma.dist
  • python313Packages.llama-index
  • python313Packages.llama-index-cli
  • python313Packages.llama-index-cli.dist
  • python313Packages.llama-index-vector-stores-chroma
  • python313Packages.llama-index-vector-stores-chroma.dist
  • python313Packages.llama-index.dist
  • seagoat
  • seagoat.dist
  • vectorcode
  • vectorcode.dist
  • vimPlugins.vectorcode-nvim

aarch64-darwin

✅ 30 packages built:
  • nixpkgs-manual
  • python312Packages.chromadb
  • python312Packages.chromadb.dist
  • python312Packages.chromadb_0
  • python312Packages.chromadb_0.dist
  • python312Packages.langchain-chroma
  • python312Packages.langchain-chroma.dist
  • python312Packages.llama-index
  • python312Packages.llama-index-cli
  • python312Packages.llama-index-cli.dist
  • python312Packages.llama-index-vector-stores-chroma
  • python312Packages.llama-index-vector-stores-chroma.dist
  • python312Packages.llama-index.dist
  • python313Packages.chromadb
  • python313Packages.chromadb.dist
  • python313Packages.chromadb_0
  • python313Packages.chromadb_0.dist
  • python313Packages.langchain-chroma
  • python313Packages.langchain-chroma.dist
  • python313Packages.llama-index
  • python313Packages.llama-index-cli
  • python313Packages.llama-index-cli.dist
  • python313Packages.llama-index-vector-stores-chroma
  • python313Packages.llama-index-vector-stores-chroma.dist
  • python313Packages.llama-index.dist
  • seagoat
  • seagoat.dist
  • vectorcode
  • vectorcode.dist
  • vimPlugins.vectorcode-nvim

Error logs: `aarch64-linux`
vectorcode
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 975 in genitems
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 809 in perform_collect
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 347 in pytest_collection
  File "/nix/store/vbkqq1wnfxyvj5508bfy0hyl2wsdzdjv-python3.12-pluggy-1.5.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/nix/store/vbkqq1wnfxyvj5508bfy0hyl2wsdzdjv-python3.12-pluggy-1.5.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/vbkqq1wnfxyvj5508bfy0hyl2wsdzdjv-python3.12-pluggy-1.5.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 336 in _main
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/nix/store/vbkqq1wnfxyvj5508bfy0hyl2wsdzdjv-python3.12-pluggy-1.5.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/nix/store/vbkqq1wnfxyvj5508bfy0hyl2wsdzdjv-python3.12-pluggy-1.5.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/vbkqq1wnfxyvj5508bfy0hyl2wsdzdjv-python3.12-pluggy-1.5.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/nix/store/a3q88rdni6kr08d2kqb9sy3zwrhyjy2a-python3.12-pytest-8.3.5/lib/python3.12/site-packages/pytest/__main__.py", line 9 in 
  File "", line 88 in _run_code
  File "", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
/nix/store/bb18p3kqczldy903jlyx8l67f8frmnhw-pytest-check-hook/nix-support/setup-hook: line 23: 460 Aborted (core dumped) /nix/store/rpf3admp58i0lwmsbv0y3dg7l7qqj4a9-python3-3.12.10/bin/python3.12 "${flagsArray[@]}"

@sarahec sarahec added 12.approvals: 1 This PR was reviewed and approved by one person. and removed 8.has: documentation This PR adds or changes documentation labels Jun 7, 2025
@sarahec sarahec requested a review from kirillrdy June 7, 2025 23:23
@sarahec sarahec added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Jun 7, 2025
@GaetanLepage GaetanLepage merged commit 80b9b2b into NixOS:master Jun 8, 2025
18 of 20 checks passed
@sarahec sarahec deleted the bump-chromadb branch June 9, 2025 14:48
Copy link
Member

@dotlambda dotlambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GaetanLepage This is unacceptable.


chromadb = callPackage ../development/python-modules/chromadb { };

chromadb_0 = callPackage ../development/python-modules/chromadb/0.nix { };
Copy link
Member

@dotlambda dotlambda Jun 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't do versioned attributes in python3Packages! Only for very exceptional cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry. I knew that this should be avoided, but could be done when necessary. It seemed legit in this case.
Is there a reference describing what would be qualified as an "exceptional case"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. But one necessary condition should be that it's used more than once across nixpkgs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But one necessary condition should be that it's used more than once across nixpkgs.

That's fair indeed.

Copy link
Contributor Author

@sarahec sarahec Jun 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this plan sound:

  1. I'll move the chromadb 0.6.3 derivation into its one client (vectorcode) under a let.
  2. I'll revert the chromadb_0 commit
  3. I'll edit chromadb 1.0.x to put the maturin build hook in nativeBuildInputs

Is that acceptable?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be inside packageOverrides inside let.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sarahec have you opened a PR already?

with python3Packages;
[
chromadb
chromadb_0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use packageOverrides.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I'll fix it.

build-system = [
setuptools
setuptools-scm
rustPlatform.maturinBuildHook
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This belongs in nativeBuildInputs as it's not a Python package.

protobuf,
rustc,
rustPlatform,
pkgs, # zstd hidden by python3Packages.zstd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use zstd-c and set zstd-c = pkgs.zstd in python-packages.nix.

@dotlambda dotlambda mentioned this pull request Jun 18, 2025
13 tasks
GaetanLepage pushed a commit that referenced this pull request Jun 24, 2025
On aarch64-linux onnxruntime cannot be imported, as it attempts to read
/sys/devices/system/cpu, which does not exist in the sandbox.

As onnxruntime cannot be imported, chromadb cannot be tested.

However, this issue only applies in the build sandbox, not at runtime.
Therefore the package doesn't need to be marked broken.

Updated the condition to check `buildPlatform` instead of `hostPlatform`,
as the issue relates to the build sandbox and is not a runtime issue.

Use `isAarch` and `isLinux` attributes instead of doing string comparison.

---

See #412528 (comment)
And upstream onnxruntime issue: microsoft/onnxruntime#10038
chrisportela pushed a commit to chrisportela/nixpkgs that referenced this pull request Jul 3, 2025
On aarch64-linux onnxruntime cannot be imported, as it attempts to read
/sys/devices/system/cpu, which does not exist in the sandbox.

As onnxruntime cannot be imported, chromadb cannot be tested.

However, this issue only applies in the build sandbox, not at runtime.
Therefore the package doesn't need to be marked broken.

Updated the condition to check `buildPlatform` instead of `hostPlatform`,
as the issue relates to the build sandbox and is not a runtime issue.

Use `isAarch` and `isLinux` attributes instead of doing string comparison.

---

See NixOS#412528 (comment)
And upstream onnxruntime issue: microsoft/onnxruntime#10038
@haylinmoore
Copy link
Contributor

I believe this PR broke the nixos module test for chromadb, it seems to have gone from passing to not passing after it #442520 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 12.approvals: 2 This PR was reviewed and approved by two persons.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants