Skip to content

python3Packages.sqlite-vec: init at 0.1.6#420393

Merged
GaetanLepage merged 2 commits intoNixOS:masterfrom
sarahec:init-sqlite-vec
Jul 10, 2025
Merged

python3Packages.sqlite-vec: init at 0.1.6#420393
GaetanLepage merged 2 commits intoNixOS:masterfrom
sarahec:init-sqlite-vec

Conversation

@sarahec
Copy link
Contributor

@sarahec sarahec commented Jun 27, 2025

sqlite-vec exists as a C library, this provides the Python interfaces.

Building from PyPi wheels as the Github sources don't provide any mechanism for building. No tests in package.

Added missing files needed for a source build plus a test.

P.S. This is a requirement for bringing python3Packages.langgraph-checkpoint-sqlite up to date

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • Nixpkgs 25.11 Release Notes (or backporting 24.11 and 25.05 Nixpkgs Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
  • NixOS 25.11 Release Notes (or backporting 24.11 and 25.05 NixOS Release notes)
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other contributing documentation in corresponding paths.

Add a 👍 reaction to pull requests you find important.

@nix-owners nix-owners bot requested a review from natsukium June 27, 2025 01:28
@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 6.topic: python Python is a high-level, general-purpose programming language. labels Jun 27, 2025
@sarahec sarahec changed the title python3Packages.sqliite-vec: init at 0.1.6 python3Packages.sqlite-vec: init at 0.1.6 Jun 28, 2025
@sarahec sarahec requested a review from drupol June 28, 2025 16:46
@sarahec sarahec marked this pull request as draft July 2, 2025 18:43
@sarahec sarahec force-pushed the init-sqlite-vec branch from fc7269c to d9e7dc0 Compare July 4, 2025 17:30
@sarahec sarahec marked this pull request as ready for review July 4, 2025 17:32
@sarahec sarahec requested a review from SuperSandro2000 July 4, 2025 17:32
@sarahec
Copy link
Contributor Author

sarahec commented Jul 4, 2025

@SuperSandro2000 Thanks for the help earlier. This has been rewritten to supply the missing files. Please take a look.

P.S. I will also send a PR upstream. asg017/sqlite-vec#230

@sarahec sarahec force-pushed the init-sqlite-vec branch 6 times, most recently from 534e6e2 to b05341d Compare July 4, 2025 18:43
@sarahec
Copy link
Contributor Author

sarahec commented Jul 4, 2025

Apologies for all the little updates, it's been death-by-trailing-whitespace

@sarahec sarahec force-pushed the init-sqlite-vec branch 2 times, most recently from ec3f74b to 114b5ee Compare July 4, 2025 19:36
@GaetanLepage
Copy link
Contributor

nixpkgs-review result

Generated using nixpkgs-review-gha

Command: nixpkgs-review pr 420393

Logs: https://github.com/GaetanLepage/nixpkgs-review-gha/actions/runs/16102556705


x86_64-linux (sandbox = true)

✅ 8 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist
  • python313Packages.txtai
  • python313Packages.txtai.dist

aarch64-linux (sandbox = true)

❌ 2 packages failed to build:
  • python313Packages.txtai
  • python313Packages.txtai.dist
✅ 6 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

x86_64-darwin (sandbox = true)

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

aarch64-darwin (sandbox = true)

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

Copy link
Contributor

@GaetanLepage GaetanLepage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these additional files be upstreamed? Maintaining them in the nixpkgs tree seems quite annoying.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
--replace "@libpath@" "${lib.getLib sqlite-vec-c}/lib/"
--replace-fail "@libpath@" "${lib.getLib sqlite-vec-c}/lib/"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done and pushed

@GaetanLepage
Copy link
Contributor

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 420393
Commit: 114b5ee2e49c69a5816dc64593ee944519458c3c


x86_64-linux

✅ 8 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist
  • python313Packages.txtai
  • python313Packages.txtai.dist

aarch64-linux

✅ 8 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python312Packages.txtai
  • python312Packages.txtai.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist
  • python313Packages.txtai
  • python313Packages.txtai.dist

@sarahec
Copy link
Contributor Author

sarahec commented Jul 6, 2025

Can these additional files be upstreamed? Maintaining them in the nixpkgs tree seems quite annoying.

I plan to replace these with the upstreamed files once the PR stabilizes. Until then, they're not difficult for me to maintain.

@sarahec sarahec force-pushed the init-sqlite-vec branch from 114b5ee to 52e69c6 Compare July 6, 2025 20:34
@nixpkgs-ci nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Jul 6, 2025
@symphorien
Copy link
Member

symphorien commented Jul 7, 2025

could you fetch the additionnal files from your PR instead of vendoring them?

@sarahec sarahec force-pushed the init-sqlite-vec branch from 52e69c6 to 9db1fd3 Compare July 7, 2025 19:56
@sarahec
Copy link
Contributor Author

sarahec commented Jul 7, 2025

could you fetch the additionnal files from your PR instead of vendrong them?

Done. Hopefully using commits can survive future changes to the PR. (The upstream package is published using nonstandard tools and I had to reverse-engineer __init__.py with a couple of changes. I don't know if that will need rework to be accepted; I'm hoping to persuade the maintainer to use standard python packaging tools from here out.)

@sarahec
Copy link
Contributor Author

sarahec commented Jul 7, 2025

@GaetanLepage note the change to using the upstream PR instead of vendored files.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Is it possible to use sourceRoot ?
  2. Move the comment out of the shell script

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move all the comments :)

Copy link
Contributor Author

@sarahec sarahec Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The patches apply to /bindings/python, but when applied with sourceRoot = "${src.name}/bindings/python", you get bindings/python/python:

  > /private/tmp/nix-build-python3.12-sqlite-vec-0.1.6.drv-0/source/bindings/python
       > total 4.0K
       > drwxr-xr-x 4 _nixbld1 nixbld  128 Jul  7 20:35 .
       > dr-xr-xr-x 5 _nixbld1 nixbld  160 Jan  1  1970 ..
       > -rw-r--r-- 1 _nixbld1 nixbld 1.4K Jan  2  1980 extra_init.py
       > drwxr-xr-x 5 _nixbld1 nixbld  160 Jul  7 20:35 python

The only working solution seems to be to set sourceRoot = "${src.name}/bindings", apply the patches with stripLen=1 (which alter bindings/python appropriately), then cd into the now-patched python directory for the build.

I've tried stripLen values from 1-3 with the full sourceRoot and it always fails as shown in ls above.

This is a known issue, see https://discourse.nixos.org/t/how-to-apply-patches-with-sourceroot/59727

@samuela did you ever find a satisfactory solution?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@drupol moved the comment out and reworded. Went back and tested the various combinations of sourceRoot and stripLen again to no avail.

Copy link
Contributor Author

@sarahec sarahec Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move all the comments :)

Our comments crossed in the mail. ;) Please take a look now.

@symphorien
Copy link
Member

this new package shadows the sqlite-vec from top-level pkgs. To pass this package in argument you bind it to sqlite-vec-c. It looks like python3.pkgs.txtai has the same problem and you should apply the same process to it.

@sarahec
Copy link
Contributor Author

sarahec commented Jul 9, 2025

this new package shadows the sqlite-vec from top-level pkgs. To pass this package in argument you bind it to sqlite-vec-c. It looks like python3.pkgs.txtai has the same problem and you should apply the same process to it.

Done.

@sarahec
Copy link
Contributor Author

sarahec commented Jul 9, 2025

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 420393
Commit: 841e6506e001add53ed4c5f3f00e9ca8167b6f25


aarch64-darwin

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

@sarahec
Copy link
Contributor Author

sarahec commented Jul 9, 2025

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 420393
Commit: 841e6506e001add53ed4c5f3f00e9ca8167b6f25


x86_64-linux

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

@sarahec
Copy link
Contributor Author

sarahec commented Jul 9, 2025

@GaetanLepage this, hopefully, fixes the build error on aarch64-linux

@GaetanLepage
Copy link
Contributor

nixpkgs-review result

Generated using nixpkgs-review-gha

Command: nixpkgs-review pr 420393

Logs: https://github.com/GaetanLepage/nixpkgs-review-gha/actions/runs/16188939030


x86_64-linux

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

aarch64-linux

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

x86_64-darwin (sandbox = true)

❌ 4 packages failed to build:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

Error logs: `x86_64-darwin`
python312Packages.sqlite-vec
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/nix/store/ymyww3j90qaaxp1c61xw9jx1vwvkpgj3-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/nix/store/ymyww3j90qaaxp1c61xw9jx1vwvkpgj3-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 337 in _main
  File "/nix/store/ymyww3j90qaaxp1c61xw9jx1vwvkpgj3-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/nix/store/ymyww3j90qaaxp1c61xw9jx1vwvkpgj3-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/snwhgxnn0ys84a8bx2gi3f6xrdcx38fs-python3.12-pluggy-1.6.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/nix/store/ymyww3j90qaaxp1c61xw9jx1vwvkpgj3-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/nix/store/ymyww3j90qaaxp1c61xw9jx1vwvkpgj3-python3.12-pytest-8.3.5/lib/python3.12/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/nix/store/ymyww3j90qaaxp1c61xw9jx1vwvkpgj3-python3.12-pytest-8.3.5/lib/python3.12/site-packages/pytest/__main__.py", line 9 in 
  File "", line 88 in _run_code
  File "", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
/nix/store/597aci5k85simnpr10aijb5lm14gx8hs-pytest-check-hook/nix-support/setup-hook: line 23: 5317 Illegal instruction: 4 /nix/store/q2y1zwszjmsap7r6nbk4dxbcsm47abfx-python3-3.12.11/bin/python3.12 "${flagsArray[@]}"

python313Packages.sqlite-vec
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/nix/store/620qb9c25m6dxfksypkpvbj43mj9wkm8-python3.13-pytest-8.3.5/lib/python3.13/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/nix/store/620qb9c25m6dxfksypkpvbj43mj9wkm8-python3.13-pytest-8.3.5/lib/python3.13/site-packages/_pytest/main.py", line 337 in _main
  File "/nix/store/620qb9c25m6dxfksypkpvbj43mj9wkm8-python3.13-pytest-8.3.5/lib/python3.13/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/nix/store/620qb9c25m6dxfksypkpvbj43mj9wkm8-python3.13-pytest-8.3.5/lib/python3.13/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/nix/store/dmbvjyd3z5yjba6xxljs2vlqa7xh8gkk-python3.13-pluggy-1.6.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/nix/store/620qb9c25m6dxfksypkpvbj43mj9wkm8-python3.13-pytest-8.3.5/lib/python3.13/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/nix/store/620qb9c25m6dxfksypkpvbj43mj9wkm8-python3.13-pytest-8.3.5/lib/python3.13/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/nix/store/620qb9c25m6dxfksypkpvbj43mj9wkm8-python3.13-pytest-8.3.5/lib/python3.13/site-packages/pytest/__main__.py", line 9 in 
  File "", line 88 in _run_code
  File "", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
/nix/store/qc9f3akx56gw73hkb6gsx296966iibnm-pytest-check-hook/nix-support/setup-hook: line 23: 5319 Illegal instruction: 4 /nix/store/il3s6rjh49l9ccg1nlqm93w67lcnfmqn-python3-3.13.4/bin/python3.13 "${flagsArray[@]}"


aarch64-darwin (sandbox = true)

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

@GaetanLepage
Copy link
Contributor

x86_64-darwin (sandbox = true)

❌ 4 packages failed to build:

Maybe, let's add x86_64-darwin to meta.badPlatforms.

@sarahec
Copy link
Contributor Author

sarahec commented Jul 10, 2025

x86_64-darwin (sandbox = true)

❌ 4 packages failed to build:

Maybe, let's add x86_64-darwin to meta.badPlatforms

Done and pushed.

@nixpkgs-ci nixpkgs-ci bot added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Jul 10, 2025
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
badPlatforms = [ "x86_64-darwin" ];
badPlatforms = [
# Segfaults during tests
"x86_64-darwin"
];

Sorry to be picky there, but I think that it is quite important to state why the package is disabled on this given platform.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're absolutely right.Fixed and pushed.

@GaetanLepage
Copy link
Contributor

nixpkgs-review result

Generated using nixpkgs-review-gha

Command: nixpkgs-review pr 420393

Logs: https://github.com/GaetanLepage/nixpkgs-review-gha/actions/runs/16206365004


x86_64-linux

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

aarch64-linux

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

aarch64-darwin (sandbox = true)

✅ 4 packages built:
  • python312Packages.sqlite-vec
  • python312Packages.sqlite-vec.dist
  • python313Packages.sqlite-vec
  • python313Packages.sqlite-vec.dist

Copy link
Contributor

@GaetanLepage GaetanLepage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@GaetanLepage GaetanLepage merged commit 34d64d9 into NixOS:master Jul 10, 2025
23 of 27 checks passed
@sarahec sarahec deleted the init-sqlite-vec branch July 11, 2025 16:19
@sarahec
Copy link
Contributor Author

sarahec commented Jul 13, 2025

Reverted in #424873 due to the underlying cause being an ARM-related issue while Hydra builds on Rosetta.

@symphorien
Copy link
Member

is it the correct link?

@sarahec
Copy link
Contributor Author

sarahec commented Jul 14, 2025

is it the correct link?

That PR is still a work in progress if that's what you're asking. I only reverted the part that marks x86_64-darwin bad as it was a side-effect of another bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 12.approvals: 2 This PR was reviewed and approved by two persons.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants