Skip to content

python311Packages.tensorflow-bin: 2.15.0 -> 2.16.1 #311945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
samuela merged 2 commits into from
May 19, 2024
Merged

python311Packages.tensorflow-bin: 2.15.0 -> 2.16.1 #311945

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e5ef2dd
python311Packages.tensorflow-bin: 2.15.0 -> 2.16.1
GaetanLepage May 15, 2024
9a6fd43
python311Packages.tensorflow-bin: Support more architectures
GaetanLepage May 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 17 additions & 9 deletions pkgs/development/python-modules/tensorflow/bin.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
, astunparse
, flatbuffers
, h5py
, llvmPackages
, typing-extensions
}:

Expand All @@ -51,12 +52,16 @@ in buildPythonPackage {

src = let
pyVerNoDot = lib.strings.stringAsChars (x: lib.optionalString (x != ".") x) python.pythonVersion;
platform = if stdenv.isDarwin then "mac" else "linux";
unit = if cudaSupport then "gpu" else "cpu";
key = "${platform}_py_${pyVerNoDot}_${unit}";
in fetchurl (packages.${key} or {});
platform = stdenv.system;
cuda = lib.optionalString cudaSupport "_gpu";
key = "${platform}_${pyVerNoDot}${cuda}";
in fetchurl (packages.${key} or (throw "tensoflow-bin: unsupported system: ${stdenv.system}"));

propagatedBuildInputs = [
buildInputs = [
llvmPackages.openmp
];

dependencies = [
astunparse
flatbuffers
typing-extensions
Expand All @@ -81,7 +86,7 @@ in buildPythonPackage {
h5py
] ++ lib.optional (!isPy3k) mock;

nativeBuildInputs = [ wheel ] ++ lib.optionals cudaSupport [ addOpenGLRunpath ];
build-system = [ wheel ] ++ lib.optionals cudaSupport [ addOpenGLRunpath ];

preConfigure = ''
unset SOURCE_DATE_EPOCH
Expand All @@ -91,7 +96,6 @@ in buildPythonPackage {

pushd dist

orig_name="$(echo ./*.whl)"
wheel unpack --dest unpacked ./*.whl
rm ./*.whl
(
Expand All @@ -113,7 +117,6 @@ in buildPythonPackage {
-e "s/Requires-Dist: numpy (.*)/Requires-Dist: numpy/"
)
wheel pack ./unpacked/tensorflow*
mv *.whl $orig_name # avoid changes to the _os_arch.whl suffix
Copy link
Contributor Author

@GaetanLepage GaetanLepage May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was making the darwin build fail:

Unpacking to: unpacked/tensorflow-2.16.1...OK
Repacking wheel as ./tensorflow-2.16.1-cp311-cp311-macosx_12_0_arm64.whl...OK
mv: 'tensorflow-2.16.1-cp311-cp311-macosx_12_0_arm64.whl' and './tensorflow-2.16.1-cp311-cp311-macosx_12_0_arm64.whl' are the same file

Apparently, removing it does not make the linux builds fail, so I completely deleted it.

Copy link
Contributor

@SomeoneSerge SomeoneSerge May 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently this was introduced specifically for darwin: 6a2dad3

Copy link
Contributor

@SomeoneSerge SomeoneSerge May 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the intention for tensorflow-bin to support multiple versions of tensorflow in a single nixpkgs revision (e.g. through overrides)? I see the attrset entries contain a "version" field. This change might be breaking non-default instantiations of tensorflow-bin, maybe we should test them and if they fail - make the mv conditional rather than remove it. I say "maybe" because I don't know if this feature actually has any users

Copy link
Contributor Author

@GaetanLepage GaetanLepage May 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, as it was introduced as a "fix" for darwin (which it probably was at the time), I assume than we can remove it now... as it is the cause of this new failure.
What is your take on this @veprbl ?


popd
'';
Expand Down Expand Up @@ -168,6 +171,7 @@ in buildPythonPackage {
"$out/${python.sitePackages}/tensorflow/python/saved_model"
"$out/${python.sitePackages}/tensorflow/python/util"
"$out/${python.sitePackages}/tensorflow/tsl/python/lib/core"
"$out/${python.sitePackages}/tensorflow.libs/"
Copy link
Contributor

@SomeoneSerge SomeoneSerge May 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A note for later: I think it's best that we get rid of the custom fixup logic. There are many other places in nixpkgs where implement a similar logic, which was the motivation for the now-abandoned #297590

"${rpath}"
)

Expand Down Expand Up @@ -205,6 +209,10 @@ in buildPythonPackage {
sourceProvenance = with sourceTypes; [ binaryNativeCode ];
license = licenses.asl20;
maintainers = with maintainers; [ jyp abbradar ];
platforms = [ "x86_64-linux" "x86_64-darwin" ];
platforms = platforms.all;
# Cannot import tensortfow on python 3.12 as it still dependends on distutils:
# ModuleNotFoundError: No module named 'distutils'
# https://github.com/tensorflow/tensorflow/issues/58073
broken = pythonAtLeast "3.12";
};
}
116 changes: 80 additions & 36 deletions pkgs/development/python-modules/tensorflow/binary-hashes.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,83 @@
{
version = "2.15.0";
linux_py_39_cpu = {
url = "https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.15.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "01m5wn9wpywd2cscklr2h8wdsf3wzg11dhgl0nfj7a997v2bvlyg";
};
linux_py_310_cpu = {
url = "https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0qjahvq9gd89cx2dwkb6gk3s947czmasp9cx1pqyw1x5f113s2zn";
};
linux_py_311_cpu = {
url = "https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow_cpu-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0w3jz7rpbf4r8hm91rjh666qyqflqy6x619fas1iqcgnfx8pjv7l";
};
linux_py_39_gpu = {
url = "https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-2.15.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0ckij6jffsi6yvjfiysnwrfblhqm5g1f8dkw4i7n3dwndnanba1z";
};
linux_py_310_gpu = {
url = "https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-2.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "1y0cxsps4rkidc2c3apaf7vd9vsnjg5lpcq30bnrnzcvvx1inq7d";
};
linux_py_311_gpu = {
url = "https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-2.15.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "1z7bd4yrdxa7rq4k1fybbwnyhxf4rm9nwx15bpk2m53c5hmyrs6y";
};
mac_py_39_cpu = {
url = "https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.15.0-cp39-cp39-macosx_10_15_x86_64.whl";
sha256 = "1kqb5h4vs4jlhlqjmqvcg6jivccwqj1rgklrgrhvw09wzzhxvz1c";
};
mac_py_310_cpu = {
url = "https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.15.0-cp310-cp310-macosx_10_15_x86_64.whl";
sha256 = "0wmawdfhva9ixal7468c0rrx8qd7nzgkxy6i9k2s7cqn8c7qw94v";
};
mac_py_311_cpu = {
url = "https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.15.0-cp311-cp311-macosx_10_15_x86_64.whl";
sha256 = "0cigk43fabps7vskz4pfgmca3wa0545b15qridyqdbyp5ri1c1qy";
version = "2.16.1";
x86_64-linux_39 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow_cpu-2.16.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0j4311fav3b4wdhw9wb3zvc3grvgxddl28i4kwnbsvvjb191pcs5";
};
x86_64-linux_310 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow_cpu-2.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0b8q3r2ql7rknlshiv3bh3hj9zll0dn6w0wdxnqgg18cljys5nhh";
};
x86_64-linux_311 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow_cpu-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "12vv2jxybdfa8nq7r2y53vl7lvmaacb1xmy2368hkwqm2ld4mplg";
};
x86_64-linux_312 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow_cpu-2.16.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0vvv3cr10r6kgkalk3hydqz18y4lp990pm4vwq8mbwdi101m59mf";
};
x86_64-linux_39_gpu = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "17v85433ns360rcccx7pnkhjzd003cr8p6rj51wqb3gl7gkldf83";
};
x86_64-linux_310_gpu = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0w3jd7siamvhgd7si59zpv3rdpq3s2ii3hhcdaimji66axv095hy";
};
x86_64-linux_311_gpu = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "014liy5wnbi2d2lnr8lqhd7221a4a1v6gzih7nv5qfnf1h86234k";
};
x86_64-linux_312_gpu = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "0qd09bxfnad969y2h3mlzib51y6xcsqipq1w278k1fl8ayxky4ja";
};
aarch64-linux_39 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl";
sha256 = "1n3qf0snjwrc0lcahls4bi2qhpd4yrd531b80vlq89j0gjkjl5cj";
};
aarch64-linux_310 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl";
sha256 = "1aygypgm5fbpskf6xvzni2yd9l49hf3ls855z2aq0as73m3m81df";
};
aarch64-linux_311 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl";
sha256 = "1c0bqiq2hbzcqhaym3s3s0dsp06m07mqgvm35z14nz395pxgjzb1";
};
aarch64-linux_312 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl";
sha256 = "1ai2hrmn1il0hididdvhd005pjcn6hjivpgcw7dbhmdv6vacs4n6";
};
x86_64-darwin_39 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp39-cp39-macosx_10_15_x86_64.whl";
sha256 = "0dkcd1v0ivjp6v5pngg0x461wci94nxvwprc6jwgi7rqgyx3zkz9";
};
x86_64-darwin_310 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp310-cp310-macosx_10_15_x86_64.whl";
sha256 = "0rnlxrb4q2cgqaifalhpji15g1f3m9flqfvfj2p2sv3lvmbg2ydb";
};
x86_64-darwin_311 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp311-cp311-macosx_10_15_x86_64.whl";
sha256 = "1xc9lis61adk2y22a82i3rvc0j7qznf4fqwg2pd1665zlhybi9gq";
};
x86_64-darwin_312 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp312-cp312-macosx_10_15_x86_64.whl";
sha256 = "10g8y02ncp5l7f133vank6h01palc445h6s9jndmmy7vm33c7jh9";
};
aarch64-darwin_39 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp39-cp39-macosx_12_0_arm64.whl";
sha256 = "0541ldysc1a06xcb632340gn1c33w245lzqwq2f2p6hm2ifqp1a2";
};
aarch64-darwin_310 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp310-cp310-macosx_12_0_arm64.whl";
sha256 = "1crrd4izayqwji5gxjw1g9qppkvfsc8p1mr7z6hiipxidys6ldwf";
};
aarch64-darwin_311 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp311-cp311-macosx_12_0_arm64.whl";
sha256 = "0kg4g4p563nhw0l2jzydhw4ycyj3dkgqzzm0m3z8k7vzsb8na86c";
};
aarch64-darwin_312 = {
url = "https://storage.googleapis.com/tensorflow/versions/2.16.1/tensorflow-2.16.1-cp312-cp312-macosx_12_0_arm64.whl";
sha256 = "1hnzfpfncw8fa84y0jivpfyb5cbbd4cvpln72wkcxpvhj23nvw5v";
};
}
33 changes: 22 additions & 11 deletions pkgs/development/python-modules/tensorflow/prefetcher.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,33 @@
#!/usr/bin/env bash

version="2.15.0"
version="2.16.1"

bucket="https://storage.googleapis.com/tensorflow"
bucket="https://storage.googleapis.com/tensorflow/versions/${version}"

# List of binary wheels for Tensorflow. The most recent versions can be found
# on the following page:
# https://www.tensorflow.org/install/pip?lang=python3#package-location
url_and_key_list=(
"linux_py_39_cpu $bucket/linux/cpu/tensorflow_cpu-${version}-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"linux_py_310_cpu $bucket/linux/cpu/tensorflow_cpu-${version}-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"linux_py_311_cpu $bucket/linux/cpu/tensorflow_cpu-${version}-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"linux_py_39_gpu $bucket/linux/gpu/tensorflow-${version}-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"linux_py_310_gpu $bucket/linux/gpu/tensorflow-${version}-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"linux_py_311_gpu $bucket/linux/gpu/tensorflow-${version}-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"mac_py_39_cpu $bucket/mac/cpu/tensorflow-${version}-cp39-cp39-macosx_10_15_x86_64.whl"
"mac_py_310_cpu $bucket/mac/cpu/tensorflow-${version}-cp310-cp310-macosx_10_15_x86_64.whl"
"mac_py_311_cpu $bucket/mac/cpu/tensorflow-${version}-cp311-cp311-macosx_10_15_x86_64.whl"
"x86_64-linux_39 $bucket/tensorflow_cpu-${version}-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"x86_64-linux_310 $bucket/tensorflow_cpu-${version}-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"x86_64-linux_311 $bucket/tensorflow_cpu-${version}-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"x86_64-linux_312 $bucket/tensorflow_cpu-${version}-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"x86_64-linux_39_gpu $bucket/tensorflow-${version}-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"x86_64-linux_310_gpu $bucket/tensorflow-${version}-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"x86_64-linux_311_gpu $bucket/tensorflow-${version}-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"x86_64-linux_312_gpu $bucket/tensorflow-${version}-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
"aarch64-linux_39 $bucket/tensorflow-${version}-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
"aarch64-linux_310 $bucket/tensorflow-${version}-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
"aarch64-linux_311 $bucket/tensorflow-${version}-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
"aarch64-linux_312 $bucket/tensorflow-${version}-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl"
"x86_64-darwin_39 $bucket/tensorflow-${version}-cp39-cp39-macosx_10_15_x86_64.whl"
"x86_64-darwin_310 $bucket/tensorflow-${version}-cp310-cp310-macosx_10_15_x86_64.whl"
"x86_64-darwin_311 $bucket/tensorflow-${version}-cp311-cp311-macosx_10_15_x86_64.whl"
"x86_64-darwin_312 $bucket/tensorflow-${version}-cp312-cp312-macosx_10_15_x86_64.whl"
"aarch64-darwin_39 $bucket/tensorflow-${version}-cp39-cp39-macosx_12_0_arm64.whl"
"aarch64-darwin_310 $bucket/tensorflow-${version}-cp310-cp310-macosx_12_0_arm64.whl"
"aarch64-darwin_311 $bucket/tensorflow-${version}-cp311-cp311-macosx_12_0_arm64.whl"
"aarch64-darwin_312 $bucket/tensorflow-${version}-cp312-cp312-macosx_12_0_arm64.whl"
)

hashfile=binary-hashes.nix
Expand Down