Build Tensorflow and libtensorflow from source

[abbradar]

Motivation for this change

Restore Tensorflow source build while leaving binary builds . Based on @yorickvP and @timokau work.with some parts taken from both.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

---

cc @uri-canva for changes regaring Bazel.

Tested both CUDA and non-CUDA builds of the Python library, both source and binary builds. Didn't test libtensorflow, I assume that if Python library works and ldd output is okay libtensorflow is okay too.

[abbradar]

Forgot to mention actual previous PRs: #63208 and #63616.

[abbradar]

Darwin still uses binary builds for now. I don't have a Darwin machine so I can't test if source build works and fix it if it doesn't - unless someone wants to take on that role!

[abbradar]

(For some reason I cannot request a review from @yorickvP on Github)

[abbradar]

Updated binary libtensorflow doesn't build for Darwin. Can someone with Darwin knowledge help with it?

[danieldk]

Updated binary libtensorflow doesn't build for Darwin. Can someone with Darwin knowledge help with it?

It seems that Google now build on newer macOS versions, I had the same problem with Tensorflow 1.13.1 when building on my own MacBook with Mojave. A newer cctools is needed, see #49371. I think this is the relevant PR: #61172

[yorickvP]

This is great! I'm hoping to get a fully static build at some point, but having the same protobuf on tensorflow as the rest of the code should fix our linker errors! :)

You can't request a review from me because I don't have commit rights, I suppose.

[yorickvP]

whoever built tensorflow upstream must have patched out that saved_model_portable_proto line. it's been in there for several version snow.

[timokau]

As I mentioned in #63616, there are still issues with the python2 build. Seems to be a general problem with parallelism on python2, only manifests itself here because its such a big project and basically hopeless to build without a lot of parallelism (used 12 cores).

I already added one patch to python2 (#64067), but I still saw the issue (sometimes, its transient) after that. I suspect the second part of the fix in the python ticket is also needed. That was never backported by debian, so I had to do it myself (being not very experienced with C). I've pushed those changes to the branch of the other PR. Its not very polished, which is why I didn't push it before.

I don't have time to work on this or give extensive review right now. But I can answer questions if you have any.

[abbradar]

@timokau I saw that one but IIUC this is a general Python problem, not Tensorflow-specific one, so I didn't really look into it. It shouldn't happen during the "big" Tensorflow build because I only build "source" package now and build the wheel separately with our buildPythonProgram. I didn't observe the problem on 8-core (16-thread) machine during ~10 builds too.

[timokau]

Neat, so you only need bazel for the C part and don't need to unnecessarily rebuild everything for libtensorflow, py2 tensorflow and py3 tensorflow?

[abbradar]

Sadly no, you still need to rebuild everything because their C libraries link to different Pythons and I presume use different APIs. Still, in the end you get something that resembles a usual source Python package with setup.py etc. I use this as src for buildPythonPackage so any Python bugs should be isolated there.

[timokau]

Oh, so you mean you just skip the final bdist_wheel as it was done previously? I actually changed that on purpose, so that we can unify the binary and source builds. That way we wouldn't have all the duplication between default.nix and bin.nix.

I don't think the bdist_wheel step alone was responsible for the build failures, but who knows.

[abbradar]

The primary reason I switched back to the previous behavior was to ensure fixupPhase runs on unpacked Tensorflow libraries. This ensures RPATH is set up correctly.

[yorickvP]

Will this work with bazel 0.28.0? #64633

[timokau]

My PR had some issues with bazel 0.27, not sure if @abbradar worked around those. But the bazel team has promised no further backwards incompatibilities for the next 3 releases (then they'll break backwards compatibility once more and release 1.0).

[abbradar]

But the bazel team has promised no further backwards incompatibilities for the next 3 releases (then they'll break backwards compatibility once more and release 1.0).

Nice news! I didn't test 0.28 though yet, wanted to do it today but sadly I won't have time for one more rebuild.

some issues with bazel 0.27, not sure if @abbradar worked around those

Huh, didn't notice anything. Maybe you remember something specific? I apply patch for 0.27 from upstream and also add a flag to enable backwards compatibility with some behavior.

[abbradar]

Now that other parts are here I'll wait till staging goes into master and merge this (so that we don't break Python 2 + Tensorflow).

[timokau]

Why not just retarget this PR to staging and merge now? Then we'll get it automatically on the next staging-next merge.

[abbradar]

Well, I usually avoid crowding staging with not-mass-rebuildy PRs to ease bisect in case it's needed (searching for problems among mass rebuild commits can be painful).

…

On July 17, 2019 2:23:15 PM GMT+03:00, Timo Kaufmann ***@***.***> wrote: Why not just retarget this PR to staging and merge now? Then we'll get it automatically on the next staging-next merge. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #64716 (comment)

-- Nikolay.

[timokau]

Fair enough.

[timokau]

I tried to build tensorflow with SSE4.1 support, but unfortunately the build failed. Apparently that has to be specified at configure time now. This is my attempt to get it working:
https://github.com/timokau/nixpkgs/commits/3323284
The patch to buildBazelPackages is necessary since tensorflow won't accept the --config=opt flag for fetching.

Feel free to cherry-pick and/or rebase.

[timokau]

@abbradar in case you haven't noticed yet, the tensorflow build (py2 and py3, but there are different issues for py2 and py3 I think) is currently failing. Unfortunately I don't have access to the necessary compute resources to debug this right now.

[abbradar]

Fixed in 7109546. Thanks for the heads up!