[tensorflow] Add C port (#14392)

* fix build issues occurring from default bazel config being used because customized bazel config is stored in wrong directory

* [tensorflow-cc] Update CONTROL and ci.baseline.txt

* fix also applies to windows static build

* fix debug and static builds as well as library naming for non-linux
- add patch to fix debug builds
- add patch to fix exports for static linking
- really build debug (instead of cloning release)
- override bazel build options for debug (work around bazel bug)
- bazel doesn't support static libraries: work around by building dynamic library and constructing static linkage commands from build log
- Windows .pdb file can't be >4GB even on x64: work around using reduced debug information
- Windows doesn't support .lib files >4GB even on x64, so split into multiple libs
- vcpkg requires equal amount of libs for debug and release: work around using handcrafted empty dummy libs
- fix naming of libs (.dll on Windows and .dylib on macOS)

* upgrade tensorflow from v1.14 to v2.3
- adapt patch files to tensorflow code changes
- update bazel from v0.25.2 to v3.1
- on Windows use python installed on the host instead of embedded python obtained via vcpkg because embedded python lacks pip, which we need to obtain numpy
- on Windows add MSYS2 to the PATH so that bazel tools can access MSYS2 GIT
- add support for custom CA certificates when using HTTPS_PROXY

* fix execute process macro
The existing implementation totally screwed up commands if the command's arguments contained semicolons (this is the case, e.g., in the FindPython modules of the cmake distribution).

* extend overriden execute_process to more than one COMMAND as there actually are use cases for this

* added another patch required for tensorflow v2.3, fixed path and working directory

* Revert "incorporate changes from microsoft:master"

* Revert "Revert "incorporate changes from microsoft:master""

* final fixes for static build + improving out messages

* enabling linux and osx in CI to see if it works now

* simplified code, fixed version numbers, fixed generated include cmake file

* fix failing postbuild check on handcrafted empty dummy library by spreading the last real libraries contents over the required number of libraries

* remove dead code commit by mistake again

* improvements from code review

* cleaner fix for debug code

* find pip3 in PATH (PYTHON3_DIR apparently not valid for pip3)

* fix error in python helper script

* fix wrong libname in postbuild script

* fix python detection + switch to python on msys2 (instead of embedded python) for Windows as we need numpy

* fix order of arguments

* fix command (it may contain spaces such as C:\Program Files\...)

* revert last commit (root cause for CI failures is something different: there are line breaks in path)

* fix regex comparision
(value needs to be escaped as it may contains regex special characters such as brackets, eg C:/Program Files (x86)/...)

* fix linebreaks in generated file

* fix CRT linkage
(macOS doesn't support static CRT linkage; it's set to dynamic even static target triplets for macOS and linux)

* refactor implemenation to avoid as much code duplication as possible -- algorithmically identical

* fix version numbers in helper scripts

* enable work-around for Windows until bazel fix is available

* install missing python3-pip on linux

* fix linux build by patching

* apply timeout feature now available via merged master branch

* correct linux build patch

* improve debug build patches
(no functional difference because LOG(FATAL, ...) macro internally anyway calls abort(), which the compiler doesn't detect in debug mode...

* improve linux patch

* temporarily add debug to inspect what's going on on macOS CI

* remove temporary debug code and fix static linking scripts for linux and macOS

* fix regex escaping

* fix ambiguous match while grepping for the framework link command

* extend fix of ambiguous match while grepping for the framework link command

* fix what merge of master broke

* fix more what got broken by merging master
(all packages and their dependencies are now maintained manually instead of using pacman...)

* remove "unofficial" from filename

* added switch do distinct classic and manifest mode when generating config.cmake file

* create symlinks for libraries without version number

* fix linux postbuild script

* temporarily disable code making problems

* add note for linking on Linux and macOS

* forget to add README file in previous commit

* add file forgotton in macro fixing patch

* fix python library path

* fix macOS static link command

* update linkage instructions in README

* Update ports/tensorflow-cc/CONTROL

* Update ports/tensorflow-cc/portfile.cmake

* Update scripts/ci.baseline.txt

* use vcpkg_execute_required_process

* pass C_FLAGS and CXX_FLAGS to bazel

* fix INTERFACE_INCLUDE_DIRECTORIES

* fix optional c/cxx arguments

* also add linker opts

* update README

* merge static libs into one
to support force_load (cannot force_load both due to duplicate symbols)

* update README

* quote python path (it might contain spaces that don't get escaped inside outer quotes of bash command)

* fix python path also for static build

* add arm(64) as currently unsupported arch

* bazel 3.7 is available -> remove workaround

* update README, remove necessary c-ares from deps

* update msys package

* add uwp specific options, and minor general improvements

* fix string replace

* fix control file and windows path separator

* revert backslashes-fix -- the root cause was missing .exe extension

* upgrade to tf 2.3.1

* fix hard-coded version

* remove uwp work-in-progress code so that PR can be merged

* add [tensorflow] C API port

* missing in previous commit

* fix include file

* remove unnecessary suffix

* update README and print out usage info in portfile

* Update ports/tensorflow-cc/CONTROL

Co-authored-by: NancyLi1013 <[email protected]>

* Update ports/tensorflow/portfile.cmake

Co-authored-by: NancyLi1013 <[email protected]>

* follow-up to code review

* fix suffix parameter

* fix quoting

* extend linux patch

* another try to fix quoting of possibly empty string parameter

* different approach to fix empty string arguments

* update list of headers for tensorflow 2.3.1 (was still 2.3.0)

* Update ports/tensorflow/CONTROL

Co-authored-by: NancyLi1013 <[email protected]>

* move common stuff into tensorflow-common as discussed in review

* Apply suggestions from code review

Co-authored-by: nicole mazzuca <[email protected]>

Co-authored-by: Gehweiler <[email protected]>
Co-authored-by: wangli28 <[email protected]>
Co-authored-by: Jack·Boos·Yu <[email protected]>
Co-authored-by: Joachim Gehweiler <[email protected]>
Co-authored-by: NancyLi1013 <[email protected]>
Co-authored-by: nicole mazzuca <[email protected]>

ports/tensorflow-cc/CONTROL

@@ -1,4 +1,6 @@
 Source: tensorflow-cc
 Version: 2.3.1
-Description: Library for computation using data flow graphs for scalable machine learning
+Port-Version: 1
+Homepage: https://github.com/tensorflow/tensorflow
+Description: Library for computation using data flow graphs for scalable machine learning (C++ API version)
 Supports: !(x86|arm|uwp)

ports/tensorflow-common/CONTROL

@@ -0,0 +1,6 @@
+Source: tensorflow-common
+Version: 2.3.1
+Port-Version: 0
+Homepage: https://github.com/tensorflow/tensorflow
+Description: This meta package holds common files for the C [tensorflow] and the C++ [tensorflow-cc] API version of TensorFlow but is not installable on its own.
+Supports: false

ports/tensorflow-cc/convert_lib_params_linux.py → ports/tensorflow-common/convert_lib_params_linux.py

@@ -1,8 +1,9 @@
 import sys
 
 version = sys.argv[1]
+lib_suffix = "" if len(sys.argv) < 3 else sys.argv[2]
 
-with open(f"libtensorflow_cc.a.{version}-2.params", "w") as f_out:
+with open(f"libtensorflow{lib_suffix}.a.{version}-2.params", "w") as f_out:
     parts = []
     with open(f"libtensorflow_framework.so.{version}-2.params", "r") as f_in:
         skip_next = False
@@ -16,7 +17,7 @@
                 f_out.write(line)
                 parts.append(line)
     parts = set(parts)
-    with open(f"libtensorflow_cc.so.{version}-2.params", "r") as f_in:
+    with open(f"libtensorflow{lib_suffix}.so.{version}-2.params", "r") as f_in:
         skip_next = False
         for line in f_in:
             if skip_next:

ports/tensorflow-cc/convert_lib_params_windows.py → ports/tensorflow-common/convert_lib_params_windows.py

@@ -1,11 +1,13 @@
 import os
 import sys
 
+lib_suffix = "" if len(sys.argv) < 2 else sys.argv[1]
+
 file_no = 1
-with open("tensorflow_cc.dll-2.params", "r") as f_in:
+with open(f"tensorflow{lib_suffix}.dll-2.params", "r") as f_in:
     lib_name = None
     acc_size = 0
-    f_out = open("tensorflow_cc.lib-2.params-part1", "w")
+    f_out = open(f"tensorflow{lib_suffix}.lib-2.params-part1", "w")
     for line in f_in:
         if line.startswith("/OUT:"):
             lib_name = line
@@ -18,10 +20,9 @@
                 # we need to split the library if it is >4GB, because it's not supported even on x64 Windows
                 f_out.close()
                 file_no += 1
-                f_out = open(f"tensorflow_cc.lib-2.params-part{file_no}", "w")
+                f_out = open(f"tensorflow{lib_suffix}.lib-2.params-part{file_no}", "w")
                 acc_size = 0
                 f_out.write(lib_name.replace(".dll", f"-part{file_no}.lib"))
             acc_size += size
             f_out.write(line)
     f_out.close()
-

ports/tensorflow-cc/fix-linux-build.patch → ports/tensorflow-common/fix-linux-build.patch

@@ -62,3 +62,16 @@ diff --git a/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc b/
  // ==== Snapshot Implementation ====
 
  /* The current snapshot on-disk layout is as follows:
+
+diff --git a/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc b/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc
+--- a/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc
++++ b/tensorflow/core/kernels/data/experimental/data_service_dataset_op.cc
+@@ -52,6 +52,8 @@
+ /* static */ constexpr const char* const
+     DataServiceDatasetOp::kMaxOutstandingRequests;
+ /* static */ constexpr const char* const
++    DataServiceDatasetOp::kTaskRefreshIntervalHintMs;
++/* static */ constexpr const char* const
+     DataServiceDatasetOp::kIterationCounter;
+ /* static */ constexpr const char* const DataServiceDatasetOp::kOutputTypes;
+ /* static */ constexpr const char* const DataServiceDatasetOp::kOutputShapes;

ports/tensorflow-cc/generate_static_link_cmd_linux.py → ports/tensorflow-common/generate_static_link_cmd_linux.py

@@ -1,10 +1,11 @@
 import re
 import sys
 
+lib_suffix = "" if len(sys.argv) < 3 else sys.argv[2]
 with open(sys.argv[1], "r") as f_in:
     with open("static_link.sh", "w") as f_out:
         p_cd = re.compile("^\\((cd .*) && \\\\$")
-        p_linker = re.compile("^\\s*(.+)gcc.+(@bazel-out\\S+libtensorflow_cc\\.so\\.\\d\\.\\d\\.\\d-2\\.params).*")
+        p_linker = re.compile(fr"^\s*(.+)gcc.+(@bazel-out\S+libtensorflow{lib_suffix}\.so\.\d\.\d\.\d-2\.params).*")
         f_out.write("#!/bin/bash\n# note: ar/binutils version 2.27 required to support output files > 4GB\n")
         env = []
         for line in f_in:

ports/tensorflow-cc/generate_static_link_cmd_macos.py → ports/tensorflow-common/generate_static_link_cmd_macos.py

@@ -1,10 +1,11 @@
 import re
 import sys
 
+lib_suffix = "" if len(sys.argv) < 3 else sys.argv[2]
 with open(sys.argv[1], "r") as f_in:
     with open("static_link.sh", "w") as f_out:
         p_cd = re.compile("^\\((cd .*) && \\\\$")
-        p_linker1 = re.compile("^.*cc_wrapper.sh.+-shared.+-o (bazel-out\\S+libtensorflow_cc\\.\\d\\.\\d\\.\\d\\.dylib)")
+        p_linker1 = re.compile(fr"^.*cc_wrapper.sh.+-shared.+-o (bazel-out\S+libtensorflow{lib_suffix}\.\d\.\d\.\d\.dylib)")
         p_linker2 = re.compile("^.*cc_wrapper.sh.+-shared.+-o (bazel-out\\S+libtensorflow_framework\\.\\d\\.\\d\\.\\d\\.dylib)")
         f_out.write("#!/bin/bash\n# note: ar/binutils version 2.27 required to support output files > 4GB\n")
         env = []
@@ -39,7 +40,7 @@
                         tmp = [t[16:] for t in tokens if t.startswith("-Wl,-force_load,")]
                         old = set(parts)
                         parts += [t for t in tmp if t not in old]
-                        line = f"libtool -static -o {m2.group(1).replace('_framework', '_cc').replace('.dylib', '.a')} {' '.join(parts)}\n"
+                        line = f"libtool -static -o {m2.group(1).replace('_framework', lib_suffix).replace('.dylib', '.a')} {' '.join(parts)}\n"
                         f_out.write(line)
                         break
                 else:

ports/tensorflow-cc/generate_static_link_cmd_windows.py → ports/tensorflow-common/generate_static_link_cmd_windows.py

@@ -2,10 +2,11 @@
 import re
 import sys
 
+lib_suffix = "" if len(sys.argv) < 3 else sys.argv[2]
 with open(sys.argv[1], "r") as f_in:
     with open("static_link.bat", "w") as f_out:
         p_setenv = re.compile("^\s*(SET .+=.*)$")
-        p_linker = re.compile(".+link\\.exe.+tensorflow_cc\\.dll-2\\.params.*")
+        p_linker = re.compile(fr".+link\.exe.+tensorflow{lib_suffix}\.dll-2\.params.*")
         env = []
         for line in f_in:
             if line.startswith("cd"):
@@ -28,7 +29,7 @@
                                 t = t[:-len("link.exe")] + "lib.exe\""
                             elif t == "/DLL" or t.lower()[1:].startswith("defaultlib:") or t.lower()[1:].startswith("ignore") or t.startswith("/OPT:") or t.startswith("/DEF:") or t.startswith("/DEBUG:") or t.startswith("/INCREMENTAL:"):
                                 continue
-                            elif t[0] == '@' and t.endswith("tensorflow_cc.dll-2.params"):
+                            elif t[0] == '@' and t.endswith(f"tensorflow{lib_suffix}.dll-2.params"):
                                 t = t[:-len("dll-2.params")] + "lib-2.params-part1"
                                 params_file = t[1:-len("-part1")]
                             line += t + " "

ports/tensorflow-common/portfile.cmake

@@ -0,0 +1 @@
+message(FATAL_ERROR "This meta package holds common files for the C [tensorflow] and the C++ [tensorflow-cc] API version of TensorFlow but is not installable on its own.")