Building TensorFlow Serving

Building TensorFlow Serving

The instructions provided here specify the steps to build TensorFlow Serving version 2.13.0 on Linux on IBM Z for the following distributions:

• Ubuntu (20.04, 22.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1: Build and Install TensorFlow Serving

1.1) Build using script

TensorFlow Serving can be built manually using STEP 1.2.

Use the following commands to build TensorFlow Serving using the build script. Please ensure wget is installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/TensorflowServing/2.13.0/build_tensorflow_serving.sh

bash build_tensorflow_serving.sh    [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 2. In case of error, check logs for more details or go to STEP 1.2 to follow manual build steps.

1.2) Build and Install TensorFlow 2.13.0

• Set environment variable SOURCE_ROOT:

  export SOURCE_ROOT=/<source_root>/

• Instructions for building TensorFlow 2.13.0 can be found here.

1.3) Build TensorFlow Serving

• Download source code

  cd $SOURCE_ROOT
  git clone https://github.com/tensorflow/serving
  cd serving
  git checkout 2.13.0

• Apply patches

  export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/TensorflowServing/2.13.0/patch"
  wget -O tfs_patch.diff $PATCH_URL/tfs_patch.diff
  sed -i "s?SOURCE_ROOT?$SOURCE_ROOT?" tfs_patch.diff
  git apply tfs_patch.diff

• Build TensorFlow Serving

  Tensorflow Serving can be built as follows:

  cd $SOURCE_ROOT/serving
  bazel build tensorflow_serving/...

  Note: TensorFlow Serving build is resource intensive operation. If build continues to fail try increasing the swap space or reducing the number of concurrent jobs by specifying --jobs=n in the build command above, where n is the number of concurrent jobs.

  Copy binary to access it from anywhere, make sure /usr/local/bin is in $PATH. Run command:

  sudo cp bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server /usr/local/bin/.
  tensorflow_model_server --version

  The following output should be seen in the console:

  TensorFlow ModelServer: 2.13.0-rc2+dev.sha.83c949cd
  TensorFlow Library: 2.13.0

1.4) Install TensorFlow Serving API pip package

sudo pip3 install tensorflow-serving-api==2.13.0

Step 2: Verify TensorFlow Serving (Optional)

• Run TensorFlow Serving from command Line

   tensorflow_model_server --rest_api_port=8501 --model_name=<model_name> --model_base_path=<model_path> &

  • For example:

   export TESTDATA="$SOURCE_ROOT/serving/tensorflow_serving/servables/tensorflow/testdata"
   # Start TensorFlow Serving model server and open the REST API port
   tensorflow_model_server --rest_api_port=8501 --model_name=half_plus_two --model_base_path=$TESTDATA/saved_model_half_plus_two_cpu &
   # Query the model using the predict API
   curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://localhost:8501/v1/models/half_plus_two:predict

  Output should look like:

  {
      "predictions": [2.5, 3.0, 4.5
      ]
  }
  

Step 3: Execute Test Suite (Optional)

• Run complete testsuite

  cd $SOURCE_ROOT/serving
  bazel test tensorflow_serving/...

  All test cases should pass.

References:

• https://www.tensorflow.org/tfx/guide/serving
• https://github.com/tensorflow/serving