[EAGLE-3230] [EAGLE-3452] Create tests users can run for triton model upload and fix error in python 3.11

clarifai/models/model_serving/README.md

@@ -14,19 +14,38 @@ $ clarifai-model-upload-init --model_name <Your model name> \
 ```
 2. Edit the `requirements.txt` file with dependencies needed to run inference on your model and the `labels.txt` (if available in dir) with the labels your model is to predict.
 3. Add your model loading and inference code inside `inference.py` script of the generated model repository under the `setup()` and `predict()` functions respectively. Refer to  The [Inference Script section]() for a description of this file.
-4. Generate a zip of your triton model for deployment via commandline.
+4. Testing your implementation locally by running `<your_triton_folder>/1/test.py` with basic predefined tests.
+To avoid missing dependencies when deploying, recommend to use conda to create clean environment from [Clarifai base envs](./envs/). Then install everything in `requirements.txt`. Follow instruction inside [test.py](./models/test.py) for implementing custom tests.
+  * Create conda env: The base envs are attached in [./envs](./envs/), these are yaml file named as `triton_conda-python_version-torch_version-xxxx.yaml` format. Make sure base env that you're about to create is matched the one in your_triton_folder/triton_conda.yaml. To create conda env and install requirements run:
+```bash
+# create env
+conda env create -n <your_env> -f <base env name>.yaml
+# activate it
+conda activate <your_env>
+# install dependencies
+pip install -r <your_triton_folder>/requirements.txt
+```
+* Then run the test by using pytest:
+
+```bash
+  # Run the test
+  pytest ./your_triton_folder/1/test.py
+  # to see std output
+  pytest --log-cli-level=INFO  -s ./your_triton_folder/1/test.py
+```
+5. Generate a zip of your triton model for deployment via commandline.
 ```console
 $ clarifai-triton-zip --triton_model_repository <path to triton model repository to be compressed> \
     --zipfile_name <name of the triton model zip> (Recommended to use 	  <model_name>_<model-type> convention for naming)
 ```
-5. Upload the generated zip to a public file storage service to get a URL to the zip. This URL must be publicly accessible and downloadable as it's necessary for the last step: uploading the model to a Clarifai app.
-6. Set your Clarifai auth credentials as environment variables.
+6. Upload the generated zip to a public file storage service to get a URL to the zip. This URL must be publicly accessible and downloadable as it's necessary for the last step: uploading the model to a Clarifai app.
+7. Set your Clarifai auth credentials as environment variables.
 ```console
 $ export CLARIFAI_USER_ID=<your clarifai user_id>
 $ export CLARIFAI_APP_ID=<your clarifai app_id>
 $ export CLARIFAI_PAT=<your clarifai PAT>
 ```
-7. Upload your model to Clarifai. Please ensure that your configuration field maps adhere to [this](https://github.com/Clarifai/clarifai-python-utils/blob/main/clarifai/models/model_serving/model_config/deploy.py)
+8. Upload your model to Clarifai. Please ensure that your configuration field maps adhere to [this](https://github.com/Clarifai/clarifai-python-utils/blob/main/clarifai/models/model_serving/model_config/deploy.py)
 ```console
 $ clarifai-upload-model --url <URL to your model zip. Your zip file name is expected to have "zipfile_name" format (in clarifai-triton-zip), if not you need to specify your model_id and model_type> \
     --model_id <Your model ID on the platform> \
@@ -47,6 +66,7 @@ $ clarifai-upload-model --url <URL to your model zip. Your zip file name is expe
     └── 1/
         ├── __init__.py
         ├── inference.py
+        ├── test.py
         └── model.py
 
 A generated triton model repository looks as illustrated in the directory tree above. Any additional files such as model checkpoints and folders needed at inference time must all be placed under the `1/` directory.
@@ -61,6 +81,7 @@ A generated triton model repository looks as illustrated in the directory tree a
 | `triton_conda.yaml` | Contains dependencies available in pre-configured execution environment. |
 | `1/inference.py` | The inference script where users write their inference code. |
 | `1/model.py` | The triton python backend model file run to serve inference requests. |
+| `1/test.py` | Contains some predefined tests in order to test inference implementation and dependencies locally. |
 
 ## Inference Script
 

clarifai/models/model_serving/envs/triton_conda-cp3.8-torch2.0.0-ce980f28.yaml

@@ -0,0 +1,51 @@
+name: triton_conda-cp3.8-torch2.0.0-ce980f28
+channels:
+  - conda-forge
+dependencies:
+  - _libgcc_mutex=0.1=conda_forge
+  - _openmp_mutex=4.5=2_gnu
+  - bzip2=1.0.8=h7f98852_4
+  - ca-certificates=2023.5.7=hbcca054_0
+  - ld_impl_linux-64=2.40=h41732ed_0
+  - libffi=3.4.2=h7f98852_5
+  - libgcc-ng=13.1.0=he5830b7_0
+  - libgomp=13.1.0=he5830b7_0
+  - libnsl=2.0.0=h7f98852_0
+  - libsqlite=3.42.0=h2797004_0
+  - libuuid=2.38.1=h0b41bf4_0
+  - libzlib=1.2.13=hd590300_5
+  - ncurses=6.4=hcb278e6_0
+  - openssl=3.1.1=hd590300_1
+  - pip=23.1.2=pyhd8ed1ab_0
+  - python=3.8.17=he550d4f_0_cpython
+  - readline=8.2=h8228510_1
+  - setuptools=68.0.0=pyhd8ed1ab_0
+  - tk=8.6.12=h27826a3_0
+  - wheel=0.40.0=pyhd8ed1ab_0
+  - xz=5.2.6=h166bdaf_0
+  - pip:
+      - cmake==3.26.4
+      - filelock==3.12.2
+      - jinja2==3.1.2
+      - lit==16.0.6
+      - markupsafe==2.1.3
+      - mpmath==1.3.0
+      - networkx==3.1
+      - numpy==1.24.2
+      - nvidia-cublas-cu11==11.10.3.66
+      - nvidia-cuda-cupti-cu11==11.7.101
+      - nvidia-cuda-nvrtc-cu11==11.7.99
+      - nvidia-cuda-runtime-cu11==11.7.99
+      - nvidia-cudnn-cu11==8.5.0.96
+      - nvidia-cufft-cu11==10.9.0.58
+      - nvidia-curand-cu11==10.2.10.91
+      - nvidia-cusolver-cu11==11.4.0.1
+      - nvidia-cusparse-cu11==11.7.4.91
+      - nvidia-nccl-cu11==2.14.3
+      - nvidia-nvtx-cu11==11.7.91
+      - opencv-python==4.7.0.72
+      - pillow==9.4.0
+      - sympy==1.12
+      - torch==2.0.0
+      - triton==2.0.0
+      - typing-extensions==4.5.0

clarifai/models/model_serving/model_config/triton_config.py

@@ -145,8 +145,8 @@ class TritonModelConfig:
   image_shape: List  #(H, W)
   input: List[InputConfig] = field(default_factory=list)
   output: List[OutputConfig] = field(default_factory=list)
-  instance_group: Device = Device()
-  dynamic_batching: DynamicBatching = DynamicBatching()
+  instance_group: Device = field(default_factory=Device)
+  dynamic_batching: DynamicBatching = field(default_factory=DynamicBatching)
   max_batch_size: int = 1
   backend: str = "python"
 

clarifai/models/model_serving/models/default_test.py

@@ -0,0 +1,230 @@
+import dataclasses
+import inspect
+import logging
+import os
+import unittest
+
+import numpy as np
+
+from ..model_config.triton_config import TritonModelConfig
+from .output import (ClassifierOutput, EmbeddingOutput, ImageOutput, MasksOutput, TextOutput,
+                     VisualDetectorOutput)
+
+PREDEFINED_TEXTS = ["Photo of a cat", "A cat is playing around"]
+
+PREDEFINED_IMAGES = [
+    np.zeros((100, 100, 3), dtype='uint8'),  #black
+    np.ones((100, 100, 3), dtype='uint8') * 255,  #white
+    np.random.uniform(0, 255, (100, 100, 3)).astype('uint8')  #noise
+]
+
+
+class DefaultTestInferenceModel(unittest.TestCase):
+  """
+  This file contains test cases:
+  * Test triton config of current model vs default config
+  * Test if labels.txt is valid for specific model types
+  * Test inference with simple inputs
+  ...
+  """
+  __test__ = False
+
+  def triton_get_predictions(self, input_data):
+    """Call InferenceModel.get_predictions method
+
+    Args:
+        input_data (Union[np.ndarray, str]):
+          if model receives image or vector then type is `np.ndarray`. Otherwise `string`
+
+    Returns:
+       One of types in models.output
+    """
+    return inspect.unwrap(self.triton_model.inference_obj.get_predictions)(
+        self.triton_model.inference_obj, input_data)
+
+  def _get_preprocess(self):
+    """ preprocess if input is image """
+    if "image" in self.triton_model_input_name:
+      h, w, _ = self.triton_model_config.input[0].dims
+      if h > -1 and w > -1:
+        import cv2
+
+        def _f(x):
+          logging.info(f"Preprocess reshape image => {(w, h, 3)}")
+          return cv2.resize(x, (w, h))
+
+    return None
+
+  def intitialize(
+      self,
+      model_type: str,
+      repo_version_dir: str,
+      is_instance_kind_gpu: bool = True,
+  ):
+    import sys
+    sys.path.append(repo_version_dir)
+    self.model_type = model_type
+    self.is_instance_kind_gpu = is_instance_kind_gpu
+    logging.info(self.model_type)
+    from model import TritonPythonModel
+
+    # Construct TritonPythonModel object
+    self.triton_model = TritonPythonModel()
+    self.triton_model.initialize(
+        dict(
+            model_repository=os.path.join(repo_version_dir, ".."),
+            model_instance_kind="GPU" if self.is_instance_kind_gpu else "cpu"))
+    # Get default config of model and model_type
+    self.default_triton_model_config = TritonModelConfig(
+        model_name=self.model_type,
+        model_version="1",
+        model_type=self.model_type,
+        image_shape=[-1, -1])
+    # Get current model config
+    self.triton_model_config = self.triton_model.config_msg
+    self.triton_model_input_name = self.triton_model.input_name
+    self.preprocess = self._get_preprocess()
+    # load labels
+    self._required_label_model_types = [
+        "visual-detector", "visual-classifier", "text-classifier", "visual-segmenter"
+    ]
+    self.labels = []
+    if self.model_type in self._required_label_model_types:
+      with open(os.path.join(repo_version_dir, "../labels.txt"), 'r') as fp:
+        labels = fp.readlines()
+        if labels:
+          self.labels = [line for line in labels if line]
+
+  def test_triton_config(self):
+    """ test Triton config"""
+    # check if input names are still matched
+    self.assertEqual(
+        self.triton_model_input_name, self.default_triton_model_config.input[0].name,
+        "input name of current model vs generated model must be matched "
+        f"{self.triton_model_input_name} != {self.default_triton_model_config.input[0].name}")
+    # check if output names are still matched
+    default_output_names = [each.name for each in self.default_triton_model_config.output]
+    for output_name in self.triton_model_config.output:
+      self.assertIn(output_name.name, default_output_names,
+                    "output name of current model vs generated model must be matched "
+                    f"{output_name.name} not in {default_output_names}")
+
+  def test_having_labels(self):
+    if self.model_type in self._required_label_model_types:
+      self.assertTrue(
+          len(self.labels),
+          f"`labels.txt` is empty!. Model type `{self.model_type}` requires input labels in `labels.txt`"
+      )
+
+  def test_inference_with_predefined_inputs(self):
+    """ Test Inference with predefined inputs """
+
+    if self.preprocess:
+      inputs = [self.preprocess(inp) for inp in PREDEFINED_IMAGES]
+    elif "image" in self.triton_model_input_name:
+      inputs = PREDEFINED_IMAGES
+      logging.info(inputs[0].shape)
+    else:
+      inputs = PREDEFINED_TEXTS
+    outputs = [self.triton_get_predictions(inp) for inp in inputs]
+
+    # Test for specific model type:
+    # 1. length of output array vs config
+    # 2. type of outputs
+    # 3. test range value, shape and dtype of output
+    def _is_valid_logit(x: np.array):
+      return np.all(0 <= x) and np.all(x <= 1)
+
+    def _is_non_negative(x: np.array):
+      return np.all(x >= 0)
+
+    def _is_integer(x):
+      return np.all(np.equal(np.mod(x, 1), 0))
+
+    for inp, output in zip(inputs, outputs):
+
+      field = dataclasses.fields(output)[0].name
+      self.assertEqual(
+          len(self.triton_model_config.output[0].dims),
+          len(getattr(output, field).shape),
+          "Length of 'dims' of config and output must be matched, but get "
+          f"Config {len(self.triton_model_config.output[0].dims)} != Output {len(getattr(output, field).shape)}"
+      )
+
+      if self.model_type == "visual-detector":
+        logging.info(output.predicted_labels)
+        self.assertEqual(
+            type(output), VisualDetectorOutput,
+            f"Output type must be `VisualDetectorOutput`, but got {type(output)}")
+        self.assertTrue(
+            _is_valid_logit(output.predicted_scores), "`predicted_scores` must be in range [0, 1]")
+        self.assertTrue(
+            _is_non_negative(output.predicted_bboxes), "`predicted_bboxes` must be >= 0")
+        self.assertTrue(
+            np.all(0 <= output.predicted_labels) and
+            np.all(output.predicted_labels < len(self.labels)),
+            f"`predicted_labels` must be in [0, {len(self.labels) - 1}]")
+        self.assertTrue(_is_integer(output.predicted_labels), "`predicted_labels` must be integer")
+
+      elif self.model_type == "visual-classifier":
+        self.assertEqual(
+            type(output), ClassifierOutput,
+            f"Output type must be `ClassifierOutput`, but got {type(output)}")
+        self.assertTrue(
+            _is_valid_logit(output.predicted_scores), "`predicted_scores` must be in range [0, 1]")
+        if self.labels:
+          self.assertEqual(
+              len(output.predicted_scores),
+              len(self.labels),
+              f"`predicted_labels` must equal to {len(self.labels)}, however got {len(output.predicted_scores)}"
+          )
+
+      elif self.model_type == "text-classifier":
+        self.assertEqual(
+            type(output), ClassifierOutput,
+            f"Output type must be `ClassifierOutput`, but got {type(output)}")
+        self.assertTrue(
+            _is_valid_logit(output.predicted_scores), "`predicted_scores` must be in range [0, 1]")
+        if self.labels:
+          self.assertEqual(
+              len(output.predicted_scores),
+              len(self.labels),
+              f"`predicted_labels` must equal to {len(self.labels)}, however got {len(output.predicted_scores)}"
+          )
+
+      elif self.model_type == "text-embedder":
+        self.assertEqual(
+            type(output), EmbeddingOutput,
+            f"Output type must be `EmbeddingOutput`, but got {type(output)}")
+        self.assertNotEqual(output.embedding_vector.shape, [])
+
+      elif self.model_type == "text-to-text":
+        self.assertEqual(
+            type(output), TextOutput, f"Output type must be `TextOutput`, but got {type(output)}")
+
+      elif self.model_type == "text-to-image":
+        self.assertEqual(
+            type(output), ImageOutput,
+            f"Output type must be `ImageOutput`, but got {type(output)}")
+        self.assertTrue(_is_non_negative(output.image), "`image` elements must be >= 0")
+
+      elif self.model_type == "visual-embedder":
+        self.assertEqual(
+            type(output), EmbeddingOutput,
+            f"Output type must be `EmbeddingOutput`, but got {type(output)}")
+        self.assertNotEqual(output.embedding_vector.shape, [])
+
+      elif self.model_type == "visual-segmenter":
+        self.assertEqual(
+            type(output), MasksOutput,
+            f"Output type must be `MasksOutput`, but got {type(output)}")
+        self.assertTrue(_is_integer(output.predicted_mask), "`predicted_mask` must be integer")
+        if self.labels:
+          self.assertTrue(
+              np.all(0 <= output.predicted_mask) and
+              np.all(output.predicted_mask < len(self.labels)),
+              f"`predicted_mask` must be in [0, {len(self.labels) - 1}]")
+
+
+if __name__ == '__main__':
+  unittest.main()