Implement integration tests in CI pipeline (won't be merged)

.buildkite/README_dynamic.md

@@ -0,0 +1,138 @@
+# Buildkite Pipeline Generator
+
+This document outlines the process for using Python scripts to automatically generate Buildkite CI/CD pipeline configuration files. These scripts leverage templates to create consistent testing pipelines for both models and features.
+
+## Overview
+
+The primary goal of these tools is to streamline the creation of Buildkite pipelines. Instead of manually creating and editing YAML files for each new model or feature, you can run a simple command to generate a standardized pipeline file.
+
+There are two main generators:
+
+1.  **Model Pipeline Generator** (`generate_model_buildkite.py`): Creates a pipeline file for testing a specific machine learning model.
+2.  **Feature Pipeline Generator** (`generate_feature_buildkite.py`): Creates a pipeline file for testing a new feature.
+
+Both scripts work by reading a corresponding template file (`.yml`), replacing placeholder variables with your command-line arguments, and saving the result as a new YAML file in a designated output directory.
+
+## Directory Structure
+
+To use the scripts, your files should be arranged as follows. The output directories (`models/` and `features/`) will be created automatically if they do not exist.
+
+```
+.
+├── generate_model_buildkite.py
+├── buildkite_ci_model_template.yml
+├── generate_feature_buildkite.py
+├── buildkite_ci_feature_template.yml
+└── README.md
+```
+
+-----
+
+## How to Use
+
+### 1\. Generating a Model Pipeline
+
+Use the `generate_model_buildkite.py` script to create a CI pipeline for a new model.
+
+**Command:**
+
+```bash
+python generate_model_buildkite.py --model-name <MODEL_NAME> --queue <QUEUE_NAME>
+```
+
+**Arguments:**
+
+  * `--model-name` (required): The name of the model to be tested. If the name contains special characters like `/` or `.`, they will be replaced with `_` in the output filename and for Buildkite step keys.
+  * `--queue` (required): The name of the Buildkite agent queue where the jobs will run (e.g., `tpu_v6e_queue`).
+
+**Example:**
+
+```bash
+python generate_model_buildkite.py --model-name meta-llama/Llama-3.1-8B-Instruct --queue tpu_v6e_queue
+```
+
+**Output:**
+
+This command will generate a new file located at `models/meta-llama_Llama-3_1-8B-Instruct.yml`.
+
+-----
+
+### 2\. Generating a Feature Pipeline
+
+Use the `generate_feature_buildkite.py` script to create a CI pipeline for a new feature.
+
+**Command:**
+
+```bash
+python generate_feature_buildkite.py --feature-name <FEATURE_NAME> --queue <QUEUE_NAME>
+```
+
+**Arguments:**
+
+  * `--feature-name` (required): The name of the feature to be tested.
+  * `--queue` (required): The name of the Buildkite agent queue.
+
+**Example:**
+
+```bash
+python generate_feature_buildkite.py --feature-name Feat-A --queue tpu_v6e_queue
+```
+
+**Output:**
+
+This command will generate a new file located at `features/Feat-A.yml`.
+
+-----
+
+## Important Notes: Placeholders & Customization
+
+The scripts work by performing a find-and-replace on specific placeholders within the template files. You can customize the `buildkite_ci_*_template.yml` files to change the structure of the generated pipelines.
+
+#### **Model Template Placeholders (`buildkite_ci_model_template.yml`)**
+
+  * `{MODEL_NAME}`: Replaced with the exact string provided to `--model-name`. This is typically used in human-readable fields like step `label`.
+  * `{SAFE_MODEL_NAME}`: A sanitized version of the model name, automatically generated by replacing characters like `/` and `.` with `_`. This is used for machine-readable fields like the step `key` and the output filename to ensure validity.
+  * `{QUEUE}`: Replaced with the string provided to `--queue`.
+
+#### **Feature Template Placeholders (`buildkite_ci_feature_template.yml`)**
+
+  * `{FEATURE_NAME}`: Replaced with the exact string provided to `--feature-name`.
+  * `{SAFE_FEATURE_NAME}`: A sanitized version of the feature name.
+  * `{QUEUE}`: Replaced with the string provided to `--queue`.
+
+## Integration with the Main Pipeline
+
+After generating a pipeline `.yml` file, you must place it in the correct subdirectory within the `.buildkite/` folder. The generator scripts create these files in the top-level `models/` and `features/` directories, so you will need to **manually move** them to the corresponding location inside `.buildkite/`.
+
+This is a crucial step, as the main CI process relies on the `dynamic_bootstrap.sh` script to automatically scan these specific directories to discover and upload the pipeline steps. Also, remember to **replace the script in your main pipeline's configuration to execute `dynamic_bootstrap.sh`**.
+
+### Target Directories
+
+Place your generated `.yml` files into the following directories for detection:
+
+  * **Standard Models**: Move the generated file to `.buildkite/models/`.
+  * **Informational Models**: For models considered "informational" (e.g., VLLM-native models), move the generated file to `.buildkite/models/informational/`.
+  * **Popular Models**: For models designated as "popular," move the generated file to `.buildkite/models/popular/`.
+  * **Features**: Move the generated feature file to `.buildkite/features/`.
+
+### Example Workflow
+
+1.  Generate a pipeline for a new model that you consider "popular":
+
+    ```bash
+    python generate_model_buildkite.py --model-name my-popular-model --queue tpu_v6e_queue
+    ```
+
+    This creates `models/my-popular-model.yml`.
+
+2.  Move the file to the correct directory for the bootstrap script to find it:
+
+    ```bash
+    # Create the directory if it doesn't exist
+    mkdir -p .buildkite/models/popular
+
+    # Move the file
+    mv models/my-popular-model.yml .buildkite/models/popular/
+    ```
+
+Once the file is in the correct `.buildkite/` subdirectory and committed, the `dynamic_bootstrap.sh` script will automatically find it and add its steps to the Buildkite pipeline.

.buildkite/README_generate.md

@@ -0,0 +1,101 @@
+# Buildkite Pipeline Generator
+
+This document outlines the process for using Python scripts to automatically generate Buildkite CI/CD pipeline configuration files. These scripts leverage templates to create consistent testing pipelines for both models and features.
+
+## Overview
+
+The primary goal of these tools is to streamline the creation of Buildkite pipelines. Instead of manually creating and editing YAML files for each new model or feature, you can run a simple command to generate a standardized pipeline file.
+
+There are two main generators:
+
+1.  **Model Pipeline Generator** (`generate_model_buildkite.py`): Creates a pipeline file for testing a specific machine learning model.
+2.  **Feature Pipeline Generator** (`generate_feature_buildkite.py`): Creates a pipeline file for testing a new feature.
+
+Both scripts work by reading a corresponding template file (`.yml`), replacing placeholder variables with your command-line arguments, and saving the result as a new YAML file in a designated output directory.
+
+## Directory Structure
+
+To use the scripts, your files should be arranged as follows. The output directories (`models/` and `features/`) will be created automatically if they do not exist.
+
+```
+.
+├── generate_model_buildkite.py
+├── buildkite_ci_model_template.yml
+├── generate_feature_buildkite.py
+├── buildkite_ci_feature_template.yml
+└── README.md
+```
+
+-----
+
+## How to Use
+
+### 1\. Generating a Model Pipeline
+
+Use the `generate_model_buildkite.py` script to create a CI pipeline for a new model.
+
+**Command:**
+
+```bash
+python generate_model_buildkite.py --model-name <MODEL_NAME> --queue <QUEUE_NAME>
+```
+
+**Arguments:**
+
+  * `--model-name` (required): The name of the model to be tested. If the name contains special characters like `/` or `.`, they will be replaced with `_` in the output filename and for Buildkite step keys.
+  * `--queue` (required): The name of the Buildkite agent queue where the jobs will run (e.g., `tpu_v6e_queue`).
+
+**Example:**
+
+```bash
+python generate_model_buildkite.py --model-name meta-llama/Llama-3.1-8B-Instruct --queue tpu_v6e_queue
+```
+
+**Output:**
+
+This command will generate a new file located at `models/meta-llama_Llama-3_1-8B-Instruct.yml`.
+
+-----
+
+### 2\. Generating a Feature Pipeline
+
+Use the `generate_feature_buildkite.py` script to create a CI pipeline for a new feature.
+
+**Command:**
+
+```bash
+python generate_feature_buildkite.py --feature-name <FEATURE_NAME> --queue <QUEUE_NAME>
+```
+
+**Arguments:**
+
+  * `--feature-name` (required): The name of the feature to be tested.
+  * `--queue` (required): The name of the Buildkite agent queue.
+
+**Example:**
+
+```bash
+python generate_feature_buildkite.py --feature-name Feat-A --queue tpu_v6e_queue
+```
+
+**Output:**
+
+This command will generate a new file located at `features/Feat-A.yml`.
+
+-----
+
+## Important Notes: Placeholders & Customization
+
+The scripts work by performing a find-and-replace on specific placeholders within the template files. You can customize the `buildkite_ci_*_template.yml` files to change the structure of the generated pipelines.
+
+#### **Model Template Placeholders (`buildkite_ci_model_template.yml`)**
+
+  * `{MODEL_NAME}`: Replaced with the exact string provided to `--model-name`. This is typically used in human-readable fields like step `label`.
+  * `{SAFE_MODEL_NAME}`: A sanitized version of the model name, automatically generated by replacing characters like `/` and `.` with `_`. This is used for machine-readable fields like the step `key` and the output filename to ensure validity.
+  * `{QUEUE}`: Replaced with the string provided to `--queue`.
+
+#### **Feature Template Placeholders (`buildkite_ci_feature_template.yml`)**
+
+  * `{FEATURE_NAME}`: Replaced with the exact string provided to `--feature-name`.
+  * `{SAFE_FEATURE_NAME}`: A sanitized version of the feature name.
+  * `{QUEUE}`: Replaced with the string provided to `--queue`.

.buildkite/buildkite_ci_feature_template.yml

@@ -0,0 +1,78 @@
+# {FEATURE_NAME}
+agents:
+  queue: {QUEUE}
+steps:
+  - label: "Unit tests for {FEATURE_NAME}"
+    key: "ut_{SAFE_FEATURE_NAME}"
+    commands:
+      # Replace to actual test commands
+      - echo "[DEBUG], unit testing for {FEATURE_NAME}"
+  - label: "Notifications: Unit tests for {FEATURE_NAME}"
+    key: "notifications_ut_{SAFE_FEATURE_NAME}"
+    depends_on: "ut_{SAFE_FEATURE_NAME}"
+    agents:
+      queue: {QUEUE}
+    env:
+      EXECUTE_ENTITY: "{FEATURE_NAME}"
+      EXECUTE_STAGE: "UnitTest"
+    commands:
+      - |
+        bash .buildkite/scripts/check_results.sh \
+          "Unit tests for {FEATURE_NAME}" ut_{SAFE_FEATURE_NAME}
+
+  - label: "Integration tests for {FEATURE_NAME}"
+    key: "it_{SAFE_FEATURE_NAME}"
+    depends_on: "notifications_ut_{SAFE_FEATURE_NAME}"
+    commands:
+      # Replace to actual test commands
+      - echo "[DEBUG], integration testing for {FEATURE_NAME}"
+  - label: "Notifications: Integration tests for {FEATURE_NAME}"
+    key: "notifications_it_{SAFE_FEATURE_NAME}"
+    depends_on: "it_{SAFE_FEATURE_NAME}"
+    agents:
+      queue: {QUEUE}
+    env:
+      EXECUTE_ENTITY: "{FEATURE_NAME}"
+      EXECUTE_STAGE: "IntTest"
+    commands:
+      - |
+        bash .buildkite/scripts/check_results.sh \
+          "Integration tests for {FEATURE_NAME}" it_{SAFE_FEATURE_NAME}
+
+  - label: "Performance benchmarks for {FEATURE_NAME}"
+    key: "pb_{SAFE_FEATURE_NAME}"
+    depends_on: "notifications_it_{SAFE_FEATURE_NAME}"
+    commands:
+      # Replace to actual test commands
+      - echo "[DEBUG], performance benchmarking for {FEATURE_NAME}"
+  - label: "Notifications: Performance benchmarks for {FEATURE_NAME}"
+    key: "notifications_pb_{SAFE_FEATURE_NAME}"
+    depends_on: "pb_{SAFE_FEATURE_NAME}"
+    agents:
+      queue: {QUEUE}
+    env:
+      EXECUTE_ENTITY: "{FEATURE_NAME}"
+      EXECUTE_STAGE: "Benchmark"
+    commands:
+      - |
+        bash .buildkite/scripts/check_results.sh \
+          "Performance benchmarks for {FEATURE_NAME}" pb_{SAFE_FEATURE_NAME}
+
+  - label: "Stress tests for {FEATURE_NAME}"
+    key: "st_{SAFE_FEATURE_NAME}"
+    depends_on: "notifications_pb_{SAFE_FEATURE_NAME}"
+    commands:
+      # Replace to actual test commands
+      - echo "[DEBUG], stress testing for {FEATURE_NAME}"
+  - label: "Notifications: Stress tests for {FEATURE_NAME}"
+    key: "notifications_st_{SAFE_FEATURE_NAME}"
+    depends_on: "st_{SAFE_FEATURE_NAME}"
+    agents:
+      queue: {QUEUE}
+    env:
+      EXECUTE_ENTITY: "{FEATURE_NAME}"
+      EXECUTE_STAGE: "StressTest"
+    commands:
+      - |
+        bash .buildkite/scripts/check_results.sh \
+          "Stress tests for {FEATURE_NAME}" st_{SAFE_FEATURE_NAME}