Skip to content

LCORE-378: Lightspeed core needs to fully support Red Hat AI Inference server LLM provider #652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 10, 2025
Merged

LCORE-378: Lightspeed core needs to fully support Red Hat AI Inference server LLM provider #652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 175 additions & 0 deletions .github/workflows/e2e_tests_rhaiis.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# .github/workflows/e2e_tests_rhaiis.yaml
name: RHAIIS E2E Tests

on:
schedule:
- cron: "0 0 * * *" # Runs once a day at midnight UTC
workflow_dispatch:


jobs:
e2e_tests:
runs-on: ubuntu-latest
strategy:
matrix:
environment: [ "rhaiis" ]
env:
RHAIIS_URL: ${{ secrets.RHAIIS_URL }}
RHAIIS_API_KEY: ${{ secrets.RHAIIS_API_KEY }}

steps:
- uses: actions/checkout@v4
with:
# On PR_TARGET → the fork (or same repo) that opened the PR.
# On push → falls back to the current repository.
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}

# On PR_TARGET → the PR head *commit* (reproducible).
# On push → the pushed commit that triggered the workflow.
ref: ${{ github.event.pull_request.head.ref || github.sha }}

# Don’t keep credentials when running untrusted PR code under PR_TARGET.
persist-credentials: ${{ github.event_name != 'pull_request_target' }}

- name: Verify actual git checkout result
run: |
echo "=== Git Status After Checkout ==="
echo "Remote URLs:"
git remote -v
echo ""
echo "Current branch: $(git branch --show-current 2>/dev/null || echo 'detached HEAD')"
echo "Current commit: $(git rev-parse HEAD)"
echo "Current commit message: $(git log -1 --oneline)"
echo ""
echo "=== Recent commits (should show setup-metrics commits) ==="
git log --oneline -5

- uses: 1arp/[email protected]
with:
path: '.'
isAbsolutePath: false
file: 'lightspeed-stack.yaml'
content: |
name: Lightspeed Core Service (LCS)
service:
host: 0.0.0.0
port: 8080
auth_enabled: false
workers: 1
color_log: true
access_log: true
llama_stack:
# Uses a remote llama-stack service
# The instance would have already been started with a llama-stack-run.yaml file
use_as_library_client: false
# Alternative for "as library use"
# use_as_library_client: true
# library_client_config_path: <path-to-llama-stack-run.yaml-file>
url: http://llama-stack:8321
api_key: xyzzy
user_data_collection:
feedback_enabled: true
feedback_storage: "/tmp/data/feedback"
transcripts_enabled: true
transcripts_storage: "/tmp/data/transcripts"

authentication:
module: "noop"

- name: Select and configure run.yaml
env:
CONFIG_ENVIRONMENT: ${{ matrix.environment || 'rhaiis' }}
run: |
CONFIGS_DIR="tests/e2e/configs"
ENVIRONMENT="$CONFIG_ENVIRONMENT"

echo "Looking for configurations in $CONFIGS_DIR/"

# List available configurations
if [ -d "$CONFIGS_DIR" ]; then
echo "Available configurations:"
ls -la "$CONFIGS_DIR"/*.yaml 2>/dev/null || echo "No YAML files found in $CONFIGS_DIR/"
else
echo "Configs directory '$CONFIGS_DIR' not found!"
exit 1
fi

# Determine which config file to use
CONFIG_FILE="$CONFIGS_DIR/run-$ENVIRONMENT.yaml"

echo "Looking for: $CONFIG_FILE"

if [ -f "$CONFIG_FILE" ]; then
echo "Found config for environment: $ENVIRONMENT"
cp "$CONFIG_FILE" run.yaml
else
echo "Configuration file not found: $CONFIG_FILE"
echo "Available files in $CONFIGS_DIR:"
ls -la "$CONFIGS_DIR/"
exit 1
fi

# Update paths for container environment (relative -> absolute)
sed -i 's|db_path: \.llama/distributions|db_path: /app-root/.llama/distributions|g' run.yaml
sed -i 's|db_path: tmp/|db_path: /app-root/.llama/distributions/|g' run.yaml

echo "Successfully configured for environment: $ENVIRONMENT"
echo "Using configuration: $(basename "$CONFIG_FILE")"

- name: Test RHAIIS connectivity
env:
RHAIIS_URL: ${{ secrets.RHAIIS_URL }}
RHAIIS_API_KEY: ${{ secrets.RHAIIS_API_KEY }}
run: |
curl ${RHAIIS_URL}:8000/v1/models -H "Authorization: Bearer ${RHAIIS_API_KEY}"

Comment on lines +119 to +125
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Make the connectivity probe fail fast on HTTP errors.

Right now curl ${RHAIIS_URL}:8000/v1/models will exit 0 even if the endpoint returns a 401/500, so the workflow keeps going with a broken configuration. Add -f (or check the status code) and quote the URL to ensure the step actually fails on bad responses.

-          curl ${RHAIIS_URL}:8000/v1/models   -H "Authorization: Bearer ${RHAIIS_API_KEY}"  
+          curl -f "${RHAIIS_URL}:8000/v1/models" \
+            -H "Authorization: Bearer ${RHAIIS_API_KEY}"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Test RHAIIS connectivity
env:
RHAIIS_URL: ${{ secrets.RHAIIS_URL }}
RHAIIS_API_KEY: ${{ secrets.RHAIIS_API_KEY }}
run: |
curl ${RHAIIS_URL}:8000/v1/models -H "Authorization: Bearer ${RHAIIS_API_KEY}"
- name: Test RHAIIS connectivity
env:
RHAIIS_URL: ${{ secrets.RHAIIS_URL }}
RHAIIS_API_KEY: ${{ secrets.RHAIIS_API_KEY }}
run: |
curl -f "${RHAIIS_URL}:8000/v1/models" \
-H "Authorization: Bearer ${RHAIIS_API_KEY}"
🤖 Prompt for AI Agents 
.github/workflows/e2e_tests_rhaiis.yaml around lines 117 to 123: the curl
connectivity probe currently won’t fail on HTTP errors and can return success on
4xx/5xx; update the run command to use curl -f and quote the URL (e.g. curl -f
"${RHAIIS_URL}:8000/v1/models" -H "Authorization: Bearer ${RHAIIS_API_KEY}") so
the step fails on non-2xx responses; alternatively check and exit non-zero based
on the HTTP status code if you prefer explicit handling.

- name: Run service manually
env:
RHAIIS_URL: ${{ secrets.RHAIIS_URL }}
RHAIIS_API_KEY: ${{ secrets.RHAIIS_API_KEY }}
run: |
docker compose version
docker compose up -d

# Check for errors and show logs if any services failed
if docker compose ps | grep -E 'Exit|exited|stopped'; then
echo "Some services failed to start - showing logs:"
docker compose logs
exit 1
else
echo "All services started successfully"
fi

- name: Wait for services
run: |
echo "Waiting for services to be healthy..."
sleep 20 # adjust depending on boot time

- name: Quick connectivity test
run: |
echo "Testing basic connectivity before full test suite..."
curl -f http://localhost:8080/v1/models || {
echo "❌ Basic connectivity failed - showing logs before running full tests"
docker compose logs --tail=30
exit 1
}

- name: Run e2e tests
run: |
echo "Installing test dependencies..."
pip install uv
uv sync
Comment on lines +160 to +161
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fix the uv installation step.

uv isn’t distributed via PyPI, so pip install uv will fail and the workflow never reaches the tests. Please switch to the official installer (or whatever method you use elsewhere) and invoke the resulting binary explicitly.

-          pip install uv
-          uv sync
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          ~/.local/bin/uv sync
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
pip install uv
uv sync
curl -LsSf https://astral.sh/uv/install.sh | sh
~/.local/bin/uv sync
🤖 Prompt for AI Agents 
.github/workflows/e2e_tests_rhaiis.yaml lines 158-159: the workflow attempts to
install `uv` via `pip install uv` which fails because `uv` is not on PyPI;
replace that step with the official installer used elsewhere in this repo (e.g.,
download or run the project's provided install script/release binary), ensure
the installer places the `uv` executable in PATH (or reference the downloaded
binary path), and then invoke the explicit `uv` binary (e.g., `uv sync` using
the installed binary path) so the subsequent test steps can run.


echo "Running comprehensive e2e test suite..."
make test-e2e

- name: Show logs on failure
if: failure()
run: |
echo "=== Test failure logs ==="
echo "=== llama-stack logs ==="
docker compose logs llama-stack

echo ""
echo "=== lightspeed-stack logs ==="
docker compose logs lightspeed-stack
5 changes: 5 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ The service includes comprehensive user data collection capabilities for various
* [Configuration](#configuration)
* [LLM Compatibility](#llm-compatibility)
* [Set LLM provider and model](#set-llm-provider-and-model)
* [Supported providers](#supported-providers)
* [Integration with Llama Stack](#integration-with-llama-stack)
* [Llama Stack as separate server](#llama-stack-as-separate-server)
* [MCP Server and Tool Configuration](#mcp-server-and-tool-configuration)
Expand Down Expand Up @@ -123,6 +124,7 @@ Lightspeed Core Stack (LCS) supports the large language models from the provider
| -------- | ---------------------------------------------- | ------------ | -------------- | -------------------------------------------------------------------------- |
| OpenAI | gpt-5, gpt-4o, gpt4-turbo, gpt-4.1, o1, o3, o4 | Yes | remote::openai | [1](examples/openai-faiss-run.yaml) [2](examples/openai-pgvector-run.yaml) |
| OpenAI | gpt-3.5-turbo, gpt-4 | No | remote::openai | |
| RHAIIS (vLLM)| meta-llama/Llama-3.1-8B-Instruct | Yes | remote::vllm | [1](tests/e2e/configs/run-rhaiis.yaml) |

The "provider_type" is used in the llama stack configuration file when refering to the provider.

Expand Down Expand Up @@ -156,6 +158,9 @@ models:
provider_model_id: gpt-4-turbo
```

## Supported providers

For a comprehensive list of supported providers, take a look [here](docs/providers.md).

## Integration with Llama Stack

Expand Down
2 changes: 2 additions & 0 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ services:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
- RHAIIS_URL=${RHAIIS_URL}
- RHAIIS_API_KEY=${RHAIIS_API_KEY}
networks:
- lightspeednet
healthcheck:
Expand Down
8 changes: 7 additions & 1 deletion docs/providers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,15 @@ The tables below summarize each provider category, containing the following atri
| tgi | remote | `huggingface_hub`, `aiohttp` | ❌ |
| together | remote | `together` | ❌ |
| vertexai | remote | `litellm`, `google-cloud-aiplatform` | ❌ |
| vllm | remote | `openai` | ❌ |
| watsonx | remote | `ibm_watsonx_ai` | ❌ |

Red Hat providers:

| Name | Version Tested | Type | Pip Dependencies | Supported in LCS |
|---|---|---|---|:---:|
| RHAIIS (vllm) | 3.2.3 (on RHEL 9.20250429.0.4) | remote | `openai` | ✅ |


---

## Agent Providers
Expand Down
137 changes: 137 additions & 0 deletions tests/e2e/configs/run-rhaiis.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
version: '2'
image_name: rhaiis-configuration

apis:
- agents
- datasetio
- eval
- files
- inference
- post_training
- safety
- scoring
- telemetry
- tool_runtime
- vector_io
benchmarks: []
container_image: null
datasets: []
external_providers_dir: null
inference_store:
db_path: .llama/distributions/ollama/inference_store.db
type: sqlite
logging: null
metadata_store:
db_path: .llama/distributions/ollama/registry.db
namespace: null
type: sqlite
providers:
files:
- config:
storage_dir: /tmp/llama-stack-files
metadata_store:
type: sqlite
db_path: .llama/distributions/ollama/files_metadata.db
provider_id: localfs
provider_type: inline::localfs
agents:
- config:
persistence_store:
db_path: .llama/distributions/ollama/agents_store.db
namespace: null
type: sqlite
responses_store:
db_path: .llama/distributions/ollama/responses_store.db
type: sqlite
provider_id: meta-reference
provider_type: inline::meta-reference
datasetio:
- config:
kvstore:
db_path: .llama/distributions/ollama/huggingface_datasetio.db
namespace: null
type: sqlite
provider_id: huggingface
provider_type: remote::huggingface
- config:
kvstore:
db_path: .llama/distributions/ollama/localfs_datasetio.db
namespace: null
type: sqlite
provider_id: localfs
provider_type: inline::localfs
eval:
- config:
kvstore:
db_path: .llama/distributions/ollama/meta_reference_eval.db
namespace: null
type: sqlite
provider_id: meta-reference
provider_type: inline::meta-reference
inference:
- provider_id: sentence-transformers # Can be any embedding provider
provider_type: inline::sentence-transformers
config: {}
- provider_id: vllm
provider_type: remote::vllm
config:
url: http://${env.RHAIIS_URL}:8000/v1/
api_token: ${env.RHAIIS_API_KEY}
tls_verify: false
max_tokens: 2048
post_training:
Comment on lines +71 to +82
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Verify vLLM provider config keys and URL scheme

  • Key name: Is api_token the correct field for llama‑stack remote::vllm? Many providers expect api_key. Please confirm and align.
  • URL: You embed http://${env.RHAIIS_URL}:8000/v1/. If RHAIIS_URL ever includes a scheme, this will double‑prefix. Consider a single env like RHAIIS_BASE_URL that includes scheme and port.
  • tls_verify: With http scheme, TLS verification is irrelevant. Either switch to https and set tls_verify appropriately, or omit.

Suggested safer structure:

-        url: http://${env.RHAIIS_URL}:8000/v1/
-        api_token: ${env.RHAIIS_API_KEY}
-        tls_verify: false
+        base_url: ${env.RHAIIS_BASE_URL}   # e.g., https://host:8000/v1/
+        api_key: ${env.RHAIIS_API_KEY}
+        tls_verify: true                   # if using https with valid certs

(Based on learnings)

🏁 Script executed:

#!/bin/bash
set -e
echo "---- remote::vllm references ----"
rg -n "remote::vllm"
echo "---- Provider registration or class definitions mentioning VLLM ----"
rg -nP "class\s+.*VLLM" || true
echo "---- Config key occurrences: api_token ----"
rg -n "api_token"
echo "---- Config key occurrences: api_key ----"
rg -n "api_key"
echo "---- tls_verify occurrences ----"
rg -n "tls_verify"
echo "---- base_url occurrences ----"
rg -n "base_url"

Length of output: 8563

🏁 Script executed:

#!/bin/bash
set -e
echo "---- LlamaStackConfiguration class ----"
rg -nP "class\s+LlamaStackConfiguration" -C5 src/models/config.py
echo "---- Fields around url, api_key, tls_verify ----"
sed -n '170,260p' src/models/config.py
echo "---- LlamaStackClient definition ----"
rg -nP "class\s+LlamaStackClient" -C5 src/client.py
echo "---- Client init parameters ----"
rg -n "base_url" -C3 src/client.py
rg -n "tls" -C3 src/client.py

Length of output: 3880

🏁 Script executed:

#!/bin/bash
set -e
echo "---- src/client.py (first 200 lines) ----"
sed -n '1,200p' src/client.py

Length of output: 2102

Align vLLM provider config with code schema

  • Use api_key (not api_token) – LlamaStackConfiguration defines api_key.
  • Set url: ${env.RHAIIS_BASE_URL} (e.g. https://host:8000/v1/) – maps to url in config.
  • Remove tls_verify – AsyncLlamaStackClient doesn’t support it; ensure HTTPS is used.
🤖 Prompt for AI Agents 
In tests/e2e/configs/run-rhaiis.yaml around lines 71 to 82, the vLLM provider
config uses the wrong key names and unsupported fields: replace api_token with
api_key to match LlamaStackConfiguration, change the URL variable from
${env.RHAIIS_URL} to ${env.RHAIIS_BASE_URL} ensuring it contains the full scheme
and path (e.g. https://host:8000/v1/), and remove the tls_verify field
(AsyncLlamaStackClient does not support it); keep other settings like max_tokens
as-is and ensure the URL uses HTTPS.

- config:
checkpoint_format: huggingface
device: cpu
distributed_backend: null
dpo_output_dir: "."
provider_id: huggingface
provider_type: inline::huggingface-gpu
safety:
- config:
excluded_categories: []
provider_id: llama-guard
provider_type: inline::llama-guard
scoring:
- config: {}
provider_id: basic
provider_type: inline::basic
- config: {}
provider_id: llm-as-judge
provider_type: inline::llm-as-judge
- config:
openai_api_key: '********'
provider_id: braintrust
provider_type: inline::braintrust
Comment on lines +95 to +105
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Braintrust scoring configured with placeholder secret

This likely fails provider initialization during startup. Either supply a real secret via env substitution or remove the provider from tests.

-  - config:
-      openai_api_key: '********'
-    provider_id: braintrust
-    provider_type: inline::braintrust
+  # Braintrust disabled in E2E until a key is provided
+  # - config:
+  #     openai_api_key: ${env.BRAINTRUST_OPENAI_API_KEY}
+  #   provider_id: braintrust
+  #   provider_type: inline::braintrust
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
scoring:
- config: {}
provider_id: basic
provider_type: inline::basic
- config: {}
provider_id: llm-as-judge
provider_type: inline::llm-as-judge
- config:
openai_api_key: '********'
provider_id: braintrust
provider_type: inline::braintrust
scoring:
- config: {}
provider_id: basic
provider_type: inline::basic
- config: {}
provider_id: llm-as-judge
provider_type: inline::llm-as-judge
# Braintrust disabled in E2E until a key is provided
# - config:
# openai_api_key: ${env.BRAINTRUST_OPENAI_API_KEY}
# provider_id: braintrust
# provider_type: inline::braintrust
🤖 Prompt for AI Agents 
In tests/e2e/configs/run-rhaiis.yaml around lines 95 to 105 the Braintrust
scoring provider is configured with a placeholder API key ('********') which
will cause provider initialization to fail; either replace the literal
placeholder with an environment-variable substitution (e.g. read from an env var
like ${BRAINTRUST_API_KEY} or the repo's config interpolation mechanism) so a
real secret is injected at runtime, or remove the entire Braintrust provider
block from the scoring list in this YAML so tests don't attempt to initialize
it.

telemetry:
- config:
service_name: 'lightspeed-stack-telemetry'
sinks: sqlite
sqlite_db_path: .llama/distributions/ollama/trace_store.db
provider_id: meta-reference
provider_type: inline::meta-reference
tool_runtime:
- provider_id: model-context-protocol
provider_type: remote::model-context-protocol
config: {}
scoring_fns: []
server:
auth: null
host: null
port: 8321
quota: null
tls_cafile: null
tls_certfile: null
tls_keyfile: null
shields: []
models:
- metadata:
embedding_dimension: 768 # Depends on chosen model
model_id: sentence-transformers/all-mpnet-base-v2 # Example embedding model
provider_id: sentence-transformers
provider_model_id: sentence-transformers/all-mpnet-base-v2 # Location of embedding model
model_type: embedding
- model_id: meta-llama/Llama-3.1-8B-Instruct
provider_id: vllm
model_type: llm
provider_model_id: meta-llama/Llama-3.1-8B-Instruct
Loading
Loading