Skip to content

MGMT-21406 Switch from Gemini to Vertex AI #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
omertuc merged 1 commit into from
Aug 19, 2025
Merged

MGMT-21406 Switch from Gemini to Vertex AI #126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
build-images \
build-inspector build-assisted-mcp build-lightspeed-stack build-lightspeed-plus-llama-stack build-ui \
generate run resume stop rm logs query query-int query-stage query-interactive mcphost test-eval psql sqlite help
deploy-template ci-test deploy-template-local

all: help ## Show help information

Expand Down Expand Up @@ -32,10 +33,28 @@ build-ui: ## Build UI image
@echo "Building UI image..."
./scripts/build-images.sh ui

.PHONY:
deploy-template:
deploy-template: ## Used by the CI. Deploys the template on the temporary CI cluster
scripts/deploy_template.sh

ci-test: ## Used by the CI to test the assisted-chat services
./scripts/ci_test.sh

deploy-template-local: ## Used to test the CI flow locally. Deploys the template on whatever cluster `oc` is currently logged in to
@echo "Setting up local secrets directory..."
@mkdir -p /tmp/secrets/vertex
@if [ -z "$(VERTEX_SERVICE_ACCOUNT_PATH)" ]; then \
echo "Error: VERTEX_SERVICE_ACCOUNT_PATH environment variable must be set"; \
exit 1; \
fi
@if [ -z "$(ASSISTED_CHAT_IMG)" ]; then \
echo "Error: ASSISTED_CHAT_IMG environment variable must be set"; \
exit 1; \
fi
@cp "$(VERTEX_SERVICE_ACCOUNT_PATH)" /tmp/secrets/vertex/service_account
@echo "Deploying template locally..."
oc create namespace assisted-chat || true
NAMESPACE=assisted-chat SECRETS_BASE_PATH=/tmp/secrets ASSISTED_CHAT_IMG="$(ASSISTED_CHAT_IMG)" scripts/deploy_template.sh

generate: ## Generate configuration files
@echo "Generating configuration files..."
./scripts/generate.sh
Expand Down
55 changes: 55 additions & 0 deletions scripts/ci_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/bin/bash

set -o nounset
set -o errexit
set -o pipefail

SECRETS_BASE_PATH="${SECRETS_BASE_PATH:-/var/run/secrets}"

oc create secret generic -n "$NAMESPACE" assisted-chat-ssl-ci --from-file=client_id=/var/run/secrets/sso-ci/client_id \
--from-file=client_secret=/var/run/secrets/sso-ci/client_secret

Comment on lines +9 to +11
Copy link

@coderabbitai coderabbitai bot Aug 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Make secret creation idempotent and use SECRETS_BASE_PATH consistently

  • Re-running this script will fail because oc create secret is unconditional.
  • You introduced SECRETS_BASE_PATH but hardcoded /var/run/secrets/....

Use apply-style creation and the base path:

-oc create secret generic -n "$NAMESPACE" assisted-chat-ssl-ci --from-file=client_id=/var/run/secrets/sso-ci/client_id \
-    --from-file=client_secret=/var/run/secrets/sso-ci/client_secret
+oc create secret generic -n "$NAMESPACE" assisted-chat-ssl-ci \
+  --from-file=client_id="${SECRETS_BASE_PATH}/sso-ci/client_id" \
+  --from-file=client_secret="${SECRETS_BASE_PATH}/sso-ci/client_secret" \
+  2>/dev/null || oc -n "$NAMESPACE" create secret generic assisted-chat-ssl-ci --dry-run=client -o yaml \
+    --from-file=client_id="${SECRETS_BASE_PATH}/sso-ci/client_id" \
+    --from-file=client_secret="${SECRETS_BASE_PATH}/sso-ci/client_secret" | oc apply -f -
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
oc create secret generic -n "$NAMESPACE" assisted-chat-ssl-ci --from-file=client_id=/var/run/secrets/sso-ci/client_id \
--from-file=client_secret=/var/run/secrets/sso-ci/client_secret
oc create secret generic -n "$NAMESPACE" assisted-chat-ssl-ci \
--from-file=client_id="${SECRETS_BASE_PATH}/sso-ci/client_id" \
--from-file=client_secret="${SECRETS_BASE_PATH}/sso-ci/client_secret" \
2>/dev/null || oc -n "$NAMESPACE" create secret generic assisted-chat-ssl-ci --dry-run=client -o yaml \
--from-file=client_id="${SECRETS_BASE_PATH}/sso-ci/client_id" \
--from-file=client_secret="${SECRETS_BASE_PATH}/sso-ci/client_secret" | oc apply -f -
🤖 Prompt for AI Agents 
In scripts/ci_test.sh around lines 9 to 11, the script uses `oc create secret`
unconditionally and hardcodes `/var/run/secrets/...`, which fails on reruns and
ignores the new SECRETS_BASE_PATH; change the secret creation to an idempotent
apply-style pattern (generate the secret YAML via `oc create secret ...
--dry-run=client -o yaml` and pipe to `oc apply -f -`) and replace hardcoded
paths with "$SECRETS_BASE_PATH/client_id" and "$SECRETS_BASE_PATH/client_secret"
so the command is safe to re-run and respects the configured base path.

oc process -p IMAGE_NAME="$ASSISTED_CHAT_TEST" -p SSL_CLIENT_SECRET_NAME=assisted-chat-ssl-ci -f test/prow/template.yaml --local | oc apply -n "$NAMESPACE" -f -

sleep 5
oc get pods -n "$NAMESPACE"
POD_NAME=$(oc get pods | tr -s ' ' | cut -d ' ' -f1 | grep assisted-chat-eval-tes)
Copy link

@coderabbitai coderabbitai bot Aug 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Bug: querying pods without namespace and brittle parsing

oc get pods here omits -n "$NAMESPACE", potentially selecting from the default namespace. The parsing is also fragile and can return multiple pods.

Use -n, -o name, and pick a single match deterministically:

-POD_NAME=$(oc get pods | tr -s ' ' | cut -d ' ' -f1 | grep assisted-chat-eval-tes)
+POD_NAME=$(oc get pods -n "$NAMESPACE" -o name | grep -m1 '^pod/assisted-chat-eval-tes' | cut -d/ -f2)
+if [[ -z "${POD_NAME:-}" ]]; then
+  echo "Pod matching prefix 'assisted-chat-eval-tes' not found in namespace ${NAMESPACE}" >&2
+  oc get pods -n "$NAMESPACE"
+  exit 1
+fi
🤖 Prompt for AI Agents 
In scripts/ci_test.sh around line 16 the pod lookup uses a bare `oc get pods`
with fragile whitespace parsing and no namespace, which can return multiple
results; change the command to query the intended namespace with `-n
"$NAMESPACE"`, request machine-readable output with `-o name` so results are
like "pod/NAME", filter deterministically (e.g. use grep -m1 or head -n1 or a
precise selector) to pick a single pod, and avoid brittle tr/split/cut parsing
so the script reliably selects one pod from the specified namespace.


TIMEOUT=600
ELAPSED=0

while [ $ELAPSED -lt $TIMEOUT ]; do
# Check if the pod's status is "Running"
CURRENT_STATUS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.phase}')
CURRENT_RESTARTS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].restartCount}')
if [[ $CURRENT_RESTARTS -gt 0 ]]; then
echo "Pod ${POD_NAME} was restarted, so the tests should run at least once, exiting"
oc logs -n "$NAMESPACE" "$POD_NAME"
exit "$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}')"
fi
Comment on lines +21 to +29
Copy link

@coderabbitai coderabbitai bot Aug 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Guard against missing containerStatuses to avoid integer comparison errors

Immediately after pod creation, .status.containerStatuses[0] may be absent, causing [[ $CURRENT_RESTARTS -gt 0 ]] to error. Default to 0 and use arithmetic comparison.

-    CURRENT_RESTARTS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].restartCount}')
-    if [[ $CURRENT_RESTARTS -gt 0 ]]; then
+    CURRENT_RESTARTS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].restartCount}')
+    CURRENT_RESTARTS=${CURRENT_RESTARTS:-0}
+    if (( CURRENT_RESTARTS > 0 )); then
         echo "Pod ${POD_NAME} was restarted, so the tests should run at least once, exiting"
         oc logs -n "$NAMESPACE" "$POD_NAME"
         exit "$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}')"
     fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
while [ $ELAPSED -lt $TIMEOUT ]; do
# Check if the pod's status is "Running"
CURRENT_STATUS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.phase}')
CURRENT_RESTARTS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].restartCount}')
if [[ $CURRENT_RESTARTS -gt 0 ]]; then
echo "Pod ${POD_NAME} was restarted, so the tests should run at least once, exiting"
oc logs -n "$NAMESPACE" "$POD_NAME"
exit "$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}')"
fi
while [ $ELAPSED -lt $TIMEOUT ]; do
# Check if the pod's status is "Running"
CURRENT_STATUS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.phase}')
CURRENT_RESTARTS=$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].restartCount}')
CURRENT_RESTARTS=${CURRENT_RESTARTS:-0}
if (( CURRENT_RESTARTS > 0 )); then
echo "Pod ${POD_NAME} was restarted, so the tests should run at least once, exiting"
oc logs -n "$NAMESPACE" "$POD_NAME"
exit "$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}')"
fi
🤖 Prompt for AI Agents 
In scripts/ci_test.sh around lines 21 to 29, the code reads
.status.containerStatuses[0] which can be absent and causes integer comparison
errors; update CURRENT_RESTARTS to default to 0 when the jsonpath returns empty
(e.g. capture the oc output and set CURRENT_RESTARTS=${CURRENT_RESTARTS:-0}) and
replace the string comparison with an arithmetic comparison like if ((
CURRENT_RESTARTS > 0 )); then, and when exiting, guard the lastState exitCode
lookup similarly by defaulting to a safe exit code if the jsonpath is empty.

if [[ "$CURRENT_STATUS" == "Succeeded" ]]; then
echo "Pod ${POD_NAME} is successfully completed, exiting"
oc logs -n "$NAMESPACE" "$POD_NAME"
exit 0
fi
if [[ "$CURRENT_STATUS" == "Completed" ]]; then
echo "Pod ${POD_NAME} is successfully completed, exiting"
oc logs -n "$NAMESPACE" "$POD_NAME"
exit 0
fi

if [[ "$CURRENT_STATUS" == "Failed" ]]; then
echo "Pod ${POD_NAME} is Failed, exiting"
oc logs -n "$NAMESPACE" "$POD_NAME"
exit "$(oc get pod "$POD_NAME" -n "$NAMESPACE" -o=jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}')"
fi

echo "Waiting for pod $POD_NAME to be ready..."
sleep 1
ELAPSED=$((ELAPSED + 1))
done

oc logs -n "$NAMESPACE" "$POD_NAME"

echo "Timeout reached. Pod $POD_NAME did not become ready in time."
exit 1
84 changes: 66 additions & 18 deletions scripts/deploy_template.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,80 @@ set -o nounset
set -o errexit
set -o pipefail

SECRETS_BASE_PATH="${SECRETS_BASE_PATH:-/var/run/secrets}"

#All the secret are expected to be mounted under /var/run/secrets by the ci-operator

#$ASSISTED_CHAT_IMG is not in repo/image:tag format but rather in repo/<image name>@sha256:<digest>
#The template needs the tag, and it references the image by <image name>:<tag> so splitting the variable by ":" works for now

echo $ASSISTED_CHAT_IMG
IMAGE=$(echo $ASSISTED_CHAT_IMG | cut -d ":" -f1)
TAG=$(echo $ASSISTED_CHAT_IMG | cut -d ":" -f2)
echo "$ASSISTED_CHAT_IMG"
IMAGE=$(echo "$ASSISTED_CHAT_IMG" | cut -d ":" -f1)
TAG=$(echo "$ASSISTED_CHAT_IMG" | cut -d ":" -f2)
Comment on lines +15 to +16
Copy link

@coderabbitai coderabbitai bot Aug 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Make image parsing robust for registries with ports and digest references

Splitting on the first colon breaks when the registry includes a port (e.g., quay.io:443/…), and it’s unnecessary for digest references. Split on the last colon instead.

Apply this diff:

-IMAGE=$(echo "$ASSISTED_CHAT_IMG" | cut -d ":" -f1)
-TAG=$(echo "$ASSISTED_CHAT_IMG" | cut -d ":" -f2)
+IMAGE="${ASSISTED_CHAT_IMG%:*}"
+TAG="${ASSISTED_CHAT_IMG##*:}"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
IMAGE=$(echo "$ASSISTED_CHAT_IMG" | cut -d ":" -f1)
TAG=$(echo "$ASSISTED_CHAT_IMG" | cut -d ":" -f2)
IMAGE="${ASSISTED_CHAT_IMG%:*}"
TAG="${ASSISTED_CHAT_IMG##*:}"
🤖 Prompt for AI Agents 
In scripts/deploy_template.sh around lines 15-16, the current split on the first
colon breaks for registries with ports and digest references; update the parsing
to split on the last colon and handle digest (@) cases: if the image string
contains an '@' treat it as a digest and set IMAGE to the substring before '@'
and TAG empty; otherwise obtain TAG as the substring after the last colon and
IMAGE as the substring before that last colon (use bash parameter expansion or
equivalent to extract the last-colon split rather than cut -d ":" -f1/f2).


# What secrets have we got?
ls -laR "$SECRETS_BASE_PATH"

if ! oc get secret -n "$NAMESPACE" vertex-service-account &>/dev/null; then
echo "Creating vertex-service-account secret in namespace $NAMESPACE"
oc create secret generic -n "$NAMESPACE" vertex-service-account --from-file=service_account="$SECRETS_BASE_PATH/vertex/service_account"
fi

if ! oc get secret -n "$NAMESPACE" insights-ingress &>/dev/null; then
echo "Creating insights-ingress secret in namespace $NAMESPACE"
oc create secret generic -n "$NAMESPACE" insights-ingress --from-literal=auth_token="dummy-token"
fi

if ! oc get secret -n "$NAMESPACE" llama-stack-db &>/dev/null; then
echo "Creating llama-stack-db secret with local postgres credentials in namespace $NAMESPACE"
oc create secret generic -n "$NAMESPACE" llama-stack-db \
--from-literal=db.host=postgres-service \
--from-literal=db.port=5432 \
--from-literal=db.name=assistedchat \
--from-literal=db.user=assistedchat \
--from-literal=db.password=assistedchat123 \
--from-literal=db.ca_cert=""
fi

if ! oc get secret -n "$NAMESPACE" postgres-secret &>/dev/null; then
echo "Creating postgres-secret in namespace $NAMESPACE"

oc create secret generic -n "$NAMESPACE" postgres-secret \
--from-literal=POSTGRESQL_DATABASE=assistedchat \
--from-literal=POSTGRESQL_USER=assistedchat \
--from-literal=POSTGRESQL_PASSWORD=assistedchat123
fi

oc create secret generic -n $NAMESPACE gemini-api-key --from-file=api_key=/var/run/secrets/gemini/api_key
oc create secret generic -n $NAMESPACE llama-stack-db --from-file=db.ca_cert=/var/run/secrets/llama-stack-db/db.ca_cert \
--from-file=db.host=/var/run/secrets/llama-stack-db/db.host \
--from-file=db.name=/var/run/secrets/llama-stack-db/db.name \
--from-file=db.password=/var/run/secrets/llama-stack-db/db.password \
--from-file=db.port=/var/run/secrets/llama-stack-db/db.port \
--from-file=db.user=/var/run/secrets/llama-stack-db/db.user
if ! oc get deployment -n "$NAMESPACE" postgres &>/dev/null; then
echo "Creating postgres deployment in namespace $NAMESPACE"
oc create deployment -n "$NAMESPACE" postgres --image=quay.io/sclorg/postgresql-16-c9s:c9s
oc set env -n "$NAMESPACE" deployment/postgres --from=secret/postgres-secret
fi

patch template.yaml -i test/prow/template_patch.diff
echo "GEMINI_API_KEY=$(cat /var/run/secrets/gemini/api_key)" > .env
make generate
sed -i 's/user_id_claim: sub/user_id_claim: client_id/g' config/lightspeed-stack.yaml
sed -i 's/username_claim: preferred_username/username_claim: clientHost/g' config/lightspeed-stack.yaml
if ! oc get service -n "$NAMESPACE" postgres-service &>/dev/null; then
echo "Creating postgres service in namespace $NAMESPACE"
oc expose -n "$NAMESPACE" deployment/postgres --name=postgres-service --port=5432
fi

oc process -p IMAGE=$IMAGE -p IMAGE_TAG=$TAG -p GEMINI_API_SECRET_NAME=gemini-api-key -p ASSISTED_CHAT_DB_SECRET_NAME=llama-stack-db -f template.yaml --local | oc apply -n $NAMESPACE -f -
if ! oc get routes -n "$NAMESPACE" &>/dev/null; then
# Don't apply routes on clusters that don't have routes (e.g. minikube)
FILTER='select(.kind != "Route")'
else
FILTER='.'
fi

oc process \
-p IMAGE="$IMAGE" \
-p IMAGE_TAG="$TAG" \
-p VERTEX_API_SECRET_NAME=vertex-service-account \
-p ASSISTED_CHAT_DB_SECRET_NAME=llama-stack-db \
-p USER_ID_CLAIM=client_id \
-p USERNAME_CLAIM=clientHost \
-p LIGHTSSPEED_STACK_POSTGRES_SSL_MODE=disable \
-p LLAMA_STACK_POSTGRES_SSL_MODE=disable \
-f template.yaml --local |
jq '. as $root | $root.items = [$root.items[] | '"$FILTER"']' |
oc apply -n "$NAMESPACE" -f -

sleep 5
POD_NAME=$(oc get pods -n $NAMESPACE | tr -s ' ' | cut -d ' ' -f1| grep assisted-chat)
oc wait --for=condition=Ready pod/$POD_NAME --timeout=300s
oc wait --for=condition=Available deployment/assisted-chat -n "$NAMESPACE" --timeout=300s
21 changes: 21 additions & 0 deletions scripts/generate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,16 @@ if [[ ! -f "$PROJECT_ROOT/.env" ]]; then
read -sr GEMINI_API_KEY
echo "GEMINI_API_KEY=$GEMINI_API_KEY" >"$PROJECT_ROOT/.env"
chmod 600 "$PROJECT_ROOT/.env"

echo 'Gemini key successfully configured.'

# Create a dummy Vertex AI service account credentials file
if [[ ! -f "$PROJECT_ROOT/config/vertex-credentials.json" ]]; then
echo 'Also creating a dummy Vertex AI service account credentials file at config/vertex-credentials.json. If you want to use to be able to use both, modify config/vertex-credentials.json manually.'
echo '{}' >"$PROJECT_ROOT/config/vertex-credentials.json"
chmod 600 "$PROJECT_ROOT/config/vertex-credentials.json"
fi

elif [[ "$auth_type" == "v" || "$auth_type" == "V" ]]; then
echo 'Please enter the path to your Vertex AI service account credentials file:'
read -r VERTEX_AI_SERVICE_ACCOUNT_CREDENTIALS_PATH
Expand All @@ -27,6 +37,15 @@ if [[ ! -f "$PROJECT_ROOT/.env" ]]; then
exit 1
fi

if [[ -f "$PROJECT_ROOT/config/vertex-credentials.json" ]]; then
echo "File $PROJECT_ROOT/config/vertex-credentials.json already exists. Do you want to overwrite it? (y/n)"
read -r overwrite
if [[ "$overwrite" != "y" && "$overwrite" != "Y" ]]; then
echo "Exiting without copying."
exit 1
fi
fi

echo "$VERTEX_AI_SERVICE_ACCOUNT_CREDENTIALS_PATH will be copied to $PROJECT_ROOT/config/vertex-credentials.json, do you want to continue? (y/n)"
read -r should_copy
if [[ "$should_copy" != "y" && "$should_copy" != "Y" ]]; then
Expand All @@ -43,6 +62,8 @@ if [[ ! -f "$PROJECT_ROOT/.env" ]]; then
echo GEMINI_API_KEY="dummy" >"$PROJECT_ROOT/.env"
chmod 600 "$PROJECT_ROOT/.env"

echo "Vertex credentials successfully configured."

echo "Your Gemini API key will be set to a dummy value, as it is not needed for Vertex AI service account authentication, if you want to be able to use both, modify .env manually."
else
echo "Invalid choice. Exiting."
Expand Down
1 change: 1 addition & 0 deletions template-params.dev.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ LLAMA_CLIENT_CONFIG_PATH=llama_stack_client_config.yaml
LIGHTSPEED_TRANSCRIPTS_ENABLED=false
LIGHTSPEED_FEEDBACK_ENABLED=false
DISABLE_QUERY_SYSTEM_PROMPT=false
ASSISTED_CHAT_DEFAULT_MODEL=gemini/gemini-2.0-flash
Loading