Skip to content

Run eval with vertex instead of Gemini #228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
Oct 15, 2025
Merged

Run eval with vertex instead of Gemini #228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion test/prow/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ FROM registry.access.redhat.com/ubi9/python-311:latest

USER 0
RUN yum install -y jq patch vim
RUN pip install yq
RUN pip install yq google-cloud-aiplatform
RUN pip install git+https://github.com/lightspeed-core/lightspeed-evaluation.git#subdirectory=lsc_agent_eval
RUN mkdir -p /opt/app-root/src
WORKDIR /opt/app-root/src
Expand Down
6 changes: 4 additions & 2 deletions test/prow/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,12 @@ TEMP_DIR=$(mktemp -d)
cd $TEMP_DIR

echo "$OCM_TOKEN" > ocm_token.txt
echo "GEMINI_API_KEY=${GEMINI_API_KEY}" > .env

echo "GEMINI_API_KEY=dummy" > .env
echo "GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}" >> .env
Comment thread
omertuc marked this conversation as resolved.

cp $TEST_DIR/eval_data.yaml $TEMP_DIR/eval_data.yaml
sed -i "s/uniq-cluster-name/${UNIQUE_ID}/g" $TEMP_DIR/eval_data.yaml
sed -i "s|: ../scripts|: ${WORK_DIR}/test/scripts|g" $TEMP_DIR/eval_data.yaml

python $TEST_DIR/eval.py --agent_endpoint "${AGENT_URL}:${AGENT_PORT}" --agent_auth_token_file $TEMP_DIR/ocm_token.txt --eval_data_yaml $TEMP_DIR/eval_data.yaml
python $TEST_DIR/eval.py --agent_endpoint "${AGENT_URL}:${AGENT_PORT}" --agent_auth_token_file $TEMP_DIR/ocm_token.txt --eval_data_yaml $TEMP_DIR/eval_data.yaml --judge_provider="vertex"
Comment thread
omertuc marked this conversation as resolved.
16 changes: 15 additions & 1 deletion test/prow/template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ objects:
secretKeyRef:
key: ${GEMINI_API_SECRET_KEY_NAME}
name: ${GEMINI_API_SECRET_NAME}
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /opt/app-root/google-vertex-service-account.json
- name: AGENT_URL
value: ${AGENT_URL}
- name: AGENT_PORT
Expand All @@ -45,6 +47,14 @@ objects:
value: ${JOB_ID}
- name: OCM_BASE_URL
value: ${OCM_BASE_URL}
volumeMounts:
- name: google-vertex-service-account
mountPath: /opt/app-root/google-vertex-service-account.json
subPath: ${VERTEX_API_SECRET_KEY_NAME}
volumes:
- name: google-vertex-service-account
secret:
secretName: ${VERTEX_API_SECRET_NAME}

parameters:
- name: JOB_ID
Expand All @@ -66,5 +76,9 @@ parameters:
value: gemini
- name: GEMINI_API_SECRET_KEY_NAME
value: api_key
- name: VERTEX_API_SECRET_NAME
value: vertex-service-account
- name: VERTEX_API_SECRET_KEY_NAME
value: service_account
- name: OCM_BASE_URL
value: https://api.stage.openshift.com
value: https://api.stage.openshift.com