Skip to content

chore(upgrade smp): Update to SMP 0.15.1 #20712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
scottopell wants to merge 13 commits into from
Closed

chore(upgrade smp): Update to SMP 0.15.1 #20712

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
e411fbb
Adds experiment.yaml and other required changes for SMP 0.15
scottopell Jun 21, 2024
924ccc2
removes replicas, total samples, and warmup seconds
scottopell Jun 21, 2024
2348e37
Adds target to config.yaml
scottopell Jun 21, 2024
2232fb0
REVERTME: hard-code image tags to skip image build
scottopell Jun 21, 2024
dd3c7d5
REVERTME: allow action to run on PR
scottopell Jun 21, 2024
784b0bf
REVERTME: correctly set metadata
scottopell Jun 21, 2024
818ada4
REVERTME: hard-code PR number
scottopell Jun 21, 2024
9e2611c
REVERTME: Disable build step entirely and set valid (if incorrect) SHAs
scottopell Jun 21, 2024
db4a384
REVERTME: hard-code even harder
scottopell Jun 21, 2024
bb6830b
REVERTME: remove attempted load of build image artifacts for comparison
scottopell Jun 21, 2024
5c0e299
REVERTME: Specify full container name of vector image
scottopell Jun 21, 2024
a194252
REVERTME: Hack to force job submit to run on this branch
scottopell Jun 21, 2024
87803a8
Corrects vector path in experiment yaml
scottopell Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 23 additions & 68 deletions .github/workflows/regression.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
should-run:
runs-on: ubuntu-latest
timeout-minutes: 5
if: github.event_name != 'pull_request'
#if: github.event_name != 'pull_request'
outputs:
source_changed: ${{ steps.filter.outputs.SOURCE_CHANGED }}
comment_valid: ${{ steps.comment.outputs.isTeamMember }}
Expand Down Expand Up @@ -121,17 +121,7 @@ jobs:
comparison-sha: ${{ steps.pr-metadata-merge-queue.outputs.COMPARISON_SHA || steps.pr-metadata-comment.outputs.COMPARISON_SHA }}
comparison-tag: ${{ steps.pr-metadata-merge-queue.outputs.COMPARISON_TAG || steps.pr-metadata-comment.outputs.COMPARISON_TAG }}

# below are used in the experiment/analyze jobs
cpus: ${{ steps.system.outputs.CPUS }}
memory: ${{ steps.system.outputs.MEMORY }}
vector-cpus: ${{ steps.system.outputs.VECTOR_CPUS }}

replicas: ${{ steps.experimental-meta.outputs.REPLICAS }}
warmup-seconds: ${{ steps.experimental-meta.outputs.WARMUP_SECONDS }}
total-samples: ${{ steps.experimental-meta.outputs.TOTAL_SAMPLES }}
p-value: ${{ steps.experimental-meta.outputs.P_VALUE }}
smp-version: ${{ steps.experimental-meta.outputs.SMP_CRATE_VERSION }}
lading-version: ${{ steps.experimental-meta.outputs.LADING_VERSION }}
steps:
- uses: actions/checkout@v3
with:
Expand All @@ -141,25 +131,29 @@ jobs:
# But, we can retrieve this info from some commands.
- name: Get PR metadata (issue_comment)
id: pr-metadata-comment
if: github.event_name == 'issue_comment'
#if: github.event_name == 'issue_comment'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
export PR_NUMBER=${{ github.event.issue.number }}
export PR_NUMBER="20712"
echo "PR_NUMBER=${PR_NUMBER}" >> $GITHUB_OUTPUT

gh pr checkout ${{ github.event.issue.number }}
gh pr checkout 20712

export BASELINE_SHA=$(git merge-base master HEAD)
#export BASELINE_SHA=$(git merge-base master HEAD)
export BASELINE_SHA="0.38.0"
echo "BASELINE_SHA=${BASELINE_SHA}" >> $GITHUB_OUTPUT

export COMPARISON_SHA=$(git rev-parse HEAD)
#export COMPARISON_SHA=$(git rev-parse HEAD)
export COMPARISON_SHA="0.39.0"
echo "COMPARISON_SHA=${COMPARISON_SHA}" >> $GITHUB_OUTPUT

export BASELINE_TAG="${PR_NUMBER}-${COMPARISON_SHA}-${BASELINE_SHA}"
#export BASELINE_TAG="${PR_NUMBER}-${COMPARISON_SHA}-${BASELINE_SHA}"
export BASELINE_TAG="0.39.0-distroless-libc"
echo "BASELINE_TAG=${BASELINE_TAG}" >> $GITHUB_OUTPUT

export COMPARISON_TAG="${PR_NUMBER}-${COMPARISON_SHA}-${COMPARISON_SHA}"
#export COMPARISON_TAG="${PR_NUMBER}-${COMPARISON_SHA}-${COMPARISON_SHA}"
export COMPARISON_TAG="0.38.0-distroless-libc"
echo "COMPARISON_TAG=${COMPARISON_TAG}" >> $GITHUB_OUTPUT

echo "pr number is: ${PR_NUMBER}"
Expand Down Expand Up @@ -189,7 +183,9 @@ jobs:
# queue to 1, we can get the PR number by parsing the merge queue temp branch's ref.
- name: Get PR metadata (merge queue)
id: pr-metadata-merge-queue
if: github.event_name != 'issue_comment'
#if: github.event_name != 'issue_comment'
# Something known to be false
if: github.repository == 'octo-org/octo-repo-prod'

Check failure

Code scanning / check-spelling

Unrecognized Spelling

[octo](#security-tab) is not a recognized word. \(unrecognized-spelling\)

Check failure

Code scanning / check-spelling

Unrecognized Spelling

[octo](#security-tab) is not a recognized word. \(unrecognized-spelling\)
run: |
export PR_NUMBER=$(echo "${{ github.ref }}" | sed -n 's|^refs/heads/gh-readonly-queue/master/pr-\([0-9]*\)-.*$|\1|p')
echo "PR_NUMBER=${PR_NUMBER}" >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -233,40 +229,15 @@ jobs:
id: experimental-meta
run: |
export WARMUP_SECONDS="45"
export REPLICAS="10"
export TOTAL_SAMPLES="600"
export P_VALUE="0.1"
export SMP_CRATE_VERSION="0.11.0"
export LADING_VERSION="0.19.1"

echo "warmup seconds: ${WARMUP_SECONDS}"
echo "replicas: ${REPLICAS}"
echo "total samples: ${TOTAL_SAMPLES}"
echo "regression p-value: ${P_VALUE}"
export SMP_CRATE_VERSION="0.15.1"

echo "smp crate version: ${SMP_CRATE_VERSION}"
echo "lading version: ${LADING_VERSION}"

echo "WARMUP_SECONDS=${WARMUP_SECONDS}" >> $GITHUB_OUTPUT
echo "REPLICAS=${REPLICAS}" >> $GITHUB_OUTPUT
echo "TOTAL_SAMPLES=${TOTAL_SAMPLES}" >> $GITHUB_OUTPUT
echo "P_VALUE=${P_VALUE}" >> $GITHUB_OUTPUT
echo "SMP_CRATE_VERSION=${SMP_CRATE_VERSION}" >> $GITHUB_OUTPUT
echo "LADING_VERSION=${LADING_VERSION}" >> $GITHUB_OUTPUT

- name: Setup system details
id: system
run: |
export CPUS="7"
export MEMORY="30g"
export VECTOR_CPUS="4"

echo "cpus total: ${CPUS}"
echo "memory total: ${MEMORY}"
echo "vector cpus: ${VECTOR_CPUS}"

echo "CPUS=${CPUS}" >> $GITHUB_OUTPUT
echo "MEMORY=${MEMORY}" >> $GITHUB_OUTPUT
echo "VECTOR_CPUS=${VECTOR_CPUS}" >> $GITHUB_OUTPUT

- name: (PR comment) Get PR branch
if: ${{ github.event_name == 'issue_comment' }}
Expand Down Expand Up @@ -390,16 +361,11 @@ jobs:
needs:
- compute-metadata
- confirm-valid-credentials
- build-baseline
steps:
- name: 'Download baseline image'
uses: actions/download-artifact@v3
with:
name: baseline-image

- name: Load baseline image
run: |
docker load --input baseline-image.tar
docker pull timberio/vector:${{ needs.compute-metadata.outputs.baseline-tag }}

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4.0.2
Expand Down Expand Up @@ -429,7 +395,6 @@ jobs:
needs:
- compute-metadata
- confirm-valid-credentials
- build-comparison
steps:
- name: 'Download comparison image'
uses: actions/download-artifact@v3
Expand All @@ -438,7 +403,7 @@ jobs:

- name: Load comparison image
run: |
docker load --input comparison-image.tar
docker pull timberio/vector:${{ needs.compute-metadata.outputs.comparison-tag }}

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4.0.2
Expand Down Expand Up @@ -509,20 +474,11 @@ jobs:
chmod +x ${{ runner.temp }}/bin/smp

${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job submit \
--lading-version ${{ needs.compute-metadata.outputs.lading-version }} \
--total-samples ${{ needs.compute-metadata.outputs.total-samples }} \
--warmup-seconds ${{ needs.compute-metadata.outputs.warmup-seconds }} \
--replicas ${{ needs.compute-metadata.outputs.replicas }} \
--baseline-image ${{ steps.login-ecr.outputs.registry }}/${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }}-vector:${{ needs.compute-metadata.outputs.baseline-tag }} \
--comparison-image ${{ steps.login-ecr.outputs.registry }}/${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }}-vector:${{ needs.compute-metadata.outputs.comparison-tag }} \
--baseline-sha ${{ needs.compute-metadata.outputs.baseline-sha }} \
--comparison-sha ${{ needs.compute-metadata.outputs.comparison-sha }} \
--target-command "/usr/local/bin/vector" \
--target-config-dir ${{ github.workspace }}/regression/ \
--target-cpu-allotment "${{ needs.compute-metadata.outputs.cpus }}" \
--target-memory-allotment "${{ needs.compute-metadata.outputs.memory }}" \
--target-environment-variables "VECTOR_THREADS=${{ needs.compute-metadata.outputs.vector-cpus }},VECTOR_REQUIRE_HEALTHY=true" \
--target-name vector \
--submission-metadata ${{ runner.temp }}/submission-metadata

- uses: actions/upload-artifact@v3
Expand All @@ -541,7 +497,6 @@ jobs:
--wait \
--wait-delay-seconds 60 \
--wait-timeout-minutes 90 \
--use-consignor welch \
--submission-metadata ${{ runner.temp }}/submission-metadata

- name: Handle cancellation if necessary
Expand Down Expand Up @@ -632,7 +587,7 @@ jobs:
chmod +x ${{ runner.temp }}/bin/smp

${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job result \
--submission-metadata ${{ runner.temp }}/submission-metadata --use-consignor welch
--submission-metadata ${{ runner.temp }}/submission-metadata

- name: Check status, cancelled
if: ${{ cancelled() }}
Expand Down Expand Up @@ -725,7 +680,7 @@ jobs:

${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job sync \
--submission-metadata ${{ runner.temp }}/submission-metadata \
--output-path "${{ runner.temp }}/outputs" --use-consignor welch
--output-path "${{ runner.temp }}/outputs"

- name: Read regression report
id: read-analysis
Expand Down Expand Up @@ -796,8 +751,8 @@ jobs:
if: always()
needs:
- compute-metadata
- build-baseline
- build-comparison
#- build-baseline
#- build-comparison
- confirm-valid-credentials
- upload-baseline-image-to-ecr
- upload-comparison-image-to-ecr
Expand Down
Loading