Skip to content

chore(smp upgrade) - Upgrade SMP to latest release #20713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
scottopell merged 20 commits into from
Jun 24, 2024
Merged

chore(smp upgrade) - Upgrade SMP to latest release #20713

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
1f052d3
Update smp job submission and associated versions
scottopell Jun 21, 2024
9d55f17
Adds experiment.yamls and config.yaml
scottopell Jun 21, 2024
bc2cc28
Remove VECTOR_REQUIRE_HEALTHY to better accomodate startup races betw…
scottopell Jun 21, 2024
be7022e
format
GeorgeHahn Jun 21, 2024
67837c5
Removes unused dotfile-wannabe
scottopell Jun 24, 2024
d0b9109
Attempt to run off PR
scottopell Jun 24, 2024
e7fa40b
Eval should_run always
scottopell Jun 24, 2024
f9cf41b
Get PR number correctly for PRs
scottopell Jun 24, 2024
86b31b2
New merge-base cmd
scottopell Jun 24, 2024
f6caaab
New git ref for merge-base cmd
scottopell Jun 24, 2024
90419e9
Only checkout PR if its not a PR
scottopell Jun 24, 2024
50f6cba
Full ref to master
scottopell Jun 24, 2024
7baa079
Show me all valid git refs
scottopell Jun 24, 2024
300d363
Fetch full git repo
scottopell Jun 24, 2024
5d0e192
Skip merge-equeue metadata step for pull request
scottopell Jun 24, 2024
5203845
normalize to 0.0.0.0 instead of localhost
scottopell Jun 24, 2024
366a9ae
Normalize to 0.0.0.0
scottopell Jun 24, 2024
c2d6570
Revert changes to manually run this on PR
scottopell Jun 24, 2024
ccf585f
Removes workload checks
scottopell Jun 24, 2024
97350cf
Removes trailing empty whitespace lines
scottopell Jun 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 6 additions & 57 deletions .github/workflows/regression.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,19 +121,9 @@ jobs:
comparison-sha: ${{ steps.pr-metadata-merge-queue.outputs.COMPARISON_SHA || steps.pr-metadata-comment.outputs.COMPARISON_SHA }}
comparison-tag: ${{ steps.pr-metadata-merge-queue.outputs.COMPARISON_TAG || steps.pr-metadata-comment.outputs.COMPARISON_TAG }}

# below are used in the experiment/analyze jobs
cpus: ${{ steps.system.outputs.CPUS }}
memory: ${{ steps.system.outputs.MEMORY }}
vector-cpus: ${{ steps.system.outputs.VECTOR_CPUS }}

replicas: ${{ steps.experimental-meta.outputs.REPLICAS }}
warmup-seconds: ${{ steps.experimental-meta.outputs.WARMUP_SECONDS }}
total-samples: ${{ steps.experimental-meta.outputs.TOTAL_SAMPLES }}
p-value: ${{ steps.experimental-meta.outputs.P_VALUE }}
smp-version: ${{ steps.experimental-meta.outputs.SMP_CRATE_VERSION }}
lading-version: ${{ steps.experimental-meta.outputs.LADING_VERSION }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

@jszwedko jszwedko Jun 24, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually withheld this version bump in #18490 but I think it's ok to do for this workflow.

with:
fetch-depth: 1000

Expand All @@ -146,6 +136,7 @@ jobs:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
export PR_NUMBER=${{ github.event.issue.number }}

echo "PR_NUMBER=${PR_NUMBER}" >> $GITHUB_OUTPUT

gh pr checkout ${{ github.event.issue.number }}
Expand Down Expand Up @@ -229,44 +220,12 @@ jobs:
exit 1
fi

- name: Setup experimental metadata
- name: Set SMP version
id: experimental-meta
run: |
export WARMUP_SECONDS="45"
export REPLICAS="10"
export TOTAL_SAMPLES="600"
export P_VALUE="0.1"
export SMP_CRATE_VERSION="0.11.0"
export LADING_VERSION="0.19.1"

echo "warmup seconds: ${WARMUP_SECONDS}"
echo "replicas: ${REPLICAS}"
echo "total samples: ${TOTAL_SAMPLES}"
echo "regression p-value: ${P_VALUE}"
export SMP_CRATE_VERSION="0.15.1"
echo "smp crate version: ${SMP_CRATE_VERSION}"
echo "lading version: ${LADING_VERSION}"

echo "WARMUP_SECONDS=${WARMUP_SECONDS}" >> $GITHUB_OUTPUT
echo "REPLICAS=${REPLICAS}" >> $GITHUB_OUTPUT
echo "TOTAL_SAMPLES=${TOTAL_SAMPLES}" >> $GITHUB_OUTPUT
echo "P_VALUE=${P_VALUE}" >> $GITHUB_OUTPUT
echo "SMP_CRATE_VERSION=${SMP_CRATE_VERSION}" >> $GITHUB_OUTPUT
echo "LADING_VERSION=${LADING_VERSION}" >> $GITHUB_OUTPUT

- name: Setup system details
id: system
run: |
export CPUS="7"
export MEMORY="30g"
export VECTOR_CPUS="4"

echo "cpus total: ${CPUS}"
echo "memory total: ${MEMORY}"
echo "vector cpus: ${VECTOR_CPUS}"

echo "CPUS=${CPUS}" >> $GITHUB_OUTPUT
echo "MEMORY=${MEMORY}" >> $GITHUB_OUTPUT
echo "VECTOR_CPUS=${VECTOR_CPUS}" >> $GITHUB_OUTPUT

- name: (PR comment) Get PR branch
if: ${{ github.event_name == 'issue_comment' }}
Expand Down Expand Up @@ -509,20 +468,11 @@ jobs:
chmod +x ${{ runner.temp }}/bin/smp

${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job submit \
--lading-version ${{ needs.compute-metadata.outputs.lading-version }} \
--total-samples ${{ needs.compute-metadata.outputs.total-samples }} \
--warmup-seconds ${{ needs.compute-metadata.outputs.warmup-seconds }} \
--replicas ${{ needs.compute-metadata.outputs.replicas }} \
--baseline-image ${{ steps.login-ecr.outputs.registry }}/${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }}-vector:${{ needs.compute-metadata.outputs.baseline-tag }} \
--comparison-image ${{ steps.login-ecr.outputs.registry }}/${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }}-vector:${{ needs.compute-metadata.outputs.comparison-tag }} \
--baseline-sha ${{ needs.compute-metadata.outputs.baseline-sha }} \
--comparison-sha ${{ needs.compute-metadata.outputs.comparison-sha }} \
--target-command "/usr/local/bin/vector" \
--target-config-dir ${{ github.workspace }}/regression/ \
--target-cpu-allotment "${{ needs.compute-metadata.outputs.cpus }}" \
--target-memory-allotment "${{ needs.compute-metadata.outputs.memory }}" \
--target-environment-variables "VECTOR_THREADS=${{ needs.compute-metadata.outputs.vector-cpus }},VECTOR_REQUIRE_HEALTHY=true" \
--target-name vector \
--submission-metadata ${{ runner.temp }}/submission-metadata

- uses: actions/upload-artifact@v3
Expand All @@ -541,7 +491,6 @@ jobs:
--wait \
--wait-delay-seconds 60 \
--wait-timeout-minutes 90 \
--use-consignor welch \
--submission-metadata ${{ runner.temp }}/submission-metadata

- name: Handle cancellation if necessary
Expand Down Expand Up @@ -632,7 +581,7 @@ jobs:
chmod +x ${{ runner.temp }}/bin/smp

${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job result \
--submission-metadata ${{ runner.temp }}/submission-metadata --use-consignor welch
--submission-metadata ${{ runner.temp }}/submission-metadata

- name: Check status, cancelled
if: ${{ cancelled() }}
Expand Down Expand Up @@ -725,7 +674,7 @@ jobs:

${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job sync \
--submission-metadata ${{ runner.temp }}/submission-metadata \
--output-path "${{ runner.temp }}/outputs" --use-consignor welch
--output-path "${{ runner.temp }}/outputs"

- name: Read regression report
id: read-analysis
Expand Down
210 changes: 0 additions & 210 deletions .github/workflows/workload_checks.yml
View file
Open in desktop
Comment thread
scottopell marked this conversation as resolved.

This file was deleted.

13 changes: 3 additions & 10 deletions regression/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
#
# LADING
#
FROM ghcr.io/datadog/lading@sha256:15d003762f0015e99f2a9772fd8d7ac31165e4af2f645c404499ee48146202c8 as lading

#
# VECTOR BUILDER
#
Expand All @@ -21,12 +16,10 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
#
FROM docker.io/debian:bookworm-slim@sha256:01bd742e2c269abf94e2fefb47b08b5b61c9a880b993417d23a1d0bd9fa60dc4
RUN apt-get update && apt-get dist-upgrade -y && apt-get -y --no-install-recommends install zlib1g ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=lading /usr/bin/lading /usr/local/bin/lading
COPY --from=builder /vector/vector /usr/local/bin/vector
COPY --from=builder /vector/vector /usr/bin/vector
RUN mkdir --parents --mode=0777 /var/lib/vector

# Smoke test
RUN ["/usr/local/bin/lading", "--help"]
RUN ["/usr/local/bin/vector", "--version"]
RUN ["/usr/bin/vector", "--version"]

ENTRYPOINT ["/usr/local/bin/lading"]
ENTRYPOINT ["/usr/bin/vector"]
12 changes: 12 additions & 0 deletions regression/cases/datadog_agent_remap_blackhole/experiment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
optimization_goal: ingress_throughput

target:
name: vector
command: /usr/bin/vector
cpu_allotment: 7
memory_allotment: 30g

environment:
VECTOR_THREADS: 4


@jszwedko jszwedko Jun 24, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Nit: extra whitespace 😄

12 changes: 12 additions & 0 deletions regression/cases/datadog_agent_remap_blackhole_acks/experiment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
optimization_goal: ingress_throughput

target:
name: vector
command: /usr/bin/vector
cpu_allotment: 7
memory_allotment: 30g

environment:
VECTOR_THREADS: 4


12 changes: 12 additions & 0 deletions regression/cases/datadog_agent_remap_datadog_logs/experiment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
optimization_goal: ingress_throughput

target:
name: vector
command: /usr/bin/vector
cpu_allotment: 7
memory_allotment: 30g

environment:
VECTOR_THREADS: 4


Loading