CI: Add AWS EFA testing infrastructure by Alexey-Rivkin · Pull Request #212 · ai-dynamo/nixl

Alexey-Rivkin · 2025-04-28T16:20:55Z

What?

Add automated testing infrastructure for NIXL on AWS EFA environment via a new contrib/aws-efa/ module and GitHub Actions workflow.

Why?

Ensure NIXL compatibility with AWS EFA environment by:

Automate validation for C++ and Python implementations
Verify AWS-specific networking features
Catche EFA-specific regressions
Support both CI and manual testing

How?

Implement:

Bash testing script for job management
AWS Batch configuration files
New GitHub Actions workflow for PRs
Testing on NVIDIA PyTorch containers

tstamler · 2025-04-29T14:09:09Z

Hey, this is pretty cool! Just a couple comments:

-We have nixlbench in the main repo now, it would be nice to see it integrated in some configuration (doesn't need to be right now)
-Why are you deleting the pre-commit yaml? Current CI is obviously not going to be happy about that, but maybe you have a plan

Alexey-Rivkin · 2025-04-29T17:25:15Z

Hey, this is pretty cool! Just a couple comments:

-We have nixlbench in the main repo now, it would be nice to see it integrated in some configuration (doesn't need to be right now) -Why are you deleting the pre-commit yaml? Current CI is obviously not going to be happy about that, but maybe you have a plan

Hi, thanks for the feedback!
This is still a work in progress - please don't judge (yet).
.pre-commit-config.yaml conflicted with my local one, but I've since realized the mistake and fetched the original.
As for nixlbench, let's have it in a separate PR since this one is already pretty big.

pvijayakrish

PR looks good. Could you please share how the changes are tested and how the results from AWS batch will be reported back?

Introduce new testing capability for AWS Elastic Fabric Adapter environments through contrib/aws-efa module: - Shell script for running tests in AWS Batch environments - Configuration templates for AWS resources - Job definition for container specification - Documentation for usage in both CI and manual testing - Support for both C++ and Python testing"

Create new CI pipeline for AWS EFA validation: - Triggers on main branch pushes and pull requests - Runs both C++ and Python test variants - Uses AWS CLI container for executing tests - Includes proper credentials handling and cleanup - Outputs test results for CI verification"

Exclude AWS job definition from copyright header requirements since AWS Batch job definitions don't support custom fields like "copyright". Also AWS job definition does not contain proprietary code by any means.

Alexey-Rivkin · 2025-05-08T18:12:49Z

PR looks good. Could you please share how the changes are tested and how the results from AWS batch will be reported back?

Results propagation from AWS

When the pod exits, we poll AWS for exit status. If the final state is FAILED, the script exits non-zero, so Github Actions marks the flow as failed.
On success we exit 0, which lets the workflow turn green.
https://github.com/ai-dynamo/nixl/blob/arivkin/aws_tests/contrib/aws-efa/aws_test.sh#L120-L125

Testing done

The changes are validated through the AWS EFA workflow, which provides end-to-end testing by executing NIXL tests on AWS infrastructure and reporting the results back to GitHub. Negative tests were also conducted to ensure robustness.

pull-request-size Bot added the size/L label Apr 28, 2025

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:21 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:26 Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 12c9495 to 8bc4b14 Compare April 28, 2025 16:28

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:28 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:31 Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 8bc4b14 to 7eb100a Compare April 28, 2025 16:33

Alexey-Rivkin had a problem deploying to SWX_AWS April 28, 2025 16:33 — with GitHub Actions Failure

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:33 Inactive

Alexey-Rivkin had a problem deploying to SWX_AWS April 28, 2025 16:33 — with GitHub Actions Failure

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:36 Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 7eb100a to 97f1d0f Compare April 28, 2025 16:37

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:37 Inactive

Alexey-Rivkin temporarily deployed to SWX_AWS April 28, 2025 16:37 — with GitHub Actions Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 97f1d0f to c5fd857 Compare April 28, 2025 16:41

Alexey-Rivkin temporarily deployed to SWX_AWS April 28, 2025 16:41 — with GitHub Actions Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:42 Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from c5fd857 to 549d869 Compare April 28, 2025 16:51

Alexey-Rivkin temporarily deployed to SWX_AWS April 28, 2025 16:51 — with GitHub Actions Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:51 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 28, 2025 16:52 Inactive

yosefe reviewed Apr 28, 2025

View reviewed changes

Comment thread contrib/aws-efa/README.md

Comment thread contrib/aws-efa/aws_test.sh Outdated

Alexey-Rivkin marked this pull request as ready for review April 28, 2025 17:37

Alexey-Rivkin requested a review from a team April 28, 2025 17:37

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 549d869 to ef4d7b2 Compare April 29, 2025 07:25

copy-pr-bot Bot temporarily deployed to GITLAB April 29, 2025 07:25 Inactive

Alexey-Rivkin temporarily deployed to SWX_AWS April 29, 2025 07:25 — with GitHub Actions Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 054e07e to 4edc040 Compare April 29, 2025 12:50

copy-pr-bot Bot temporarily deployed to GITLAB April 29, 2025 12:50 Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 4edc040 to 9dd8ab8 Compare April 29, 2025 13:09

Alexey-Rivkin had a problem deploying to SWX_AWS April 29, 2025 13:09 — with GitHub Actions Failure

copy-pr-bot Bot temporarily deployed to GITLAB April 29, 2025 13:09 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB April 29, 2025 13:10 Inactive

Alexey-Rivkin force-pushed the arivkin/aws_tests branch from 9dd8ab8 to e229428 Compare April 29, 2025 13:13

dpressle reviewed May 4, 2025

View reviewed changes

Comment thread contrib/aws-efa/aws_test.sh Outdated

Comment thread contrib/aws-efa/README.md

dpressle requested changes May 5, 2025

View reviewed changes

Comment thread .github/workflows/aws_efa_validation.yml Outdated

dpressle approved these changes May 5, 2025

View reviewed changes

yosefe approved these changes May 5, 2025

View reviewed changes

dpressle requested changes May 8, 2025

View reviewed changes

Comment thread .github/workflows/aws_efa_validation.yml Outdated

Comment thread .github/workflows/aws_efa_validation.yml Outdated

dpressle approved these changes May 8, 2025

View reviewed changes

pvijayakrish reviewed May 8, 2025

View reviewed changes

Comment thread .github/workflows/aws_efa_validation.yml

pvijayakrish reviewed May 8, 2025

View reviewed changes

Comment thread .github/workflows/aws_efa_validation.yml

pvijayakrish reviewed May 8, 2025

View reviewed changes

Alexey-Rivkin added 5 commits May 8, 2025 20:18

CI: Add AWS job definition to ignored in copyright check

d9c7f74

Exclude AWS job definition from copyright header requirements since AWS Batch job definitions don't support custom fields like "copyright". Also AWS job definition does not contain proprietary code by any means.

CI: Implement remarks and fixes

390f9d7

CI: Change trigger events and naming

0c4e1c9

pvijayakrish approved these changes May 9, 2025

View reviewed changes

Merge branch 'main' into arivkin/aws_tests

5c0bf0f

NirWolfer mentioned this pull request Oct 11, 2026

CI: pin AWS EFA pod checkout to GITHUB_SHA #2384

Merged

svc-nixl mentioned this pull request Oct 12, 2026

ci: authenticate meson's wrap-git clones in the test pipelines #2393

Merged

Navigation Menu

Repository navigation

CI: Add AWS EFA testing infrastructure - #212

CI: Add AWS EFA testing infrastructure #212
yosefe merged 6 commits into
mainfrom
arivkin/aws_tests

Alexey-Rivkin commented Apr 28, 2025

Uh oh!

Uh oh!

Uh oh!

tstamler commented Apr 29, 2025

Uh oh!

Alexey-Rivkin commented Apr 29, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pvijayakrish left a comment

Uh oh!

Alexey-Rivkin commented May 8, 2025 •

edited

Loading

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Alexey-Rivkin commented Apr 28, 2025

What?

Why?

How?

Uh oh!

Uh oh!

Uh oh!

tstamler commented Apr 29, 2025

Uh oh!

Alexey-Rivkin commented Apr 29, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pvijayakrish left a comment

Choose a reason for hiding this comment

Uh oh!

Alexey-Rivkin commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Results propagation from AWS

Testing done

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Alexey-Rivkin commented May 8, 2025 •

edited

Loading