Skip to content

Add CI for testing AReaLite #150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
garrett4wade merged 25 commits into from
Jul 7, 2025
Merged

Add CI for testing AReaLite #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
0290371
ci: add test-arealite
futrime Jul 3, 2025
becea9a
ci: add checkout before running test-arealite
futrime Jul 3, 2025
c13279a
ci: add USERNAME
futrime Jul 3, 2025
5a0b8d3
ci: add test script
futrime Jul 3, 2025
5fe18b1
ci: add GitHub mirror
futrime Jul 3, 2025
c274029
ci: fix typo
futrime Jul 3, 2025
d9926ff
ci: clone one commit
futrime Jul 3, 2025
732f12e
ci: fix condition
futrime Jul 3, 2025
9efbfa5
ci: set command timeout to 60m
futrime Jul 3, 2025
95cfc73
ci: enable pip cache
futrime Jul 3, 2025
0ee70d5
ci: optimize container lifecycle
futrime Jul 3, 2025
4ab4fc7
ci: split into many stages
futrime Jul 4, 2025
c75b570
ci(test-arealite): fix typo
futrime Jul 4, 2025
4d11e16
Merge remote-tracking branch 'origin/fw/refactor' into zz/refactor/ar…
futrime Jul 4, 2025
59609c2
ci: fix wrong env
futrime Jul 4, 2025
07b574f
ci: fix pytest
futrime Jul 4, 2025
49c30b4
ci: uninstall transformer-engine
futrime Jul 4, 2025
5cd2d73
ci: uninstall transformer-engine
futrime Jul 4, 2025
7f50c5d
ci: fix model paths
futrime Jul 4, 2025
1d63632
ci: show stdout/stderr
futrime Jul 4, 2025
96745be
ci: fix not clean up
futrime Jul 4, 2025
27aac79
ci: backup sglang
futrime Jul 5, 2025
18e9d2f
ci: remove tmp repo dir when run
futrime Jul 5, 2025
c60a39f
ci: fix docker run exit 1 condition
futrime Jul 6, 2025
08d7b68
ci(test-arealite): limit the concurrency and extend command timeout
futrime Jul 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/test-arealite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Test AReaLite

on:
push:
paths:
- .github/workflows/test-arealite.yml
- arealite/**
- ci/**
workflow_dispatch:

jobs:
test-arealite:
runs-on: ubuntu-latest
concurrency:
group: test-arealite
steps:
- uses: actions/checkout@v4

- uses: appleboy/ssh-action@v1
env:
GIT_REPO_URL: https://github.bibk.top/${{ github.repository }}
GIT_COMMIT_SHA: ${{ github.sha }}
with:
host: ${{ secrets.CI_NODE_ADDR }}
username: ${{ secrets.CI_NODE_USER }}
key: ${{ secrets.REMOTE_SSH_KEY }}
envs: GIT_REPO_URL,GIT_COMMIT_SHA
script_path: ci/clone_repo.sh

- uses: appleboy/ssh-action@v1
env:
GIT_COMMIT_SHA: ${{ github.sha }}
with:
host: ${{ secrets.CI_NODE_ADDR }}
username: ${{ secrets.CI_NODE_USER }}
key: ${{ secrets.REMOTE_SSH_KEY }}
command_timeout: 2h
envs: GIT_COMMIT_SHA
script_path: ci/build_env_image.sh

- uses: appleboy/ssh-action@v1
env:
GIT_COMMIT_SHA: ${{ github.sha }}
with:
host: ${{ secrets.CI_NODE_ADDR }}
username: ${{ secrets.CI_NODE_USER }}
key: ${{ secrets.REMOTE_SSH_KEY }}
command_timeout: 1h
envs: GIT_COMMIT_SHA
script_path: ci/test_arealite.sh
2 changes: 1 addition & 1 deletion arealite/tests/test_engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
from realhf.impl.model.utils.padding import unpad_input

VOCAB_SIZE = 100
MODEL_PATH = "/storage/openpsi/models/Qwen__Qwen3-1.7B/"
MODEL_PATH = "Qwen/Qwen2-0.5B"


@pytest.fixture(scope="module")
Expand Down
2 changes: 1 addition & 1 deletion arealite/tests/test_grpo.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

EXPR_NAME = "test_grpo"
TRIAL_NAME = "test_grpo"
MODEL_PATH = "/storage/openpsi/models/Qwen__Qwen3-1.7B/"
MODEL_PATH = "Qwen/Qwen2-0.5B"


@pytest.fixture(scope="module")
Expand Down
2 changes: 1 addition & 1 deletion arealite/tests/test_rollout.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@

EXPR_NAME = "test_rollout"
TRIAL_NAME = "test_rollout"
MODEL_PATH = "/storage/openpsi/models/Qwen__Qwen3-1.7B/"
MODEL_PATH = "Qwen/Qwen2-0.5B"


@pytest.fixture(scope="module")
Expand Down
2 changes: 1 addition & 1 deletion arealite/tests/test_rollout_controller.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

EXPR_NAME = "test_rollout_controller"
TRIAL_NAME = "test_rollout_controller"
MODEL_PATH = "/storage/openpsi/models/Qwen__Qwen3-1.7B/"
MODEL_PATH = "Qwen/Qwen2-0.5B"


@pytest.fixture(scope="module")
Expand Down
2 changes: 1 addition & 1 deletion arealite/tests/test_sft.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def test_sft():
)

engine_config = EngineConfig(
path="/storage/openpsi/models/Qwen__Qwen3-1.7B/",
path="Qwen/Qwen2-0.5B",
gradient_checkpointing=False,
optimizer=OptimizerConfig(),
backend=EngineBackendConfig(type="hf"),
Expand Down
2 changes: 1 addition & 1 deletion arealite/tests/test_sglang_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@

EXPR_NAME = "test_sglang_client"
TRIAL_NAME = "test_sglang_client"
MODEL_PATH = "/storage/openpsi/models/Qwen__Qwen3-1.7B/"
MODEL_PATH = "Qwen/Qwen2-0.5B"
Copy link

Copilot AI Jul 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] This MODEL_PATH is duplicated across multiple tests; consider extracting it into a shared fixture or constant to reduce duplication and ease future updates.

Copilot uses AI. Check for mistakes.


@pytest.fixture(scope="module")
Expand Down
39 changes: 39 additions & 0 deletions ci/build_env_image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#!/usr/bin/env bash

set -e

GIT_COMMIT_SHA=${GIT_COMMIT_SHA:?"GIT_COMMIT_SHA is not set"}

echo "GIT_COMMIT_SHA: $GIT_COMMIT_SHA"

# If there is already an image named areal-env, skip.
if docker images --format '{{.Repository}}:{{.Tag}}' | grep -q 'areal-env:latest'; then
echo "Image areal-env already exists, skipping build."
exit 0
fi

RUN_ID="areal-$GIT_COMMIT_SHA"
cd "/tmp/$RUN_ID"

if docker ps -a --format '{{.Names}}' | grep -q "$RUN_ID"; then
docker rm -f $RUN_ID
fi

docker run \
--name $RUN_ID \
--gpus all \
--shm-size=8g \
-v $(pwd):/workspace \
Copy link

Copilot AI Jul 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quote the volume path to handle directories with spaces, e.g., -v "$(pwd)":/workspace.

Suggested change
-v $(pwd):/workspace \
-v "$(pwd)":/workspace \

Copilot uses AI. Check for mistakes.
-w /workspace \
nvcr.io/nvidia/pytorch:25.01-py3 \
bash -c "
python -m pip install --upgrade pip
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip config unset global.extra-index-url
bash examples/env/scripts/setup-pip-deps.sh
pip uninstall -y transformer-engine
mv ./sglang /sglang
" || { docker rm -f $RUN_ID; exit 1; }

docker commit $RUN_ID areal-env:latest
docker rm -f $RUN_ID
19 changes: 19 additions & 0 deletions ci/clone_repo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/usr/bin/env bash

set -e

GIT_REPO_URL=${GIT_REPO_URL:?"GIT_REPO_URL is not set"}
GIT_COMMIT_SHA=${GIT_COMMIT_SHA:?"GIT_COMMIT_SHA is not set"}

echo "GIT_REPO_URL: $GIT_REPO_URL"
echo "GIT_COMMIT_SHA: $GIT_COMMIT_SHA"

RUN_ID="areal-$GIT_COMMIT_SHA"
rm -rf "/tmp/$RUN_ID"
mkdir -p "/tmp/$RUN_ID"
cd "/tmp/$RUN_ID"

git init
git remote add origin "$GIT_REPO_URL"
git fetch --depth 1 origin "$GIT_COMMIT_SHA"
git checkout FETCH_HEAD
28 changes: 28 additions & 0 deletions ci/test_arealite.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/usr/bin/env bash

set -e
Copy link

Copilot AI Jul 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding set -u (or set -eu) to treat unset variables as errors and avoid silent failures.

Suggested change
set -e
set -eu

Copilot uses AI. Check for mistakes.

GIT_COMMIT_SHA=${GIT_COMMIT_SHA:?"GIT_COMMIT_SHA is not set"}

echo "GIT_COMMIT_SHA: $GIT_COMMIT_SHA"

RUN_ID="areal-$GIT_COMMIT_SHA"
cd "/tmp/$RUN_ID"

if docker ps -a --format '{{.Names}}' | grep -q "$RUN_ID"; then
docker rm -f $RUN_ID
fi

docker run \
--name $RUN_ID \
--gpus all \
--shm-size=8g \
-v $(pwd):/workspace \
Copy link

Copilot AI Jul 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quote the volume path to handle directories with spaces, e.g., -v "$(pwd)":/workspace.

Suggested change
-v $(pwd):/workspace \
-v "$(pwd)":/workspace \

Copilot uses AI. Check for mistakes.
-w /workspace \
areal-env:latest \
bash -c "
mv /sglang ./sglang
HF_ENDPOINT=https://hf-mirror.com python -m pytest -s arealite/
" || { docker rm -f $RUN_ID; exit 1; }

docker rm -f $RUN_ID
Loading