Repository navigation
ci(docker): build 3 engine versions on release, test all engines on PR #858
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
841336d
82e8847
83c3b7d
c504095
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,16 +2,24 @@ name: Release SMG+TRTLLM Docker Image | |
|
|
||
| run-name: >- | ||
| SMG+TRTLLM | | ||
| base=${{ inputs.base_image_ref }} | | ||
| engine=${{ inputs.trtllm_commit }} | | ||
| smg=${{ inputs.smg_commit }} | | ||
| ${{ github.event_name == 'push' && '3-version matrix' || format('base={0}', inputs.base_image_ref || 'nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8') }} | | ||
| engine=${{ inputs.trtllm_commit || 'latest' }} | | ||
| smg=${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }} | | ||
| ${{ github.event_name == 'pull_request' && 'dry-run |' || '' }} | ||
| by @${{ github.actor }} | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| paths: | ||
| - bindings/python/pyproject.toml | ||
| pull_request: | ||
| branches: [main] | ||
| paths: | ||
| - 'docker/engine.Dockerfile' | ||
| - '.github/workflows/_build-engine-image.yml' | ||
| - '.github/workflows/release-*-docker.yml' | ||
| - 'scripts/installation/**' | ||
| workflow_dispatch: | ||
| inputs: | ||
| base_image_ref: | ||
|
|
@@ -49,15 +57,20 @@ permissions: | |
|
|
||
| jobs: | ||
| build: | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| base_image: ${{ github.event_name == 'push' && fromJSON('["nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8","nvcr.io/nvidia/tensorrt-llm/release:1.2.0","nvcr.io/nvidia/tensorrt-llm/release:1.1.0"]') || fromJSON(format('["{0}"]', inputs.base_image_ref || 'nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8')) }} | ||
| uses: ./.github/workflows/_build-engine-image.yml | ||
| with: | ||
| engine: trtllm | ||
| base_image_ref: ${{ inputs.base_image_ref || 'nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8' }} | ||
| base_image_ref: ${{ matrix.base_image }} | ||
| engine_repo: ${{ inputs.trtllm_repo }} | ||
| engine_commit: ${{ inputs.trtllm_commit || 'latest' }} | ||
| smg_repo: ${{ inputs.smg_repo || 'https://github.com/lightseekorg/smg' }} | ||
| smg_commit: ${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }} | ||
| tag: ${{ inputs.tag }} | ||
| source_build_repo: ${{ inputs.trtllm_repo }} | ||
| source_build_ref: ${{ inputs.trtllm_commit || 'latest' }} | ||
| dry_run: ${{ github.event_name == 'pull_request' }} | ||
|
Comment on lines
+60
to
+75
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Check that _build-engine-image.yml declares dry_run as boolean input and uses it to skip push
rg -n 'dry_run' .github/workflows/_build-engine-image.yml -A 3 -B 1Repository: lightseekorg/smg Length of output: 582 Declare The 🤖 Prompt for AI Agents |
||
| secrets: inherit | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,16 +2,24 @@ name: Release SMG+vLLM Docker Image | |
|
|
||
| run-name: >- | ||
| SMG+vLLM | | ||
| base=${{ inputs.base_image_ref }} | | ||
| engine=${{ inputs.vllm_commit }} | | ||
| smg=${{ inputs.smg_commit }} | | ||
| ${{ github.event_name == 'push' && '3-version matrix' || format('base={0}', inputs.base_image_ref || 'vllm/vllm-openai:v0.18.0') }} | | ||
| engine=${{ inputs.vllm_commit || 'latest' }} | | ||
| smg=${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }} | | ||
| ${{ github.event_name == 'pull_request' && 'dry-run |' || '' }} | ||
| by @${{ github.actor }} | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| paths: | ||
| - bindings/python/pyproject.toml | ||
| pull_request: | ||
| branches: [main] | ||
| paths: | ||
|
Comment on lines
+16
to
+18
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This new Useful? React with 👍 / 👎. |
||
| - 'docker/engine.Dockerfile' | ||
| - '.github/workflows/_build-engine-image.yml' | ||
| - '.github/workflows/release-*-docker.yml' | ||
| - 'scripts/installation/**' | ||
| workflow_dispatch: | ||
| inputs: | ||
| base_image_ref: | ||
|
|
@@ -49,13 +57,18 @@ permissions: | |
|
|
||
| jobs: | ||
| build: | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| base_image: ${{ github.event_name == 'push' && fromJSON('["vllm/vllm-openai:v0.18.0","vllm/vllm-openai:v0.17.1","vllm/vllm-openai:v0.16.0"]') || fromJSON(format('["{0}"]', inputs.base_image_ref || 'vllm/vllm-openai:v0.18.0')) }} | ||
| uses: ./.github/workflows/_build-engine-image.yml | ||
| with: | ||
| engine: vllm | ||
| base_image_ref: ${{ inputs.base_image_ref || 'vllm/vllm-openai:v0.18.0' }} | ||
| base_image_ref: ${{ matrix.base_image }} | ||
| engine_repo: ${{ inputs.vllm_repo }} | ||
| engine_commit: ${{ inputs.vllm_commit || 'latest' }} | ||
| smg_repo: ${{ inputs.smg_repo || 'https://github.com/lightseekorg/smg' }} | ||
| smg_commit: ${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }} | ||
| tag: ${{ inputs.tag }} | ||
| dry_run: ${{ github.event_name == 'pull_request' }} | ||
| secrets: inherit | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,19 +29,32 @@ Shepherd Model Gateway (SMG) routes and manages LLM traffic across workers. This | |
|
|
||
| === "Docker" | ||
|
|
||
| **SMG only** (gateway/router, no inference engine): | ||
|
|
||
| Multi-architecture images are available for x86_64 and ARM64. | ||
|
|
||
| ```bash | ||
| docker pull lightseekorg/smg:latest | ||
| ``` | ||
|
|
||
| Verify: | ||
| Available tags: `latest` (stable), `v1.3.x` (specific version), `nightly` (development, from `ghcr.io/lightseekorg/smg:nightly`). | ||
|
|
||
| **SMG + Engine** (all-in-one, ready to serve models): | ||
|
|
||
| Engine images bundle SMG with a specific inference engine (x86_64/CUDA only). Use these when you want a single container that can both route and serve. | ||
|
|
||
| ```bash | ||
| docker run --rm lightseekorg/smg:latest --version | ||
| # SGLang | ||
| docker pull ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9 | ||
|
|
||
| # vLLM | ||
| docker pull ghcr.io/lightseekorg/smg:1.3.3-vllm-v0.18.0 | ||
|
|
||
| # TensorRT-LLM | ||
| docker pull ghcr.io/lightseekorg/smg:1.3.3-trtllm-1.3.0rc8 | ||
| ``` | ||
|
|
||
| Available tags: `latest` (stable), `v0.3.x` (specific version), `nightly` (development, from `ghcr.io/lightseekorg/smg:nightly`). | ||
| Tag format: `{smg_version}-{engine}-{engine_version}`. Browse all tags at [ghcr.io/lightseekorg/smg](https://github.com/lightseekorg/smg/pkgs/container/smg). | ||
|
|
||
| === "From Source" | ||
|
|
||
|
|
@@ -421,6 +434,29 @@ docker ps | grep smg | |
| curl http://localhost:30000/health | ||
| ``` | ||
|
|
||
| ### All-in-one with engine images | ||
|
|
||
| Engine images include both SMG and an inference engine. Use `serve` to launch workers and the gateway together: | ||
|
|
||
| ```bash | ||
| docker run -d --gpus all \ | ||
| --name smg \ | ||
| -p 30000:30000 \ | ||
| -v /path/to/models:/models \ | ||
| ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9 \ | ||
| serve \ | ||
| --backend sglang \ | ||
| --model-path /models/meta-llama/Llama-3.1-8B-Instruct \ | ||
| --port 30000 | ||
|
Comment on lines
+447
to
+450
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This command omits Useful? React with 👍 / 👎. |
||
| ``` | ||
|
|
||
| Verify: | ||
|
|
||
| ```bash | ||
| curl http://localhost:30000/health | ||
| curl http://localhost:30000/v1/models | ||
| ``` | ||
|
|
||
| ## Deploy to Kubernetes (Quick Start) | ||
|
|
||
| Run SMG in-cluster and use service discovery to pick up worker pods automatically. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This newly added
pull_requesttrigger allows fork-originated PRs that touch these paths to run the reusable build job, and that job executes on the self-hosted8-gpu-h200runner (.github/workflows/_build-engine-image.yml). Becausedry_runonly skips pushing images, attacker-controlled PR content can still execute arbitrary build steps on that host; for public/forked PRs this is a runner-compromise risk and should be gated to trusted actors/branches or moved to isolated GitHub-hosted runners.Useful? React with 👍 / 👎.