Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions .github/workflows/release-sglang-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@ name: Release SMG+SGLang Docker Image

run-name: >-
SMG+SGLang |
base=${{ inputs.base_image_ref || 'lmsysorg/sglang:v0.5.9' }} |
${{ github.event_name == 'push' && '3-version matrix' || format('base={0}', inputs.base_image_ref || 'lmsysorg/sglang:v0.5.9') }} |
engine=${{ inputs.sglang_commit || 'latest' }} |
smg=${{ inputs.smg_commit || 'v1.3.3' }} |
smg=${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }} |
${{ github.event_name == 'pull_request' && 'dry-run |' || '' }}
by @${{ github.actor }}

on:
Expand Down Expand Up @@ -56,10 +57,14 @@ permissions:

jobs:
build:
strategy:
fail-fast: false
matrix:
base_image: ${{ github.event_name == 'push' && fromJSON('["lmsysorg/sglang:v0.5.9","lmsysorg/sglang:v0.5.8","lmsysorg/sglang:v0.5.7"]') || fromJSON(format('["{0}"]', inputs.base_image_ref || 'lmsysorg/sglang:v0.5.9')) }}
uses: ./.github/workflows/_build-engine-image.yml
with:
engine: sglang
base_image_ref: ${{ inputs.base_image_ref || 'lmsysorg/sglang:v0.5.9' }}
base_image_ref: ${{ matrix.base_image }}
engine_repo: ${{ inputs.sglang_repo }}
engine_commit: ${{ inputs.sglang_commit || 'latest' }}
smg_repo: ${{ inputs.smg_repo || 'https://github.com/lightseekorg/smg' }}
Expand Down
21 changes: 17 additions & 4 deletions .github/workflows/release-trtllm-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,24 @@ name: Release SMG+TRTLLM Docker Image

run-name: >-
SMG+TRTLLM |
base=${{ inputs.base_image_ref }} |
engine=${{ inputs.trtllm_commit }} |
smg=${{ inputs.smg_commit }} |
${{ github.event_name == 'push' && '3-version matrix' || format('base={0}', inputs.base_image_ref || 'nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8') }} |
engine=${{ inputs.trtllm_commit || 'latest' }} |
smg=${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }} |
${{ github.event_name == 'pull_request' && 'dry-run |' || '' }}
by @${{ github.actor }}

on:
push:
branches: [main]
paths:
- bindings/python/pyproject.toml
pull_request:
branches: [main]
paths:
- 'docker/engine.Dockerfile'
- '.github/workflows/_build-engine-image.yml'
Comment on lines +16 to +20

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restrict TRTLLM PR builds on self-hosted runner

This newly added pull_request trigger allows fork-originated PRs that touch these paths to run the reusable build job, and that job executes on the self-hosted 8-gpu-h200 runner (.github/workflows/_build-engine-image.yml). Because dry_run only skips pushing images, attacker-controlled PR content can still execute arbitrary build steps on that host; for public/forked PRs this is a runner-compromise risk and should be gated to trusted actors/branches or moved to isolated GitHub-hosted runners.

Useful? React with 👍 / 👎.

- '.github/workflows/release-*-docker.yml'
- 'scripts/installation/**'
workflow_dispatch:
inputs:
base_image_ref:
Expand Down Expand Up @@ -49,15 +57,20 @@ permissions:

jobs:
build:
strategy:
fail-fast: false
matrix:
base_image: ${{ github.event_name == 'push' && fromJSON('["nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8","nvcr.io/nvidia/tensorrt-llm/release:1.2.0","nvcr.io/nvidia/tensorrt-llm/release:1.1.0"]') || fromJSON(format('["{0}"]', inputs.base_image_ref || 'nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8')) }}
uses: ./.github/workflows/_build-engine-image.yml
with:
engine: trtllm
base_image_ref: ${{ inputs.base_image_ref || 'nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8' }}
base_image_ref: ${{ matrix.base_image }}
engine_repo: ${{ inputs.trtllm_repo }}
engine_commit: ${{ inputs.trtllm_commit || 'latest' }}
smg_repo: ${{ inputs.smg_repo || 'https://github.com/lightseekorg/smg' }}
smg_commit: ${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }}
tag: ${{ inputs.tag }}
source_build_repo: ${{ inputs.trtllm_repo }}
source_build_ref: ${{ inputs.trtllm_commit || 'latest' }}
dry_run: ${{ github.event_name == 'pull_request' }}
Comment on lines +60 to +75

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check that _build-engine-image.yml declares dry_run as boolean input and uses it to skip push
rg -n 'dry_run' .github/workflows/_build-engine-image.yml -A 3 -B 1

Repository: lightseekorg/smg

Length of output: 582


Declare dry_run as type: boolean in the reusable workflow input.

The dry_run: ${{ github.event_name == 'pull_request' }} expression passes a boolean value, but the _build-engine-image.yml workflow does not explicitly declare dry_run as a boolean type input. While the default is correctly set to false, the input type should be explicitly declared as boolean to ensure proper type coercion and maintain consistency with the calling expression.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/release-trtllm-docker.yml around lines 60 - 75, The
reusable workflow called by this file expects a dry_run input but does not
declare its type; update .github/workflows/_build-engine-image.yml to add an
explicit input definition for dry_run (e.g., under inputs: add dry_run with
type: boolean and a default of false) so the caller's expression dry_run: ${{
github.event_name == 'pull_request' }} is correctly coerced as a boolean;
reference the dry_run input name and the _build-engine-image.yml workflow when
making the change.

secrets: inherit
21 changes: 17 additions & 4 deletions .github/workflows/release-vllm-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,24 @@ name: Release SMG+vLLM Docker Image

run-name: >-
SMG+vLLM |
base=${{ inputs.base_image_ref }} |
engine=${{ inputs.vllm_commit }} |
smg=${{ inputs.smg_commit }} |
${{ github.event_name == 'push' && '3-version matrix' || format('base={0}', inputs.base_image_ref || 'vllm/vllm-openai:v0.18.0') }} |
engine=${{ inputs.vllm_commit || 'latest' }} |
smg=${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }} |
${{ github.event_name == 'pull_request' && 'dry-run |' || '' }}
by @${{ github.actor }}

on:
push:
branches: [main]
paths:
- bindings/python/pyproject.toml
pull_request:
branches: [main]
paths:
Comment on lines +16 to +18

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restrict untrusted PR execution on GPU self-hosted runners

This new pull_request trigger lets fork-originated PRs run the reusable build job, which executes checkout and Docker build steps on the self-hosted 8-gpu-h200 runner (.github/workflows/_build-engine-image.yml, runs-on). dry_run only skips pushing images; it does not prevent arbitrary PR code from running on that host. If public forks can open PRs (default for public repos), this materially increases runner-compromise risk; gate these jobs to trusted actors/branches or move PR validation to isolated GitHub-hosted runners.

Useful? React with 👍 / 👎.

- 'docker/engine.Dockerfile'
- '.github/workflows/_build-engine-image.yml'
- '.github/workflows/release-*-docker.yml'
- 'scripts/installation/**'
workflow_dispatch:
inputs:
base_image_ref:
Expand Down Expand Up @@ -49,13 +57,18 @@ permissions:

jobs:
build:
strategy:
fail-fast: false
matrix:
base_image: ${{ github.event_name == 'push' && fromJSON('["vllm/vllm-openai:v0.18.0","vllm/vllm-openai:v0.17.1","vllm/vllm-openai:v0.16.0"]') || fromJSON(format('["{0}"]', inputs.base_image_ref || 'vllm/vllm-openai:v0.18.0')) }}
uses: ./.github/workflows/_build-engine-image.yml
with:
engine: vllm
base_image_ref: ${{ inputs.base_image_ref || 'vllm/vllm-openai:v0.18.0' }}
base_image_ref: ${{ matrix.base_image }}
engine_repo: ${{ inputs.vllm_repo }}
engine_commit: ${{ inputs.vllm_commit || 'latest' }}
smg_repo: ${{ inputs.smg_repo || 'https://github.com/lightseekorg/smg' }}
smg_commit: ${{ github.event_name == 'pull_request' && 'main' || inputs.smg_commit || 'v1.3.3' }}
tag: ${{ inputs.tag }}
dry_run: ${{ github.event_name == 'pull_request' }}
secrets: inherit
3 changes: 3 additions & 0 deletions docs/concepts/architecture/high-availability.md
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,9 @@ spec:
name: mesh
```

!!! tip "Engine images"
For all-in-one deployments where each pod runs both gateway and engine, use an engine image tag (e.g., `ghcr.io/lightseekorg/smg:{smg_version}-{engine}-{engine_version}`). See [Getting Started](../../getting-started/index.md#install) for available tags.

### Headless Service

```yaml
Expand Down
3 changes: 3 additions & 0 deletions docs/concepts/architecture/service-discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,9 @@ spec:
name: admin
```

!!! tip "Engine images"
For all-in-one deployments where each pod runs both gateway and engine, use an engine image tag (e.g., `ghcr.io/lightseekorg/smg:{smg_version}-{engine}-{engine_version}`). See [Getting Started](../../getting-started/index.md#install) for available tags.

### Worker StatefulSet

```yaml
Expand Down
42 changes: 39 additions & 3 deletions docs/getting-started/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,32 @@ Shepherd Model Gateway (SMG) routes and manages LLM traffic across workers. This

=== "Docker"

**SMG only** (gateway/router, no inference engine):

Multi-architecture images are available for x86_64 and ARM64.

```bash
docker pull lightseekorg/smg:latest
```

Verify:
Available tags: `latest` (stable), `v1.3.x` (specific version), `nightly` (development, from `ghcr.io/lightseekorg/smg:nightly`).

**SMG + Engine** (all-in-one, ready to serve models):

Engine images bundle SMG with a specific inference engine (x86_64/CUDA only). Use these when you want a single container that can both route and serve.

```bash
docker run --rm lightseekorg/smg:latest --version
# SGLang
docker pull ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9

# vLLM
docker pull ghcr.io/lightseekorg/smg:1.3.3-vllm-v0.18.0

# TensorRT-LLM
docker pull ghcr.io/lightseekorg/smg:1.3.3-trtllm-1.3.0rc8
```

Available tags: `latest` (stable), `v0.3.x` (specific version), `nightly` (development, from `ghcr.io/lightseekorg/smg:nightly`).
Tag format: `{smg_version}-{engine}-{engine_version}`. Browse all tags at [ghcr.io/lightseekorg/smg](https://github.com/lightseekorg/smg/pkgs/container/smg).

=== "From Source"

Expand Down Expand Up @@ -421,6 +434,29 @@ docker ps | grep smg
curl http://localhost:30000/health
```

### All-in-one with engine images

Engine images include both SMG and an inference engine. Use `serve` to launch workers and the gateway together:

```bash
docker run -d --gpus all \
--name smg \
-p 30000:30000 \
-v /path/to/models:/models \
ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9 \
serve \
--backend sglang \
--model-path /models/meta-llama/Llama-3.1-8B-Instruct \
--port 30000
Comment on lines +447 to +450

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Expose router on all interfaces in Docker all-in-one example

This command omits --host 0.0.0.0, so users following it can end up with the router bound to loopback inside the container (smg serve defaults --host to 127.0.0.1 in bindings/python/src/smg/serve.py), which makes the published -p 30000:30000 port unreachable from the host and breaks the immediately following curl http://localhost:30000/... verification steps.

Useful? React with 👍 / 👎.

```

Verify:

```bash
curl http://localhost:30000/health
curl http://localhost:30000/v1/models
```

## Deploy to Kubernetes (Quick Start)

Run SMG in-cluster and use service discovery to pick up worker pods automatically.
Expand Down
Loading