Skip to content

refactor: move library crates into crates/ directory #676

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CatherineSue merged 4 commits into from
Mar 8, 2026
Merged

refactor: move library crates into crates/ directory #676

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a463d7d
refactor: move library crates into crates/ directory
CatherineSue Mar 8, 2026
4d44546
fix(wasm): update REPO_ROOT path in build_fixtures.sh after crate move
CatherineSue Mar 8, 2026
963f80f
fix(wasm,scripts): update WIT paths in examples and add legacy path f…
CatherineSue Mar 8, 2026
6e98d23
fix: update remaining hardcoded paths for crates/ move (e2e, multimod…
CatherineSue Mar 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
16 changes: 8 additions & 8 deletions .github/workflows/benchmark-radix-tree.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,18 @@ on:
branches: [ main ]
paths:
- 'model_gateway/benches/radix_tree_benchmark.rs'
- 'kv_index/**'
- '!kv_index/**/tests/**'
- '!kv_index/**/*_test.rs'
- '!kv_index/**/test_*.rs'
- 'crates/kv_index/**'
- '!crates/kv_index/**/tests/**'
- '!crates/kv_index/**/*_test.rs'
- '!crates/kv_index/**/test_*.rs'
pull_request:
branches: [ main ]
paths:
- 'model_gateway/benches/radix_tree_benchmark.rs'
- 'kv_index/**'
- '!kv_index/**/tests/**'
- '!kv_index/**/*_test.rs'
- '!kv_index/**/test_*.rs'
- 'crates/kv_index/**'
- '!crates/kv_index/**/tests/**'
- '!crates/kv_index/**/*_test.rs'
- '!crates/kv_index/**/test_*.rs'
workflow_dispatch:
schedule:
- cron: '0 0 * * 0'
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/benchmark-tokenizer.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,18 @@ on:
branches: [ main ]
paths:
- 'model_gateway/benches/tokenizer_benchmark.rs'
- 'tokenizer/**'
- '!tokenizer/**/tests/**'
- '!tokenizer/**/*_test.rs'
- '!tokenizer/**/test_*.rs'
- 'crates/tokenizer/**'
- '!crates/tokenizer/**/tests/**'
- '!crates/tokenizer/**/*_test.rs'
- '!crates/tokenizer/**/test_*.rs'
pull_request:
branches: [ main ]
paths:
- 'model_gateway/benches/tokenizer_benchmark.rs'
- 'tokenizer/**'
- '!tokenizer/**/tests/**'
- '!tokenizer/**/*_test.rs'
- '!tokenizer/**/test_*.rs'
- 'crates/tokenizer/**'
- '!crates/tokenizer/**/tests/**'
- '!crates/tokenizer/**/*_test.rs'
- '!crates/tokenizer/**/test_*.rs'
workflow_dispatch:
schedule:
- cron: '0 0 * * 0'
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/benchmark-tool-parser.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,18 @@ on:
branches: [ main ]
paths:
- 'model_gateway/benches/tool_parser_benchmark.rs'
- 'tool_parser/**'
- '!tool_parser/**/tests/**'
- '!tool_parser/**/*_test.rs'
- '!tool_parser/**/test_*.rs'
- 'crates/tool_parser/**'
- '!crates/tool_parser/**/tests/**'
- '!crates/tool_parser/**/*_test.rs'
- '!crates/tool_parser/**/test_*.rs'
pull_request:
branches: [ main ]
paths:
- 'model_gateway/benches/tool_parser_benchmark.rs'
- 'tool_parser/**'
- '!tool_parser/**/tests/**'
- '!tool_parser/**/*_test.rs'
- '!tool_parser/**/test_*.rs'
- 'crates/tool_parser/**'
- '!crates/tool_parser/**/tests/**'
- '!crates/tool_parser/**/*_test.rs'
- '!crates/tool_parser/**/test_*.rs'
workflow_dispatch:
schedule:
- cron: '0 0 * * 0'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nightly-benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
path: |
bindings/python/dist/*.whl
clients/python/smg_client/types/_generated.py
key: nightly-wheel-${{ runner.os }}-${{ hashFiles('Cargo.lock', '**/Cargo.toml', 'auth/src/**', 'bindings/python/src/**', 'data_connector/src/**', 'grpc_client/src/**', 'kv_index/src/**', 'mcp/src/**', 'mesh/src/**', 'model_gateway/src/**', 'multimodal/src/**', 'protocols/src/**', 'reasoning_parser/src/**', 'tokenizer/src/**', 'tool_parser/src/**', 'wasm/src/**', 'workflow/src/**') }}
key: nightly-wheel-${{ runner.os }}-${{ hashFiles('Cargo.lock', '**/Cargo.toml', 'crates/auth/src/**', 'bindings/python/src/**', 'crates/data_connector/src/**', 'crates/grpc_client/src/**', 'crates/kv_index/src/**', 'crates/mcp/src/**', 'crates/mesh/src/**', 'model_gateway/src/**', 'crates/multimodal/src/**', 'crates/protocols/src/**', 'crates/reasoning_parser/src/**', 'crates/tokenizer/src/**', 'crates/tool_parser/src/**', 'crates/wasm/src/**', 'crates/workflow/src/**') }}
Comment thread
CatherineSue marked this conversation as resolved.
restore-keys: |
nightly-wheel-${{ runner.os }}-

Expand Down
46 changes: 23 additions & 23 deletions .github/workflows/pr-test-rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,16 +90,16 @@ jobs:

- name: Copy protos and build
run: |
rm -f grpc_client/python/smg_grpc_proto/proto
mkdir -p grpc_client/python/smg_grpc_proto/proto
cp grpc_client/proto/*.proto grpc_client/python/smg_grpc_proto/proto/
rm -f crates/grpc_client/python/smg_grpc_proto/proto
mkdir -p crates/grpc_client/python/smg_grpc_proto/proto
cp crates/grpc_client/proto/*.proto crates/grpc_client/python/smg_grpc_proto/proto/
pip install build grpcio-tools
rm -rf grpc_client/python/dist/
cd grpc_client/python && python -m build
rm -rf crates/grpc_client/python/dist/
cd crates/grpc_client/python && python -m build

- name: Test import
run: |
pip install grpc_client/python/dist/*.whl
pip install crates/grpc_client/python/dist/*.whl
python -c "from smg_grpc_proto import sglang_scheduler_pb2; print('OK')"

build-wheel:
Expand All @@ -116,9 +116,9 @@ jobs:
path: |
bindings/python/dist/*.whl
bindings/golang/target/release/libsmg_go.*
wasm/tests/fixtures/*.wasm
crates/wasm/tests/fixtures/*.wasm
clients/python/smg_client/types/_generated.py
key: wheel-${{ runner.os }}-${{ hashFiles('Cargo.lock', '**/Cargo.toml', 'auth/src/**', 'bindings/python/src/**', 'data_connector/src/**', 'grpc_client/src/**', 'kv_index/src/**', 'mcp/src/**', 'mesh/src/**', 'model_gateway/src/**', 'multimodal/src/**', 'protocols/src/**', 'reasoning_parser/src/**', 'tokenizer/src/**', 'tool_parser/src/**', 'wasm/src/**', 'workflow/src/**', 'bindings/golang/src/**', 'examples/wasm/**') }}
key: wheel-${{ runner.os }}-${{ hashFiles('Cargo.lock', '**/Cargo.toml', 'crates/auth/src/**', 'bindings/python/src/**', 'crates/data_connector/src/**', 'crates/grpc_client/src/**', 'crates/kv_index/src/**', 'crates/mcp/src/**', 'crates/mesh/src/**', 'model_gateway/src/**', 'crates/multimodal/src/**', 'crates/protocols/src/**', 'crates/reasoning_parser/src/**', 'crates/tokenizer/src/**', 'crates/tool_parser/src/**', 'crates/wasm/src/**', 'crates/workflow/src/**', 'bindings/golang/src/**', 'examples/wasm/**') }}
restore-keys: |
wheel-${{ runner.os }}-

Expand Down Expand Up @@ -157,7 +157,7 @@ jobs:
if: steps.cache-wheel.outputs.cache-hit != 'true'
run: |
source "$HOME/.cargo/env"
bash wasm/tests/fixtures/build_fixtures.sh
bash crates/wasm/tests/fixtures/build_fixtures.sh

@chatgpt-codex-connector chatgpt-codex-connector Bot Mar 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update WASM fixture script for new crates/ path

This workflow step now invokes crates/wasm/tests/fixtures/build_fixtures.sh, but that script still computes REPO_ROOT via ../../.. and then reads examples from $REPO_ROOT/examples/wasm; from the new location this resolves to <repo>/crates/examples/wasm, which does not exist. As a result, whenever the wheel cache is cold and this step runs, fixture generation fails and blocks the Rust PR pipeline.

Useful? React with 👍 / 👎.

@CatherineSue CatherineSue Mar 8, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4d44546 — updated REPO_ROOT to use ../../../.. and FIXTURES_DIR to use crates/wasm/tests/fixtures.


- name: Upload wheel artifact
uses: actions/upload-artifact@v7
Expand All @@ -184,7 +184,7 @@ jobs:
uses: actions/upload-artifact@v7
with:
name: wasm-test-fixtures
path: wasm/tests/fixtures/*.wasm
path: crates/wasm/tests/fixtures/*.wasm
retention-days: 1
if-no-files-found: ignore

Expand Down Expand Up @@ -250,7 +250,7 @@ jobs:
- name: Build WASM test fixtures
run: |
source "$HOME/.cargo/env"
bash wasm/tests/fixtures/build_fixtures.sh
bash crates/wasm/tests/fixtures/build_fixtures.sh

- name: Run lint
run: |
Expand All @@ -270,7 +270,7 @@ jobs:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

pip install transformers pillow numpy scipy
python multimodal/scripts/generate_vision_golden.py
python crates/multimodal/scripts/generate_vision_golden.py
Comment thread
CatherineSue marked this conversation as resolved.

- name: Run Rust tests
timeout-minutes: 30
Expand Down Expand Up @@ -300,7 +300,7 @@ jobs:
filters: |
common:
- 'model_gateway/**'
- 'protocols/**'
- 'crates/protocols/**'
- 'bindings/**'
- 'e2e_test/conftest.py'
- 'e2e_test/infra/**'
Expand All @@ -311,20 +311,20 @@ jobs:
- 'scripts/ci_setup_python_venv.sh'
- 'scripts/ci_install_sglang.sh'
- 'scripts/ci_build_wheel.sh'
- 'tokenizer/**'
- 'tool_parser/**'
- 'crates/tokenizer/**'
- 'crates/tool_parser/**'
chat-completions:
- 'reasoning_parser/**'
- 'multimodal/**'
- 'grpc_client/**'
- 'crates/reasoning_parser/**'
- 'crates/multimodal/**'
- 'crates/grpc_client/**'
- 'grpc_servicer/**'
- 'e2e_test/chat_completions/**'
- 'e2e_test/router/test_pd_mmlu.py'
- 'scripts/ci_install_vllm.sh'
- 'scripts/ci_install_trtllm.sh'
agentic:
- 'mcp/**'
- 'data_connector/**'
- 'crates/mcp/**'
- 'crates/data_connector/**'
- 'e2e_test/responses/**'
- 'e2e_test/messages/**'
- 'scripts/ci_agentic_svc_deps.sh'
Expand Down Expand Up @@ -388,7 +388,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: wasm-test-fixtures
path: wasm/tests/fixtures/
path: crates/wasm/tests/fixtures/
continue-on-error: true

- name: Download Python client types
Expand Down Expand Up @@ -512,7 +512,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: wasm-test-fixtures
path: wasm/tests/fixtures/
path: crates/wasm/tests/fixtures/
continue-on-error: true

- name: Download Python client types
Expand Down Expand Up @@ -588,7 +588,7 @@ jobs:
uses: actions/download-artifact@v8
with:
name: wasm-test-fixtures
path: wasm/tests/fixtures/
path: crates/wasm/tests/fixtures/
Comment thread
CatherineSue marked this conversation as resolved.
continue-on-error: true

- name: Download Python client types
Expand Down
26 changes: 13 additions & 13 deletions .github/workflows/release-crates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,21 +22,21 @@ jobs:
matrix:
include:
- crate: openai-protocol
path: protocols
path: crates/protocols
- crate: reasoning-parser
path: reasoning_parser
path: crates/reasoning_parser
- crate: kv-index
path: kv_index
path: crates/kv_index
- crate: data-connector
path: data_connector
path: crates/data_connector
- crate: wfaas
path: workflow
path: crates/workflow
- crate: smg-wasm
path: wasm
path: crates/wasm
- crate: smg-auth
path: auth
path: crates/auth
- crate: llm-tokenizer
path: tokenizer
path: crates/tokenizer
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/publish-crate
Expand All @@ -54,15 +54,15 @@ jobs:
matrix:
include:
- crate: tool-parser
path: tool_parser
path: crates/tool_parser
- crate: smg-mcp
path: mcp
path: crates/mcp
- crate: smg-grpc-client
path: grpc_client
path: crates/grpc_client
- crate: llm-multimodal
path: multimodal
path: crates/multimodal
- crate: smg-mesh
path: mesh
path: crates/mesh
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/publish-crate
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/release-grpc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:
branches:
- main
paths:
- grpc_client/python/pyproject.toml
- crates/grpc_client/python/pyproject.toml
- grpc_servicer/pyproject.toml
workflow_dispatch:

Expand Down Expand Up @@ -35,7 +35,7 @@ jobs:
base="${{ github.event.before }}"
head="${{ github.sha }}"
changed="$(git diff --name-only "$base" "$head")"
proto=$(echo "$changed" | grep -q '^grpc_client/python/pyproject.toml$' && echo true || echo false)
proto=$(echo "$changed" | grep -q '^crates/grpc_client/python/pyproject.toml$' && echo true || echo false)
servicer=$(echo "$changed" | grep -q '^grpc_servicer/pyproject.toml$' && echo true || echo false)
echo "proto=$proto" >> "$GITHUB_OUTPUT"
echo "servicer=$servicer" >> "$GITHUB_OUTPUT"
Expand All @@ -60,20 +60,20 @@ jobs:

- name: Copy proto files (replace symlink with real files)
run: |
rm -f grpc_client/python/smg_grpc_proto/proto
mkdir -p grpc_client/python/smg_grpc_proto/proto
cp grpc_client/proto/*.proto grpc_client/python/smg_grpc_proto/proto/
rm -f crates/grpc_client/python/smg_grpc_proto/proto
mkdir -p crates/grpc_client/python/smg_grpc_proto/proto
cp crates/grpc_client/proto/*.proto crates/grpc_client/python/smg_grpc_proto/proto/

- name: Build package
run: cd grpc_client/python && python -m build
run: cd crates/grpc_client/python && python -m build

- name: Check package
run: twine check --strict grpc_client/python/dist/*
run: twine check --strict crates/grpc_client/python/dist/*

- uses: actions/upload-artifact@v7
with:
name: grpc-proto-dist
path: grpc_client/python/dist/
path: crates/grpc_client/python/dist/

upload-proto:
name: Upload proto to PyPI
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ repos:
src/proto/.*\.proto$|
src/mesh/proto/.*\.proto$|
src/tokenizer/chat_template\.rs|
multimodal/tests/fixtures/.*|
crates/multimodal/tests/fixtures/.*|
target/.*
)$

Expand Down
28 changes: 14 additions & 14 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
[workspace]
members = ["model_gateway", "protocols", "reasoning_parser", "tool_parser", "workflow", "tokenizer", "auth", "mcp", "kv_index", "data_connector", "multimodal", "wasm", "mesh", "grpc_client", "bindings/python", "bindings/golang", "clients/rust", "clients/openapi-gen"]
members = ["model_gateway", "crates/protocols", "crates/reasoning_parser", "crates/tool_parser", "crates/workflow", "crates/tokenizer", "crates/auth", "crates/mcp", "crates/kv_index", "crates/data_connector", "crates/multimodal", "crates/wasm", "crates/mesh", "crates/grpc_client", "bindings/python", "bindings/golang", "clients/rust", "clients/openapi-gen"]
resolver = "2"

[workspace.dependencies]
# Internal workspace crates
openai-protocol = { version = "1.2.0", path = "protocols" }
reasoning-parser = { version = "1.2.0", path = "reasoning_parser" }
tool-parser = { version = "1.1.1", path = "tool_parser" }
wfaas = { version = "1.0.2", path = "workflow" }
llm-tokenizer = { version = "1.2.0", path = "tokenizer" }
smg-auth = { version = "1.1.1", path = "auth" }
smg-mcp = { version = "2.1.0", path = "mcp" }
kv-index = { version = "1.0.1", path = "kv_index" }
smg-data-connector = { version = "2.0.0", path = "data_connector", package = "data-connector" }
llm-multimodal = { version = "1.2.0", path = "multimodal" }
smg-wasm = { version = "1.0.1", path = "wasm", package = "smg-wasm" }
smg-mesh = { version = "1.1.1", path = "mesh", package = "smg-mesh" }
smg-grpc-client = { version = "1.2.0", path = "grpc_client" }
openai-protocol = { version = "1.2.0", path = "crates/protocols" }
reasoning-parser = { version = "1.2.0", path = "crates/reasoning_parser" }
tool-parser = { version = "1.1.1", path = "crates/tool_parser" }
wfaas = { version = "1.0.2", path = "crates/workflow" }
llm-tokenizer = { version = "1.2.0", path = "crates/tokenizer" }
smg-auth = { version = "1.1.1", path = "crates/auth" }
smg-mcp = { version = "2.1.0", path = "crates/mcp" }
kv-index = { version = "1.0.1", path = "crates/kv_index" }
smg-data-connector = { version = "2.0.0", path = "crates/data_connector", package = "data-connector" }
llm-multimodal = { version = "1.2.0", path = "crates/multimodal" }
smg-wasm = { version = "1.0.1", path = "crates/wasm", package = "smg-wasm" }
smg-mesh = { version = "1.1.1", path = "crates/mesh", package = "smg-mesh" }
smg-grpc-client = { version = "1.2.0", path = "crates/grpc_client" }

# Shared dependencies
anyhow = "1.0"
Expand Down
4 changes: 2 additions & 2 deletions bindings/golang/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ default-features = true
workspace = true

[dependencies.llm-tokenizer]
path = "../../tokenizer"
path = "../../crates/tokenizer"

[dependencies.tool-parser]
path = "../../tool_parser"
path = "../../crates/tool_parser"
Comment thread
CatherineSue marked this conversation as resolved.

[dependencies.smg-grpc-client]
workspace = true
Expand Down
4 changes: 2 additions & 2 deletions bindings/python/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ path = "../../model_gateway"
default-features = true

[dependencies.smg-auth]
path = "../../auth"
path = "../../crates/auth"

[dependencies.tool-parser]
path = "../../tool_parser"
path = "../../crates/tool_parser"
Comment thread
CatherineSue marked this conversation as resolved.

[features]
default = ["pyo3/extension-module"]
Expand Down
Loading
Loading