refactor: move async model config into AsyncDispatchConfig (#526) - #1
Draft
pazqo wants to merge 14 commits into
Draft
refactor: move async model config into AsyncDispatchConfig (#526)#1pazqo wants to merge 14 commits into
pazqo wants to merge 14 commits into
Conversation
pazqo
force-pushed
the
restructure-async-config
branch
2 times, most recently
from
August 3, 2026 09:17
fbb4b54 to
b4409dd
Compare
Async dispatch mode reused ModelGateways (a sync-oriented struct with HTTP/TLS/retry fields) but only read InferencePoolName. Move async model-to-pool mappings into a dedicated AsyncDispatchConfig.Models field with a purpose-built AsyncModelConfig struct. InferencePoolName on ModelGatewayConfig is marked as deprecated. Signed-off-by: Stefano Pascolutti <spascolu@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Stefano Pascolutti <spascolu@redhat.com> Signed-off-by: Stefano Pascolutti <pascoluttistefano@gmail.com>
pazqo
force-pushed
the
restructure-async-config
branch
from
August 3, 2026 09:39
b4409dd to
2fef3a9
Compare
Signed-off-by: Stefano Pascolutti <pascoluttistefano@gmail.com>
pazqo
force-pushed
the
restructure-async-config
branch
from
August 3, 2026 10:05
2fef3a9 to
2f86867
Compare
* release: change release - rename GHA workflow: ci-release.yaml to ci-build.yaml only build "lateset" image for new commits on main branch move build image function to create-release.yaml which now does build binary, helm chart, docker image, push - update generate-release.sh do not rely on local SHA1, it uses "gh" remote create tag, and release branch if it is an offical release, and take SHA1 as input so user no need create tag just give a SHA1 to script. - make RELEASE_TEMPLATE.md clear: not in use - update docs Signed-off-by: Wen Zhou <wenzhou@redhat.com> * update: code review - create-release should not need tag image with SHA1 but only tag Signed-off-by: Wen Zhou <wenzhou@redhat.com> * update: code review Signed-off-by: Wen Zhou <wenzhou@redhat.com> --------- Signed-off-by: Wen Zhou <wenzhou@redhat.com>
Cherry-picks the FOSSA badge change from llm-d#567, which was closed because the bot-generated commit was unsigned and lacked a sign-off (failing DCO, signed-commits, pre-commit, and title-and-labels checks). Addresses review feedback on llm-d#567 by preserving the trailing newline at end of file. Signed-off-by: Wen Zhou <wenzhou@redhat.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add RequestQueueName and ResultQueueName fields to AsyncModelConfig so that async queue names are owned end-to-end by the async config section. When omitted, names are derived from InferencePoolName as before (deprecated fallback). Validation rejects partial config (only one of the two queue names set). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Stefano Pascolutti <pascoluttistefano@gmail.com>
Bumps the go-dependencies group with 7 updates: | Package | From | To | | --- | --- | --- | | [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) | `1.43.0` | `1.43.2` | | [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) | `1.32.31` | `1.32.33` | | [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) | `1.19.30` | `1.19.32` | | [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) | `1.22.35` | `1.22.37` | | [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) | `1.106.0` | `1.106.2` | | [github.com/llm-d/llm-d-async/api](https://github.com/llm-d/llm-d-async) | `0.7.4` | `0.9.0` | | [github.com/llm-d/llm-d-async/producer](https://github.com/llm-d/llm-d-async) | `0.7.4` | `0.9.0` | Updates `github.com/aws/aws-sdk-go-v2` from 1.43.0 to 1.43.2 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@v1.43.0...v1.43.2) Updates `github.com/aws/aws-sdk-go-v2/config` from 1.32.31 to 1.32.33 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.32.31...config/v1.32.33) Updates `github.com/aws/aws-sdk-go-v2/credentials` from 1.19.30 to 1.19.32 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@credentials/v1.19.30...credentials/v1.19.32) Updates `github.com/aws/aws-sdk-go-v2/feature/s3/manager` from 1.22.35 to 1.22.37 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.22.35...feature/s3/manager/v1.22.37) Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.106.0 to 1.106.2 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.106.0...service/s3/v1.106.2) Updates `github.com/llm-d/llm-d-async/api` from 0.7.4 to 0.9.0 - [Release notes](https://github.com/llm-d/llm-d-async/releases) - [Changelog](https://github.com/llm-d/llm-d-async/blob/main/RELEASE-NOTES.md) - [Commits](llm-d/llm-d-async@v0.7.4...v0.9.0) Updates `github.com/llm-d/llm-d-async/producer` from 0.7.4 to 0.9.0 - [Release notes](https://github.com/llm-d/llm-d-async/releases) - [Changelog](https://github.com/llm-d/llm-d-async/blob/main/RELEASE-NOTES.md) - [Commits](llm-d/llm-d-async@v0.7.4...v0.9.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-version: 1.43.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.32.33 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-version: 1.19.32 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-version: 1.22.37 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.106.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: go-dependencies - dependency-name: github.com/llm-d/llm-d-async/api dependency-version: 0.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies - dependency-name: github.com/llm-d/llm-d-async/producer dependency-version: 0.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Wen Zhou <wenzhou@redhat.com>
InferenceObjectiveFor checked AsyncDispatchConfig.Models regardless of dispatch mode, so leftover async config would shadow sync gateway objectives after switching modes. Gate the async lookup behind IsAsync(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Stefano Pascolutti <pascoluttistefano@gmail.com>
Signed-off-by: Sudipto Baral <sudiptobaral.me@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Async dispatch mode reused ModelGateways (a sync-oriented struct with HTTP/TLS/retry fields) but only read InferencePoolName. Move async model-to-pool mappings into a dedicated AsyncDispatchConfig.Models field with a purpose-built AsyncModelConfig struct.
InferencePoolName on ModelGatewayConfig is marked as deprecated.
Why is this PR needed?
Async dispatch mode reuses the sync-oriented ModelGateways config section, but only reads InferencePoolName — the other ~12 fields (URL, TLS, retries, API keys) are silently ignored. This is confusing for users configuring async, who can't tell which fields matter. This PR gives async its own config section (async_dispatch.models) where every field is meaningful, making the config self-documenting per dispatch mode.
What does this PR do?
Introduces AsyncModelConfig (with a single InferencePoolName field) and adds a Models map to AsyncDispatchConfig. In async mode, validation and resolution now read from async_dispatch.models instead of model_gateways. InferencePoolName on ModelGatewayConfig is marked as deprecated.
The Helm chart, e2e, and benchmark values files are updated to use the new config structure. Sync mode is unaffected.
How was this tested?
Checklist
git commit -s) per DCOmake ci)make test-e2e)Related Issues
This PR addresses #526, originated from a comment to #458.