refactor: restructure async config with explicit queue names (#526)

[pazqo]

Async dispatch mode reused ModelGateways (a sync-oriented struct with HTTP/TLS/retry fields) but only read InferencePoolName. Move async model-to-pool mappings into a dedicated AsyncDispatchConfig.Models field with a purpose-built AsyncModelConfig struct.

InferencePoolName on ModelGatewayConfig is marked as deprecated.

Why is this PR needed?

Async dispatch mode reuses the sync-oriented ModelGateways config section, but only reads InferencePoolName — the other ~12 fields (URL, TLS, retries, API keys) are silently ignored. This is confusing for users configuring async, who can't tell which fields matter. This PR gives async its own config section (async_dispatch.models) where every field is meaningful, making the config self-documenting per dispatch mode.

What does this PR do?

Introduces AsyncModelConfig (with a single InferencePoolName field) and adds a Models map to AsyncDispatchConfig. In async mode, validation and resolution now read from async_dispatch.models instead of model_gateways. InferencePoolName on ModelGatewayConfig is marked as deprecated.

Additionally, adds InferenceObjective to AsyncModelConfig so the async config owns objective routing end-to-end (previously only available via the sync-oriented ModelGatewayConfig).

Adds explicit RequestQueueName and ResultQueueName fields to AsyncModelConfig, making queue naming a first-class config concern rather than a hidden convention. When omitted, queue names are still derived from InferencePoolName as a deprecated fallback. Validation rejects partial config (only one of the two set).

The Helm chart, e2e, and benchmark values files are updated to use the new config structure. Sync mode is unaffected.

How was this tested?

• Unit tests added/updated/verified
• Integration/e2e tests added/updated/verified
• Manual testing performed

Checklist

• Commits are signed off (git commit -s) per DCO
• Code follows project contributing guidelines
• CI checks pass (make ci)
• E2E tests pass (make test-e2e) (also async)

Related Issues

This PR addresses #526, originated from a comment to #458.

[github-actions]

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[evacchi]

looks fine to me, the async dispatcher is a new component so if we land this soon we shouldn't break anything.

FYI @vishbhat @j-mok-dev

let @pazqo know if you think we should a backwards-compat path too

[yizhaodev]

looks fine to me, the async dispatcher is a new component so if we land this soon we shouldn't break anything.

FYI @vishbhat @j-mok-dev

let @pazqo know if you think we should a backwards-compat path too

i think inferenceObjective is also needed per model

[pazqo]

looks fine to me, the async dispatcher is a new component so if we land this soon we shouldn't break anything.
FYI @vishbhat @j-mok-dev
let @pazqo know if you think we should a backwards-compat path too

i think inferenceObjective is also needed per model

Nice catch, adding it now

[yizhaodev]

I'd prefer keeping modelGateways for async mode, it already holds both inferencePoolName and inferenceObjective, and the current PR breaks InferenceObjectiveFor() by leaving inferenceObjective orphaned.

If we do want to move async model config into asyncDispatch.models, then consider to go further: use explicit queue names instead of having the code concatenate llm-d-async:requests: + poolName. This removes the implicit naming convention between batch-gateway and async-processor, and gives operators full control over queue names without code changes:

 processor:
   config:
     dispatchMode: async
     asyncDispatch:
       resultPollTimeout: "30s"
       models:
         sim-model:
           requestQueue: "llm-d-async:requests:sim-pool"
           resultQueue: "llm-d-async:results:sim-pool"
           inferenceObjective: "obj"

@evacchi

[evacchi]

@yizhaodev this originated by the comment at #458 (comment) I am not particularly opinionated about this, but I feel like the config would indeed need some restructuring. You can feel free to propose an alternative!

[pazqo]

@yizhaodev I added inferenceObjective, so that part should be fine.
Since we wanted to move to async (I do not have a strong feeling, but I agree that current version is non-ideal), do you think we need to also use explicit queue name as part of this PR? If so, I can make a proposal for that. Otherwise we can probably move on with this PR and prepare a follow up (provided that there are not other breaking issues)

[pazqo]

Maybe we can allow the user to specify the queue name, but we need to keep this auto-generate version as well until all the rest of the configs are migrated.

[yizhaodev]

@yizhaodev I added inferenceObjective, so that part should be fine. Since we wanted to move to async (I do not have a strong feeling, but I agree that current version is non-ideal), do you think we need to also use explicit queue name as part of this PR? If so, I can make a proposal for that. Otherwise we can probably move on with this PR and prepare a follow up (provided that there are not other breaking issues)

@lioraron @j-mok-dev
any suggestion here?

[j-mok-dev]

I’m leaning toward doing the fuller cleanup in this PR rather than stopping at a partial move.
If we are introducing an async-specific config section, I think it would be cleaner for it to own the async-specific concerns end-to-end, instead of splitting them between asyncDispatch.models and modelGateways (or implicit queue naming rules in code). Otherwise, we improve the shape of the config, but still leave part of the async behavior hidden in conventions.
So my preference would be:

• keep inferenceObjective with the async model config
• consider making request/result queue names explicit as part of the async config as well
• if backward compatibility is needed, support the old derived naming as a temporary fallback, but make the new structure the clear source of truth

@lioraron what do you think?

[pazqo]

I think keeping the old naming as a fallback is important in during this refactoring, at least to document the previous way of naming queues. We can also use comments, but the fallback (deprecated) is probably more robust.

[pazqo]

@yizhaodev @j-mok-dev @gkneighb I added explicit queue names as first-class async config fields and kept the fallback from InferencePoolName (marked as deprecated).

Please let me know if you think there is more work expected on this PR or if we can move forward, thanks!

[yizhaodev]

it looks good to me overral
but i think you need to update benchmarks, examples/deploy-demo, scripts/dev-deploy-dispatcher.sh and operator CRD in the following

[zdtsw]

with this change i assume we will need some sort of release notes to cover the changes, even we keep old still valid and probably do a full cleanup in 2-3 release once deprecation preiod is done