feat(processor): tenant-scoped model_gateways lookups (route_key_by_tenant)

[todayim]

Why is this PR needed?

In multi-tenant deployments that share a single batch-apiserver, different tenants may serve models under identical runtime model names (--served-model-name) backed by different inference gateways — e.g. per-InferencePool admission where each tenant's gateway is reached on its own hostname.

Today the processor keys model_gateways lookups solely on the body.model field of each input line. Two tenants advertising the same model name therefore collapse onto a single map entry: the second configuration silently wins, and one tenant's requests are routed to the other tenant's backend. The only workaround is requiring globally-unique model names, which operators cannot enforce when users bring their own serving images.

The apiserver already extracts and persists a tenant ID per job (X-MaaS-Username header by default, remappable via input_headers.tenant), and that tenant ID already reaches the processor's job context — but it is not used for gateway selection.

What does this PR do?

Adds an opt-in processor flag route_key_by_tenant. When enabled, the gateway lookup key becomes <tenantID>/<modelID>:

• With model_gateways entries keyed accordingly (e.g. team-a/qwen3, team-b/qwen3), each tenant's requests route to its own gateway even though the in-request model name is identical.
• The request body is forwarded verbatim — runtimes keep validating the bare served model name, and no response-path translation is needed.
• The inference-objective header lookup follows the same scoped key, so flow-control objectives can also be configured per tenant/model.
• Default false: lookups use the bare model ID exactly as before. Existing deployments and configs are unaffected (the flag is needed because the apiserver always assigns a tenant ID, defaulting to default — an implicit always-on change would silently alter routing for existing users).

Implementation:

• worker.routeKey(byTenant, tenantID, modelID) — single helper used by both lookup sites
• plan-file source (source_planfile.go): resolves RequestItem.ModelID and the objective header via the scoped key; plan grouping and error messages keep the raw model name
• preprocessor (preprocessor.go): per-model registration check uses the scoped key (the job's TenantID is already in scope)
• helm: renders route_key_by_tenant: true into the processor config when processor.config.routeKeyByTenant is set

How was this tested?

• Unit tests added/updated/verified
  • TestRouteKey: disabled / enabled-with-empty-tenant / enabled-scoped cases
  • TestPlanFileSource_Produce_TenantScopedLookup: with the flag on and a team/model gateway entry, produced items carry the scoped ModelID while the body keeps the raw model name and the objective header resolves via the scoped key; the same test re-runs with the flag off (default) to assert the bare-model behavior is preserved
  • Config loading verifies route_key_by_tenant; Helm tests cover both default omission and enabled rendering (99/99 pass)
  • Full go test ./... passes; make test-regression passes; go vet / gofmt clean
  • Note: TestExecuteJob_SLOExpiredDuringDispatch and TestCancelInProgressThrottled flake intermittently on both this branch and clean main (pre-existing timing sensitivity, unrelated to this change — the code paths they exercise do not touch gateway-key resolution)
• Manual testing performed
  • helm template verified rendering of route_key_by_tenant: true and slash-containing model_gateways keys; helm lint passes

Checklist

• Commits are signed off (git commit -s) per DCO
• Code follows project contributing guidelines
• CI checks pass (make ci)
• E2E tests pass (make test-e2e) — not run locally; relies on CI

Related Issues

Tracks #634 — related to multi-tenant operation of a shared batch-apiserver; complements (does not replace) name uniqueness conventions.

[zdtsw]

can we have an issue created to track this change?

[todayim]

Good point — tracking issue created: #634. Linked it to this PR.

[yizhaodev]

So the core requirement here is: route to different InferencePools by tenant + model.

This is an L7 routing problem, so HTTPRoute rules might be a good fit.

I haven't validated this on a real cluster yet, but a practical approach might be:

1) Ensure tenant header is passed through by batch-gateway apiserver:

apiserver:
  config:
    batchAPI:
      passThroughHeaders:
        - Authorization
        - X-MaaS-Username

2) Point processor to a shared inference gateway url.

processor:
  config:
    modelGateways:
      qwen3:
        url: "http://batch-shared-gateway-istio.istio-ingress.svc.cluster.local/llm/qwen3"

3) Use HTTPRoute to dispatch to differnt inference pool by tenant header

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: batch-llm-route
  namespace: llm
spec:
  parentRefs:
    - name: batch-shared-gateway
      namespace: istio-ingress
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /llm/qwen3/v1/chat/completions
          headers:
            - type: Exact
              name: X-MaaS-Username
              value: team-a
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: team-a-pool
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /v1/chat/completions
    - matches:
        - path:
            type: PathPrefix
            value: /llm/qwen3/v1/chat/completions
          headers:
            - type: Exact
              name: X-MaaS-Username
              value: team-b
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: team-b-pool
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /v1/chat/completions

Adding a new tenant then becomes adding a new HTTPRoute match rule. no processor code change, and usually no restart.

[todayim]

Thanks @yizhaodev for the detailed write-up! We verified the pass-through chain end to end (apiserver stores configured headers in job tags with the pth: prefix, and the processor re-injects them on dispatch), so the HTTPRoute approach works today with zero code changes. For deployments where a single shared gateway fronts all pools in the cluster, that's arguably the right shape — routing stays declarative, and adding a tenant is just a new HTTPRoute rule with no processor restart.

The scenario motivating this PR is a different deployment shape, though: single cluster, multi-tenant (multi-project), with per-namespace entry points — each project has its own gateway/hostname, and there is no shared gateway in front of all InferencePools. In that shape:

• Building a cluster-wide shared gateway just for batch dispatch means standing up infrastructure the platform deliberately avoids (single choke point for all tenants' sustained batch throughput, plus a cross-team config ownership split).
• Each tenant's dispatch target is already a distinct URL (the project's own entry), so the model ↔ tenant ↔ URL mapping naturally lives next to model_gateways, rendered by the platform as model deployments come and go.
• With route_key_by_tenant, the processor resolves <tenant>/<model> to that tenant's URL and forwards body.model unchanged — no shared L7 hop, per-tenant failure isolation.

So I'd frame these as complementary patterns for different topologies:

Deployment shape	Recommended pattern
Single shared gateway in cluster	passThroughHeaders + HTTPRoute header rules (your proposal) — no code change
Per-tenant / per-namespace entry points	processor-side tenant-scoped lookup (this PR, opt-in)

Would you be open to keeping the opt-in lookup for the second shape? I'm also happy to add a docs note describing the HTTPRoute pattern as the alternative for single-gateway deployments, so users can pick per topology.

[todayim]

Hi @yizhaodev, following up on the route-key discussion. We confirmed the HTTPRoute approach works for single-gateway clusters, while this tenant-scoped lookup remains needed for per-project/per-namespace entry paths. The PR is CI-green and mergeable; could you please confirm whether you have any remaining concerns or approve the current direction?

[zdtsw]

@todayim need resolve conflict from main branch first ^

[github-actions]

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[todayim]

Resolved by merging current main in dcb039e. The only conflict was the processor chart values block; both the async-dispatch documentation and routeKeyByTenant setting are preserved. Verified with helm lint, 100 Helm chart tests, and processor config Go tests.

[todayim]

Follow-up: the conflict-resolution merge is now DCO signed and cryptographically verified in a07aede. All required checks are green; the PR remains mergeable with the existing approval.

[todayim]

Thanks @zdtsw for updating the branch. The new head ef2704b is mergeable and the update tree matches the locally verified merge (processor tests, helm lint, and 100 chart tests all pass). The required Pre-commit workflow is currently blocked as action_required with no jobs: https://github.com/llm-d/llm-d-batch-gateway/actions/runs/31375249393. Could you approve the fork workflow run when convenient?

[yizhaodev]

the pr may introduce side effect.

Currently the processor assume model ID and endpoint are 1-to-1, it will use model ID as the key for:

1. Plan preprocessing — grouping and prefix-cache-aware sorting
2. AIMD dispatch — per-endpoint concurrent request limiting and backoff
3. Inference endpoint lookup

Now Only 3) was updated to use the tenant-scoped key ("tenant-a/qwen3"), while 1) and 2) still use the bare model ID ("qwen3"). This mismatch means the other model ID based features are silently disabled.

[yizhaodev]

Additionally, consider replacing the boolean route_key_by_tenant with a route_key_method enum for extensibility:

route_key_method: tenant --> <tenant>/<model>

// in the future, for any new requirement, e.g
route_key_method: geography --> <geo>/<model>