feat(examples): compare workflow and bespoke vision ensemble

[furionw]

Based on #13293.

Why

Dynamo endpoints already let applications build this ensemble manually, so the workflow API should demonstrate its value against an equivalent baseline. Both variants share the inline components, stock vLLM worker, and the same GenerateEndpointInvoker for request validation, stream folding, and remote-call cancellation. This isolates the comparison to orchestration ownership: the workflow runtime owns graph scheduling and binding, while the bespoke handler owns coroutine and sibling-task lifecycle.

Effect

                  ┌─> inline classifier ────────────┐
request → encoder ┤                                 ├→ inline merge → chunk
                  └─> request adapter → stock vLLM ─┘

What Change

• Add declarative and bespoke mixed-placement vision ensembles.
• Share stages, launch topology, and Generate endpoint invocation.
• Keep scheduling and sibling lifecycle as the comparison.

classDiagram
    WorkflowOrchestrator *-- StageDispatcher : schedules graph
    StageDispatcher ..> GenerateEndpointInvoker : invokes remote Generate
    BespokeEnsembleHandler ..> GenerateEndpointInvoker : invokes same adapter
    GenerateEndpointInvoker ..> DynamoClient : calls discovered endpoint
    BespokeEnsembleHandler --> EncoderStage : schedules inline

    note for WorkflowOrchestrator "Owns authored graph execution"
    note for BespokeEnsembleHandler "Owns equivalent task orchestration"
    note for GenerateEndpointInvoker "Shares validation, folding, and call cancellation"

Loading

Test Plan

• Generate adapter and bespoke orchestration CPU behavior pass.
• Targeted pre-commit hooks pass.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a03f0d-3097-721a-997f-fb523eb25624.docs.buildwithfern.com/dynamo/dev

[caronzh03]

Does user have to author this worker.py for every single ensemble pipeline?
If so, this file seems to contain a lot of boilerplate code.
It might be a cleaner interface if a user only needs to author L87-L105 (the top portion of this start() function) to register the custom stages and workflow.

[furionw]

makes sense. will take this into consideration

[caronzh03]

What does this adapter do?

[furionw]

given an output of encoder, users may need to pass different kinds of Prompt into engine (vllm in this case).

e.g.

• it may be a EmbedPrompt for LLM model
• it may be a TokenPrompt for VLM model (because Qwen VL requires grid_thw alongside embedding)

I am seeing different cases where one of these prompts is needed.

[caronzh03]

If I'm not mistaken, this function is the post-processing step that stitches multiple outputs from different stages together, correct?

Even though this is pure Python logic without ML, would it be OK to model this step as a stage as well? That way, nothing lives outside of the workflow.

[furionw]

correct. it's totally doable to model it inside stage.

[caronzh03]

From user-interface perspective, the workflow approach seems cleaner, assuming this doesn't require a custom WorkflowEndpointHandler implementation like BespokeEnsembleHandler.

From functionality perspective, I cannot tell whether either approach can support remote stage calling in the future. For example, in bespoke approach, does the classifier_task have to be a local, in-process call? In workflow approach, does classifier stage have to be a local, in-process call?

I'm wondering if we can fuse the two approaches into one:

• Let user-interface use the workflow approach - define graph topology without asyncio.create_task() boilerplate.
• Let the Dynamo-provided WorkflowEndpointHandler be the default handler for workflow execution, and if users want to customize the execution logic, like adding prioritization or throttling, they can override WorkflowEndpointHandler, and at that point, the custom handler will look something like BespokeEnsembleHandler.

[caronzh03]

Where is WorkflowEndpointHandler defined?
Does user have to implement custom WorkflowEndpointHandler for each workflow?

[furionw]

Where is WorkflowEndpointHandler defined?

#13083

Does user have to implement custom WorkflowEndpointHandler for each workflow?

No, WorkflowEndpointHandler kicks off orchestrator which executes users' workflow.

[furionw]

From user-interface perspective, the workflow approach seems cleaner, assuming this doesn't require a custom WorkflowEndpointHandler implementation like BespokeEnsembleHandler.

the assumption is correct

Inline- and Remote-

From functionality perspective, I cannot tell whether either approach can support remote stage calling in the future. For example, in bespoke approach, does the classifier_task have to be a local, in-process call? In workflow approach, does classifier stage have to be a local, in-process call?

Existing Dynamo primitives already support it remote stage calling. both bespoke and workflow approach, can support remote stage calling.

Dynamo Primitive

tl;dr users' Python can

• import Dynamo's runtime class, call runtime.endpoint("demo.backend.generate").serve_endpoint(my_function)
• on client side, (await runtime.endpoint("demo.backend.generate").client()).generate(request)

Workflow (inline- and remote- switch)

the workflow abstract the above logic so that users only need to

• define a stage (input, output and run function)
• specify whether they want in-process or separate-process (e.g. remote): InlineBinding, RemoteBinding(endpoint_id)

This would allow users to switch between inline- and remote- easily. Does this abstraction make things more complicated than helpful?

// This PR
orchestrator = await WorkflowOrchestrator.bind(
    define_workflow(),
    bindings={
        "encoder": InlineBinding(encoder),
        "generator": GenerateEndpointBinding(GENERATOR_ENDPOINT),
        ...
    },
    runtime=runtime,
)

// hello-world example 
// https://github.com/ai-dynamo/dynamo/pull/13423/changes
orchestraor = WorkflowOrchestrator.bind(
    define_workflow(),
    bindings={
        stage_id: RemoteBinding(endpoint_id)
        for stage_id, endpoint_id in ENDPOINTS.items()
    },
    runtime=runtime,
)

[furionw]

I'm wondering if we can fuse the two approaches into one:>

• Let user-interface use the workflow approach - define graph topology without asyncio.create_task() boilerplate.
• Let the Dynamo-provided WorkflowEndpointHandler be the default handler for workflow execution, and if users want to customize the execution logic, like adding prioritization or throttling, they can override WorkflowEndpointHandler, and at that point, the custom handler will look something like BespokeEnsembleHandler.

agreed.

• there will be a point, where users need to customize beyond a static graph
  • we have users doing that today based on dynamo's endpoint primitives)
  • from workflow point of view, we can support Triton-BLS-like (business logic scripting) capabilities (loop, conditional, etc.) -- 4. Imperative authoring
    
• dynamo has been supporting prioritization, requests admission (a form of throttling). If we proceed with workflow approach, Dynamo can also provide these optional utilities at stage level.

[caronzh03]

Does this abstraction make things more complicated than helpful?

The Workflow abstraction hides the execution details/complexity, giving users a clean way to define graph topology; it also decouples graph definition and instance binding nicely - reminding me of "dependency injection" design pattern in Java world, so I would definitely prefer using the WF approach.
In the bespoke approach, the graph topology definition and execution happens together in the Handler class. Perhaps if the graph is simple, there's not much difference in terms of maintenance effort, but if the graph becomes complex, and with remote calls, the code could be un-readable.