[WIP] Prototype/refactor dynamic node params

[yiyixuxu]

along with this PR huggingface/diffusers#12560

This PR is a POC for completely dynamically generating mellon node params based on selected pipeline type. currently they are hardcoded to support multiple pipelines, it will be unmaintinable when we expand support for more pipelines

We already have a dynamic controlnet, I refactored Denoise node in this PR, similar pattern could also be applied to other node types

The code is only a reference implementation, feel free to do what's best to reach the goal.

I made comments next to the code, but here is a summary on how it works

1. User selects a model type in the Load Models node (e.g., "QwenImageEditModularPipeline")
2. Signal propagates to the Denoise node via onChange → onSignal
3. update_node() is called, which:
   • Looks up the pipeline-specific config from QwenImageEdit_NODE_TYPES_PARAMS_MAP
   • Generates appropriate parameters using MellonNodeConfig
   • Calls send_node_definition() to update the UI dynamically

[yiyixuxu]

I moved ModularMellonNodeRegistry from diffusers to mellon and so this registry gets absorbed into that https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/mellon_node_utils.py#L703

[yiyixuxu]

we should config params for each model_type we support here, this one is for QwenImage-Edit

[yiyixuxu]

Each key represents a node type in the Mellon UI. Setting a value to None (like controlnet) indicates that pipeline doesn't support that node type.

QwenImageEdit_NODE_TYPES_PARAMS_MAP = {
    "controlnet": None,  # No ControlNet support for Qwen Edit
    "denoise": {...},
    "vae_encoder": {...},
    "text_encoder": {...},
    "decoder": {...},
}

[yiyixuxu]

The "block_names": ["denoise"] field maps to the actual pipeline block in the modular pipeline. For Qwen Edit, this corresponds to the "denoise" block in EDIT_BLOCKS:

EDIT_BLOCKS = InsertableDict([
    ("text_encoder", QwenImageEditVLEncoderStep()),
    ("vae_encoder", QwenImageEditVaeEncoderStep()),
    ("input", QwenImageEditInputStep()),
    ("prepare_latents", QwenImagePrepareLatentsStep()),
    ("set_timesteps", QwenImageSetTimestepsStep()),
    ("prepare_rope_inputs", QwenImageEditRoPEInputsStep()),
    ("denoise", QwenImageEditDenoiseStep()),  # ← Maps here
    ("decode", QwenImageDecodeStep()),
])

[yiyixuxu]

for each node_type, you have to config inputs/model_inputs/outputs/block_name

inputs/model_inputs/outputs will be used to compose our final params. I structured this way to mirror the modular pipeline structure (we have inputs/components/outputs) -> this way in the future we can support direct conversion between mellon nodes defination <-> modular pipeline blocks definition

[yiyixuxu]

this map is used together with a common schema we maintain, currently hosted here https://github.com/huggingface/diffusers/blob/main/src/diffusers/modular_pipelines/mellon_node_utils.py#L29

if a parameter is defined as a string like "embedding" here, we will feetch the UI configuration from the common schema

"embeddings": {
    "label": "Text Embeddings",
    "display": "input",
    "type": "embeddings",
}

This is automatically resolved at runtime without needing explicit configuration in the node map.

[yiyixuxu]

you can also customize a parameter using MellonParam, e.g. here we override the definition for image_latents

the common one is this

"image_latents": {
    "label": "Image Latents",
    "type": "latents",
    "display": "input",
    "onChange": {False: ["height", "width"], True: ["strength"]},  # ← Shows strength when connected
}

but since qwen-image-edit does not need strength, we can don;t need the onChange part

[yiyixuxu]

also you may notice we did not list height/width here for qwen-edit since we don't want user to set custom size for qwen-edit,
but these two parameters are listed for qwen-image

[yiyixuxu]

I'm not able to use a variable for data, here, I created a function to dynamically create this map but will run into an issue if I put the function here, we need to not have to hard-code this

get_model_type_signal_data()

[asomoza]

this is one of the petitions I made about the signals, we should be able to do it soon.

[yiyixuxu]

we should refactor the execute method too, to take kwarggs, so that we will be parameters dynamically defined in param, similar to how we implemented DynamicCOntrolnet, but that will be next step and i will help

[yiyixuxu]

Denoise node must have a unet input to receive signals, the rest should all be dynamic