feat: run sub recipe multiple times in parallel (Experimental feature in CLI)

[lifeizhou-ap]

Adds ability to run sub-recipes multiple times with parallel execution, real-time progress tracking.

Key Features:

• Parallel Execution of sub-recipes: Run multiple sub-recipe instances concurrently with configurable worker pools.
  • You can run different sub-recipes in parallel, as long as you mentioned in your prompt. For example: analyze-pagerduty-incident and analyze-service-issues in parallel
  • You can also run the same sub-recipe in parallel. For example: get the weather for 3 largest cities in Australia in parallel
• Real-time Dashboard: Live progress tracking with task statistics and status updates

Changes:

• Add executions and sequential_when_repeated in the sub_recipe section

  ```
  sub_recipes:
  - name: weather
    path: "{{recipe_dir}}/weather.yaml"
    values:
      date: "2025-01-01"
   sequential_when_repeated: true
  ```
  

  • values section: always use the data in this section when running sub-recipe
  • sequential_when_repeated: whether it should be sequential when running this sub recipe with different parameters. optional field, default is false
    • If sequential_when_repeated: true means the sub-recipe should NOT run in parallel with different params even user requested in prompts.
    • If sequential_when_repeated is false, or not specified, it is by default to run parallel unless user requested to run sequentially in prompts

• New notification Event (notification_events.rs) for real-time task updates

• Enhanced executor engine with parallel worker support and error handling

• Console UI components for task execution dashboard (task_execution_display/)

How it works

1. Input Phase

• User provides recipe with multiple sub-recipe executions
• Recipe parser creates individual tasks from the input
• Tasks are queued for parallel execution

2. Execution Phase

• Executor spawns configurable number of workers
• Workers pull tasks from queue and execute sub-recipes concurrently
• Each worker runs independently with its own sub-recipe instance

3. Monitoring Phase

• Task Execution Tracker monitors all worker progress
• Real-time events are generated for status changes
• Statistics are calculated and updated continuously

4. UI Updates

• Notification events stream to the dashboard
• Real-time display shows: completed, running, failed, pending counts
• Users see live progress without polling

5. Error Handling

• Failed tasks are captured with detailed error context

[lifeizhou-ap]

I wonder if it would be even more powerful if we could somehow move the recipe path into executions. Currently this PR allows us to run one recipe with different params multiple times in parallel. But what if we had a predefined collection of recipes that we wanted to all run in parallel (effectively as one tool for the agent). That could be powerful.

Something like:

sub_recipes:
- name: code_review
  values:
    severity_level: "high"
  executions:
    parallel: true
    runs:
    - path: "{{recipe_dir}}/lint.yaml"
      values:
        tool: "clippy"
    - path: "{{recipe_dir}}/sql-bugs.yaml" 
      values:
        db: "postgres"

I feel this is like an inline subrecipe with subrecipe :) (ie. nested subrecipe)

Currently subrecipe block maps to one recipe. and the values are only for the subrecipe

values:
>     severity_level: "high"

This part will be a little bit confusing with this block. This requires both lint.yaml and sql-bugs.yaml recipes have the same params severity_level.

In the main branch, the parallel has been implemented for different recipes as below

sub_recipes:

• name: lint
  path: "{{recipe_dir}}/lint.yaml"
  values:
  severity_level: "high"
  . tool: "clippy"
• name: sql-bugs
  path: "{{recipe_dir}}/sql-bugs.yaml"
  values:
  severity_level: "high"
  . db: "postgres"

[lifeizhou-ap]

There is also another perspective to take on this. You could argue that the agent can already choose to run multiple sub_recipes in parallel (and they are getting better at understanding prompts) so we should not add code complexity and schema rigidity in this PR.

Instead we should rely entirely on the llm/agent to follow prompts and run tools in the correct order/parallelism natively.

There is the danger here that we are getting very close to building a limited version of prescribed workflows (particularly if we go this route). If we do decide we want to be able to program full on workflows then we would be better off building dedicated workflow logic outside of recipes and have recipes just as the agent node (specification) building block.

If we go ahead with this functionality for recipes as is I think we should mark it as 'experimental' so that its more feasible to remove if we discover it doesn't align well with goose future recipe directions.

The schema level parallelism is an optional field for user to enforce the deterministic flow. I guess you are concerning the field has been defined in the recipe schema for now, and later on it is hard to remove it if we don't need it?

Marking this as experimental makes sense if we proceed — it'll give us room to iterate or revert if needed.

I’m on board with marking it experimental if we decide to include it, but I’m also up for talking more about where we see this going long term

[jsibbison-square]

Am I understanding correctly that this new executions and runs section of recipe is more like a tool 'prompt' influence for the agent? The only thing that's enforced is the number of params and maybe number of times the recipe will be called (runs).

If I understand right (I'm new to sub-agent code 😬 ) but it looks like the agent first calls a tool to 'create sub recipe tasks' and then with the response it is supposed to call a 'execute tasks' tool. But ultimately if the agent decides it wants to run the recipes sequential instead of parallel or run less/more subrecipes or override the params itself it total can because the there is not a lot of enforcement to follow the subrecipe config. It mostly generates prompt engineering for the agent (apart from the value defaults we provide).

If this is the case that these new schema fields are mostly optional guidance that the agent can potentially ignore and do differently then I think this is a lot of extra complexity and schema changes are painful to revert.

Sub_recipes can already be run in parallel with different parameters without this PR so I think we should rely on standard prompt engineering in the prompt/system prompt rather than trying to use this schema as hints for the agent.

I think there a lot of other useful things in this PR around subtask rendering in the cli and probably other refactoring but I think we should not merge the recipe schema changes 🙅 . But happy to chat and hear other opinions 😃 .

[michaelneale]

conceptually - would you describe sub recipes as a way to do "sub agents" (just as that seems to be the topic of the day - ie many agents working together, would that be one way to describe this to users not familiar with goose terminology?)

[lifeizhou-ap]

conceptually - would you describe sub recipes as a way to do "sub agents" (just as that seems to be the topic of the day - ie many agents working together, would that be one way to describe this to users not familiar with goose terminology?)

Examples of use case

• In the mono repo build, there are 3 services that failed the build. I can find that 3 build urls and use a sub-recipe (diagnose failure sub recipe) by passing the build url to diagnose these 3 service in parallel.
• I have a list of document link in a csv file, I would like to summarise each document (summarise document sub recipe)

[jamadeo]

My main feedback is around the need for all the flexibility of parallel vs sequential execution within a task. IMO the model should not decide how that works -- it's just getting a batch of results anyway. Sub-recipes setting a flag to specify their execution when they get a batch of parameters seems OK.

[jamadeo]

Yeah you could probably leave it out -- the sub recipes themselves will be calling tools, and they'll have their own timeouts.

One thing we definitely don't want here is a default 5-minute timeout. IMO that doesn't make much sense for tool calls in general, but that's a separate issue.

[jamadeo]

probably should rename this without the underscore since it's used now

[jamadeo]

should this be an enum?

[lifeizhou-ap]

good catch, i thought I have refactored, but missed out

[jamadeo]

I know it's not in this PR but this should probably handle an Err some other way than .unwrap()

[jamadeo]

IMO the model shouldn't pick how the executor runs a set of tasks. The model will always get the results back together, so it only impacts the implementation details of the tasks themselves. If the model decides it needs the result of one task to run the next, it will have to separate them anyway.

[jamadeo]

from earlier discussion -- consider whether you can just have the model pass the ID and leave it at that, since the task has already been "created"

[jamadeo]

It still seems like we're going down a pretty complex road here, moving away from how models would typically call tools, but I think that conclusion should really come from experimentation. As long as we're still calling this experimental let's get it merged and keep trying things.

[jamadeo]

This property name is a little confusing IMO, maybe use something like allow_concurrent?

[lifeizhou-ap]

Hi @jamadeo

Thank you for the review and your time!

I thought of allow_concurrent, and i feel that it might have 2 concerns:

1. allow_concurrent: false it might confuse people that the sub-recipe cannot be running concurrently with other sub-recipes. The ...when_repeated makes it clear that it cannot be run parallel when running repeatedly.

2. By default the sub-recipe can be run in parallel. Usually if the boolean attribute is not specified, the default value is false. If allow_concurrent is specified, people might be think the allow_concurrent: false

Naming is usually hard. Happy to change to a better name.