Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a drag and drop team builder interface for AGS #4202

Closed
2 tasks done
Tracked by #4006
victordibia opened this issue Nov 14, 2024 · 5 comments · Fixed by #4609
Closed
2 tasks done
Tracked by #4006

Implement a drag and drop team builder interface for AGS #4202

victordibia opened this issue Nov 14, 2024 · 5 comments · Fixed by #4609
Assignees
Labels
needs-design A design needs to be created and agreed upo proj-studio Related to AutoGen Studio. size-medium takes up to a week
Milestone

Comments

@victordibia
Copy link
Collaborator

victordibia commented Nov 14, 2024

The current rewrite of AutoGen studio has focused on representing teams declaratively in JSON.
It would be great to add a flexible GUI to let users modify team configurations. A team is made of up compontents - agents (models, tools, etc), termination conditions etc.

  • Visualize team config (what are the agents, what are the tools, models etc)
  • Add/remove components (agents, teams, models) either directly in JSON or via UI

Component Serialization Decisions

Image

The team (which defines the main workflow/orchestration behaviors) is the primary component in AGS.

There are two potential approaches for storing declarative team (and other component) specifications in the same internal ags database:

Fully Linked Components

In this approach, each component (model, agent, team, memory etc) maintains relationships at the database level. For example, an Agent table row would have foreign keys to its models and tools. Fetching a team's complete configuration requires traversing these relationships to assemble the full picture.

Pros:

  • Components stay synchronized - when a model or tool is updated, all agents using it automatically see the change
  • Referential integrity is enforced by the database
  • Storage efficiency since components are stored once and referenced many times

Cons:

  • Complex fetching logic requiring multiple joins to assemble full configurations
  • Updates must cascade through relationships
  • Schema changes needed to support new component types (a deployment challenge each time AGS users update their app)
  • Harder to version control since components can change underneath a team
  • More complex deployment as schema updates needed for new features

Independent Components

This approach serializes the complete configuration of each component. Rather than maintaining relationships, a team's configuration includes the full definition of all its agents, including their tools, models, and other settings. When you store a team using the weather_api tool, that tool's entire configuration is part of the team definition.

Pros:

  • Configurations are self-contained and portable
  • Simple updates - just modify a single record, test it and reuse elsewhere (helps with trust)
  • Easy versioning since each team captures a complete snapshot
  • Schema flexibility - new component types don't require database changes
  • Simpler deployment with less database coordination

Cons:

  • Storage duplication when components are reused across teams (not really an issue, configurations are very light).
  • No automatic propagation of component updates (we can mitigate this by providing UX that helps users easily validate/test all components to build trust)
  • UX needs to clearly communicate the independence of configurations
  • Potential confusion if users expect linked behavior (UX can help here)

Key Decision Factors

The choice depends heavily on use case. For experimentation and reproducibility, Independent Components shines - each team captures a precise snapshot of its configuration at a point in time. For production systems prioritizing consistency, Fully Linked Components ensures changes propagate automatically.

A hybrid approach storing both relationships and snapshots is possible but adds complexity. The benefits of Independent Components - reproducibility, schema flexibility, and configuration portability - likely outweigh the storage overhead from duplication for most use cases.

Happy to get thoughts here.
@husseinmozannar , @ekzhu , @JingyaChen

@ekzhu
Copy link
Collaborator

ekzhu commented Nov 16, 2024

Thanks @victordibia for the analysis.

Thoughts:

  1. Storage cost decreases faster than compute, so storage overhead is not an issue.
  2. Important to associate class version number (e.g., a hash code) with each serialized component, and ability to retrieve the provider for each version. When an irreversible breaking change in the underlying agent/tool/team class happens in the future we can use the version number to resolve or migrate the stored component to the latest version.
  3. A provider class maps (serialized component, version) -> (component). On store, always migrate to the latest version.
  4. Independent component with loose, un-linked, key-value like schema is good to start. Otherwise, the overhead of schema migration will be burdensome when code changes frequently.

@ekzhu
Copy link
Collaborator

ekzhu commented Nov 16, 2024

I think we also want to separate these two concepts for component:

  1. Component definition: a JSON serialized config that can be used to create a new component instance.
  2. Component state: a JSON serialized state that can be used to bring a component back to an existing state.

Separating these two allows us to ease the breaking changes: if a change impacts just the definition, we don't need to worry about also updating the state, and vice versa. This also allows us to run analytics on the state without the analytic code to reconstruct the components, as some components can be heavy and doing so consumes a lot of compute.

@ekzhu
Copy link
Collaborator

ekzhu commented Nov 16, 2024

More regarding the state.

I think in the beginning we might want to consider supporting only "ephemeral" components: that is their states exists in memory only. We still store their definitions. So, user can retrieve their designed components, but every time they do the components are all new.

We can just have an export function to export all the state in JSON, if user want to see. But no import to start.

@victordibia
Copy link
Collaborator Author

Good ideas and points @ekzhu

  • Agreed that we should start with independent components each with a version number. Current spec has version numbers
  • Agree that state is different from definition/spec. The current work in AGS has focused on definition/spec. Support the ability to to save and review "sessions" in AGS #4222 will look at how best to persist exchanged messages. If we do this right, the structure of state should not be far from the actual messages exchanged. This should allow us some lattitude to "restore" agent state by loading messages they sent previously?

@ekzhu
Copy link
Collaborator

ekzhu commented Nov 16, 2024

Current spec has version numbers

👍 👍

This should allow us some lattitude to "restore" agent state by loading messages they sent previously?

Yep. I think that's right.

@victordibia victordibia added the needs-design A design needs to be created and agreed upo label Nov 16, 2024
@fniedtner fniedtner modified the milestones: 0.4.0, 0.4.1 Nov 25, 2024
@victordibia victordibia added the size-medium takes up to a week label Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-design A design needs to be created and agreed upo proj-studio Related to AutoGen Studio. size-medium takes up to a week
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants