diff --git a/.gitignore b/.gitignore index a4c65e95d9..1ac4f8ed5f 100644 --- a/.gitignore +++ b/.gitignore @@ -93,9 +93,6 @@ web/src/api/types/generated.d.ts # (hand-off artefact for issue #1417; regenerate locally as needed). web/src/i18n/_extracted_catalog.json -# Generated comparison page (built by scripts/generate_comparison.py) -docs/reference/comparison.md - # Astro / Node.js (landing page) site/node_modules/ site/dist/ diff --git a/CLAUDE.md b/CLAUDE.md index 043c7a2380..e0822014a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -210,7 +210,7 @@ When tests fail due to timeout, slowness, or xdist resource contention: - **Timeout**: 30 seconds per test (global in `pyproject.toml`; do not add per-file `pytest.mark.timeout(30)` markers; non-default overrides like `timeout(60)` are allowed) - **Parallelism**: `pytest-xdist` via `-n 8`. **ALWAYS** include `-n 8` when running pytest locally, never run tests sequentially. CI uses `-n auto` (fewer cores on runners). - **Parametrize**: Prefer `@pytest.mark.parametrize` for testing similar cases -- **Vendor-agnostic everywhere**: NEVER use real vendor names (Anthropic, OpenAI, Claude, GPT, etc.) in project-owned code, docstrings, comments, tests, or config examples. Use generic names: `example-provider`, `example-large-001`, `example-medium-001`, `example-small-001`, `large`/`medium`/`small` as aliases. Vendor names may only appear in: (1) Operations design page provider list (`docs/design/operations.md`), (2) `.claude/` skill/agent files, (3) third-party import paths/module names (e.g. `litellm.types.llms.openai`), (4) provider presets (`src/synthorg/providers/presets.py`) which are user-facing runtime data, (5) provider logo assets (`web/public/provider-logos/*.svg` plus the matching README) -- the SVG filenames mirror the preset names by necessity, and `` falls back to a generic icon if a file is absent. Tests must use `test-provider`, `test-small-001`, etc. +- **Vendor-agnostic everywhere**: NEVER use real vendor names (Anthropic, OpenAI, Claude, GPT, etc.) in project-owned code, docstrings, comments, tests, or config examples. Use generic names: `example-provider`, `example-large-001`, `example-medium-001`, `example-small-001`, `large`/`medium`/`small` as aliases. Vendor names may only appear in: (1) `.claude/` skill/agent files, (2) third-party import paths/module names (e.g. `litellm.types.llms.openai`), (3) provider presets (`src/synthorg/providers/presets.py`) which are user-facing runtime data, (4) provider logo assets (`web/public/provider-logos/*.svg` plus the matching README) -- the SVG filenames mirror the preset names by necessity, and `` falls back to a generic icon if a file is absent. Tests must use `test-provider`, `test-small-001`, etc. - **Property-based testing**: Python uses [Hypothesis](https://hypothesis.readthedocs.io/), React uses [fast-check](https://fast-check.dev/), Go uses `testing.F` fuzz functions. CI runs 10 deterministic examples per property test (`derandomize=True`, no flakes). When Hypothesis finds a failure, it is a **real bug**: read the shrunk example, fix the underlying bug, and add an explicit `@example(...)` decorator so the case is permanently covered. See [docs/reference/claude-reference.md](docs/reference/claude-reference.md) §"Property-based Testing (Hypothesis): Deep Dive" for profile catalog, local fuzzing commands, and failure-handling workflow. - **Flaky tests**: NEVER skip, dismiss, or ignore flaky tests; always fix them fully and fundamentally. For timing-sensitive tests, the **FakeClock-first** rule applies: if the class under test accepts a `clock=` parameter, inject `FakeClock` from `tests._shared.fake_clock` and drive virtual time via `clock.advance(...)` / `await clock.advance_async(...)`. Patch `time.monotonic()` / `asyncio.sleep()` at the module level only for legacy code paths that don't yet have a `Clock` seam. For tasks that must block indefinitely until cancelled (e.g. simulating a slow provider or stubborn coroutine), use `asyncio.Event().wait()` instead of `asyncio.sleep(large_number)`; it is cancellation-safe and carries no timing assumptions. diff --git a/README.md b/README.md index f3b019b71b..baf667e93c 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ SynthOrg is a Python framework for building **synthetic organizations**, autonom Define your company in YAML. Agents collaborate through a message bus, follow workflows (Kanban, Agile sprints, or custom), track costs against budgets, and produce real artifacts. The framework is provider-agnostic (100+ LLMs via [LiteLLM](https://github.com/BerriAI/litellm)), configuration-driven ([Pydantic v2](https://docs.pydantic.dev/) models), and designed for the full autonomy spectrum, from human approval on every action to fully autonomous operation. -> **Early access.** Core subsystems are built and tested (25,000+ unit tests, 80%+ coverage). APIs may change between releases. See the [roadmap](https://synthorg.io/docs/roadmap/) for what's next. +> **Early access.** Core subsystems are built and tested (27,000+ tests, 80%+ coverage). APIs may change between releases. See the [roadmap](https://synthorg.io/docs/roadmap/) for what's next. ## Why SynthOrg? diff --git a/docs/DESIGN_SPEC.md b/docs/DESIGN_SPEC.md index 6317255e03..f1f2a465ed 100644 --- a/docs/DESIGN_SPEC.md +++ b/docs/DESIGN_SPEC.md @@ -43,6 +43,7 @@ The design specification has been split into focused documentation pages for bet | [Client Simulation](design/client-simulation.md) | Client Types, Intake, Review Pipeline, Simulation | Synthetic client framework for workload generation and evaluation | | [Strategy & Trendslop Mitigation](design/strategy.md) | Lenses, Principles, Confidence, Impact | Anti-trendslop mitigation for strategic agents | | [Self-Improvement](design/self-improvement.md) | Meta-Loop, Signals, Rules, Proposals, Rollout | Self-improving company: signal aggregation, rule engine, improvement proposals, staged rollout | +| [Internationalization](design/internationalization.md) | Locale Resolution, UI Text, Translation Scope | British English UI default, locale-aware Intl-based display formatting, no planned translation framework | ## Supporting Pages diff --git a/docs/architecture/decisions.md b/docs/architecture/decisions.md index 7166415914..d2c19058ba 100644 --- a/docs/architecture/decisions.md +++ b/docs/architecture/decisions.md @@ -61,7 +61,7 @@ All significant design and architecture decisions in force today, organized by d | ID | Decision | Rationale | Alternatives considered | |----|----------|-----------|------------------------| -| D16 | Docker MVP via `aiodocker`; `SandboxBackend` protocol for future backends | Docker cold start (1-2s) invisible against LLM latency (2-30s). Pre-built image + user config. Fail if Docker unavailable; no unsafe subprocess fallback. gVisor as config-level hardening upgrade | Docker + WASM (CPython can't run pip packages in WASM), Docker + Firecracker (Linux-only, requires KVM), docker-py (sync, no 3.14 support). Precedents: E2B, major cloud providers, Daytona (none offer unsandboxed fallback) | +| D16 | Layered `SandboxBackend` protocol via `aiodocker`. Subprocess default for low-risk categories (file_system, git); Docker required for high-risk (code_execution, terminal, database, web) | Subprocess is genuinely safe for read-only / workspace-scoped categories: env filtering (allowlist + denylist), restricted PATH, workspace-scoped cwd, timeout + process-group kill, library-injection-var blocking. Docker is required where arbitrary code or network egress can land. Layered design preserves the local-first quickstart (file/git tools work without Docker) without weakening isolation where it matters. Docker cold start (1-2s) is invisible against LLM latency (2-30s). gVisor remains a config-level hardening upgrade for the Docker tier | Docker + WASM (CPython can't run pip packages in WASM), Docker + Firecracker (Linux-only, requires KVM), docker-py (sync, no 3.14 support), Docker-only for every category (rejected: forces a running container for trivial file reads, breaks local-first quickstart, no isolation gain over subprocess for read-only file_system tools). Precedents: E2B, major cloud providers, Daytona | | D17 | Official `mcp` Python SDK, exact-pinned (`==`), updated via Renovate; `MCPBridgeTool` adapter | Used by every major framework (LangChain, CrewAI, major agent SDKs, Pydantic AI). Python 3.14 compatible. Pydantic v2 compatible. Thin adapter isolates codebase from SDK changes | Custom MCP client (must implement protocol handshake, track spec changes manually) | | D18 | MCP result mapping via adapter in `MCPBridgeTool` | Keep `ToolResult` as-is. Text concatenation for LLM path. Rich content in metadata. Zero disruption to existing codebase | Extend ToolResult for multi-modal (cascading changes across codebase; LLM providers consume as text anyway) | diff --git a/docs/architecture/tech-stack.md b/docs/architecture/tech-stack.md index 27e5aa46b9..28157b83fe 100644 --- a/docs/architecture/tech-stack.md +++ b/docs/architecture/tech-stack.md @@ -87,7 +87,7 @@ The SynthOrg engine is structured as a set of loosely coupled subsystems. Each b | **Product Telemetry** | Optional: Logfire (via `logfire` SDK), NoopReporter (default) | Opt-in anonymous product telemetry (disabled by default). Pluggable `TelemetryReporter` protocol with `PrivacyScrubber` (allowlist validation). Optional dependency: `telemetry = ["logfire"]`. | | **Agent Communication** | A2A Protocol compatible | Future-proof inter-agent communication. See [Industry Standards](../reference/standards.md). | | **Authentication** | PyJWT + argon2-cffi | JWT (HMAC HS256/384/512) for session tokens, Argon2id for password hashing, HMAC-SHA256 for API key storage (keyed with server secret). | -| **Name Generation** | Faker | Multi-locale agent name generation for templates and setup wizard. 57 Latin-script locales across 12 world regions, cached Faker instances, deterministic seeding for reproducible names. | +| **Name Generation** | Faker | Multi-locale agent name generation for templates and setup wizard. 57 Latin-script locales across 11 world regions, cached Faker instances, deterministic seeding for reproducible names. | | **Config Format** | YAML + Pydantic validation | Human-readable config with strict validation. | | **CLI** | Go (Cobra + charm.land/huh/v2, charm.land/lipgloss/v2) | Cross-platform binary for Docker lifecycle management: `init`, `start`, `stop`, `status`, `logs`, `update`, `doctor`, `uninstall`, `version`, `cleanup`, `backup`, `wipe`, `config`, `completion-install`. Update channel (stable/dev) selectable via `synthorg config set channel dev`. Distributed via GoReleaser + install scripts (`curl \| bash`, `irm \| iex`). Syft generates CycloneDX JSON SBOMs per archive (via GoReleaser `sboms:` stanza). Cosign keyless signing of checksums file (`.sig` + `.pem`). SLSA Level 3 provenance attestations on all release archives. Sigstore provenance bundle (`.sigstore.json`) attached to releases. | diff --git a/docs/design/communication.md b/docs/design/communication.md index 76c15eaa6a..096cb01088 100644 --- a/docs/design/communication.md +++ b/docs/design/communication.md @@ -196,7 +196,7 @@ a bidirectional reference for the gateway translation layer. | `COMPLETED` | `completed` | Bidirectional | Successful completion | | `FAILED` | `failed` | Bidirectional | Unrecoverable failure | | `CANCELLED` | `canceled` | Bidirectional | Client-initiated cancellation | -| `REJECTED` *(proposed)* | `rejected` | Bidirectional | Task refused by agent or guard (requires new TaskStatus value) | +| `REJECTED` | `rejected` | Bidirectional | Task refused by agent or guard | **Gate verdict mapping** (not a task state): diff --git a/docs/design/self-improvement.md b/docs/design/self-improvement.md index 72e03d463a..76b62a1788 100644 --- a/docs/design/self-improvement.md +++ b/docs/design/self-improvement.md @@ -75,7 +75,6 @@ src/synthorg/meta/ ab_test.py -- A/B test group assignment and observation loop ab_comparator.py -- Control vs treatment comparison (Welch-backed) ab_models.py -- GroupAssignment, ABTestVerdict, GroupMetrics (sample-backed) - clock.py -- Clock protocol + RealClock (wall-time abstraction for tests) roster.py -- OrgRoster protocol + CallableOrgRoster / NoOpOrgRoster group_aggregator.py -- GroupSignalAggregator protocol + TrackerGroupAggregator inverse_dispatch.py -- RollbackHandler protocol + 4 mutator protocols + default handlers diff --git a/docs/design/tools.md b/docs/design/tools.md index 6c441f127d..1abb302e0a 100644 --- a/docs/design/tools.md +++ b/docs/design/tools.md @@ -120,7 +120,8 @@ the local-first experience lightweight while providing strong isolation where it Docker MVP uses `aiodocker` (async-native) with a pre-built image (Python 3.14 + Node.js LTS + basic utils, <500MB). If Docker is unavailable, the framework -fails with a clear error; no unsafe subprocess fallback for code execution +fails with a clear error for any tool category whose configured backend is Docker; +low-risk categories (file_system, git) continue to run via subprocess ([Decision Log](../architecture/decisions.md) D16). ### Container Log Shipping diff --git a/docs/reference/claude-reference.md b/docs/reference/claude-reference.md index 66b0bc4e16..aba2e78a0d 100644 --- a/docs/reference/claude-reference.md +++ b/docs/reference/claude-reference.md @@ -37,7 +37,7 @@ curl http://localhost:3000/api/v1/readyz # backend (via web proxy) ```text src/synthorg/ - api/ # Litestar REST + WebSocket API, RFC 9457 errors, setup wizard, personality presets, auth/ (role-based access control, HttpOnly cookie sessions, CSRF double-submit, lockout_store, refresh_store, concurrent session enforcement, session store, user presence, OrgRole enum for org config permissions), guards (HumanRole-based + OrgRole-based with department scoping via require_org_mutation), user management (CRUD + org-role grant/revoke), dto_org (request DTOs for company/department/agent mutations), dto_workflow (request/response DTOs for workflow definition and execution operations), services/org_mutations (read-modify-write config mutation service), auto-wiring, lifecycle (auto-promote first owner), bootstrap (agent registry init from config), template packs (list + live-apply), memory admin (fine-tuning pipeline with orchestrator, checkpoint management, preflight checks, run history, embedder queries), optimistic concurrency (ETag/If-Match), TLS config, tiered rate limiting (unauth by IP, auth by user ID), rate_limits/policies (RATE_LIMIT_POLICIES canonical registry of (max_requests, window_seconds) defaults per operation id + per_op_rate_limit_from_policy helper; operator overrides flow through PerOpRateLimitConfig.overrides separately), workflows (visual workflow definition CRUD, validation, YAML export, blueprint listing, blueprint instantiation, version history, diff, rollback), workflow executions (activate, list, get, cancel), ceremony policy (project + per-department query/override, resolved policy with field origins), quality overrides (per-agent quality score override CRUD), reports (on-demand report generation, period listing), notification_dispatcher (fan-out notification sink), training (training plan CRUD, execution, preview, overrides) + api/ # Litestar REST + WebSocket API, RFC 9457 errors, setup wizard, personality presets, auth/ (role-based access control, HttpOnly cookie sessions, CSRF double-submit, lockout / refresh-token / session repositories under persistence/{sqlite,postgres}/, concurrent session enforcement, user presence, OrgRole enum for org config permissions), guards (HumanRole-based + OrgRole-based with department scoping via require_org_mutation), user management (CRUD + org-role grant/revoke), dto_org (request DTOs for company/department/agent mutations), dto_workflow (request/response DTOs for workflow definition and execution operations), services/org_mutations (read-modify-write config mutation service), auto-wiring, lifecycle (auto-promote first owner), bootstrap (agent registry init from config), template packs (list + live-apply), memory admin (fine-tuning pipeline with orchestrator, checkpoint management, preflight checks, run history, embedder queries), optimistic concurrency (ETag/If-Match), TLS config, tiered rate limiting (unauth by IP, auth by user ID), rate_limits/policies (RATE_LIMIT_POLICIES canonical registry of (max_requests, window_seconds) defaults per operation id + per_op_rate_limit_from_policy helper; operator overrides flow through PerOpRateLimitConfig.overrides separately), workflows (visual workflow definition CRUD, validation, YAML export, blueprint listing, blueprint instantiation, version history, diff, rollback), workflow executions (activate, list, get, cancel), ceremony policy (project + per-department query/override, resolved policy with field origins), quality overrides (per-agent quality score override CRUD), reports (on-demand report generation, period listing), notification_dispatcher (fan-out notification sink), training (training plan CRUD, execution, preview, overrides) backup/ # Backup/restore orchestrator, scheduler, retention, handlers/ budget/ # Cost tracking, budget enforcement, quota degradation (including synchronous peek for routing-time selector hints), CFO optimization, trend analysis, budget forecasting, configurable currency formatting, risk budget (cumulative risk-unit tracking, risk scoring integration, risk check, risk records), automated reporting (periodic comprehensive reports, spending/performance/task-completion/risk-trends templates, report scheduling config), coordination metrics (9 empirical metrics: efficiency, overhead, error amplification, message density, redundancy, Amdahl ceiling, straggler gap, token/speedup ratio, message overhead), project cost aggregates (durable per-project lifetime cost totals surviving retention pruning) cli/ # Python CLI module (superseded by top-level cli/ Go binary) diff --git a/docs/reference/comparison.md b/docs/reference/comparison.md new file mode 100644 index 0000000000..aeb0a713ed --- /dev/null +++ b/docs/reference/comparison.md @@ -0,0 +1,268 @@ +--- +title: Framework Comparison +description: >- + How SynthOrg compares to every notable agent orchestration + framework, platform, and research project. +--- + + + +# Framework Comparison + +How SynthOrg compares to agent orchestration frameworks, platforms, and research projects. + +Last updated: 2026-05-01 + +**Legend:** +✔ Full support | ~ Partial support | - Not supported | ⏲ Planned + +!!! tip "Interactive Version" + For a filterable, sortable version of this comparison, visit the [interactive comparison page](https://synthorg.io/compare/). + +## Organization & Coordination + +| Framework | Category | License | Pricing | Self-Hosted | Org Structure | Multi-Agent | Task Delegation | Human-in-the-Loop | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ✔ | ~ | ✔ | ✔ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ✔ | ✔ | ✔ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ✔ | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | - | ✔ | ✔ | ✔ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ✔ | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ✔ | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | - | ~ | ~ | ✔ | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ✔ | ~ | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | ~ | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | ~ | ✔ | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ✔ | ✔ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ~ | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | ✔ | ✔ | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | ✔ | ~ | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | ~ | ✔ | ✔ | ~ | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ~ | ~ | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | - | ~ | ✔ | ✔ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | - | ~ | ~ | ✔ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | - | ~ | ~ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | - | ~ | ~ | ~ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | - | ~ | ~ | ✔ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | - | ✔ | ✔ | ✔ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | - | ✔ | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | - | ~ | ~ | ~ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | - | ✔ | ~ | ✔ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | - | - | - | - | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | - | - | - | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | - | ~ | ~ | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | - | - | - | ~ | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | - | ~ | ~ | ~ | + +## Technical Capabilities + +| Framework | Category | License | Pricing | Self-Hosted | Memory | Tool Use | Security | Workflows | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ✔ | ✔ | ~ | ~ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ✔ | ✔ | ~ | ~ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | ✔ | ✔ | ~ | ✔ | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | ✔ | ✔ | - | ✔ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | ✔ | ✔ | ~ | ✔ | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | - | ✔ | ~ | ~ | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ✔ | ✔ | ~ | ~ | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | ~ | ✔ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | ~ | - | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | ~ | ✔ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ✔ | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ✔ | ✔ | - | ✔ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | ~ | ~ | - | ~ | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | ~ | ~ | ✔ | ✔ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | ~ | ✔ | ~ | ~ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | ~ | ~ | ~ | ✔ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | ~ | ✔ | ✔ | ✔ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | ✔ | ~ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | ✔ | ✔ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ~ | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ✔ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | ~ | ✔ | - | ✔ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | - | ✔ | ~ | - | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | - | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | ~ | ✔ | - | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | ~ | ✔ | ~ | - | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | ~ | ✔ | ~ | ~ | + +## Operations & Tooling + +| Framework | Category | License | Pricing | Self-Hosted | Budget / Cost | Observability | Dashboard | CLI | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ~ | ✔ | ✔ | ~ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | ~ | ✔ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | - | ~ | ~ | - | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | - | ~ | ~ | ✔ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ~ | - | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ✔ | - | - | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ~ | ✔ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | - | ~ | - | - | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | - | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | ~ | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | - | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ~ | ✔ | ✔ | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ~ | ~ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ~ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ~ | - | - | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | - | ~ | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | - | - | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ~ | - | ~ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ~ | ~ | ✔ | ~ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | - | - | - | ~ | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | - | ~ | ✔ | ~ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ✔ | ~ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | - | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ✔ | ✔ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ✔ | ~ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | - | ~ | ✔ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | - | ~ | ✔ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | - | ✔ | ~ | ✔ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | ~ | ✔ | ✔ | ✔ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | - | ✔ | ✔ | ✔ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | - | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | - | ~ | - | ✔ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | ~ | ✔ | - | ~ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | - | ~ | ~ | ~ | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ~ | ~ | ✔ | ✔ | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | - | ~ | - | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | - | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | - | ~ | ~ | ~ | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | - | ~ | - | ~ | + +## Maturity + +| Framework | Category | License | Pricing | Self-Hosted | Production Ready | Templates | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ~ | ✔ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ~ | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | - | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | ✔ | ✔ | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ~ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | - | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | - | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | - | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ~ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ~ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | - | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ✔ | - | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ~ | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ~ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ~ | ✔ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | - | - | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | - | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | ✔ | ~ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | ✔ | ✔ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ~ | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ✔ | ~ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | ~ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | ~ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | ~ | ~ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | ✔ | ~ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ✔ | ~ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | ✔ | ~ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | ~ | ~ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | ~ | ~ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | - | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | ~ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | ~ | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ~ | - | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | ~ | ~ | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | ~ | - | + +## Project Links + +- **SynthOrg** -- [Website](https://synthorg.io) -- [Repository](https://github.com/Aureliolo/synthorg) +- **CrewAI** -- [Website](https://crewai.com) -- [Repository](https://github.com/crewAIInc/crewAI) +- **AutoGen** -- [Website](https://microsoft.github.io/autogen/) -- [Repository](https://github.com/microsoft/autogen) +- **LangGraph** -- [Website](https://www.langchain.com/langgraph) -- [Repository](https://github.com/langchain-ai/langgraph) +- **OpenAI Agents SDK** -- [Website](https://openai.github.io/openai-agents-python/) -- [Repository](https://github.com/openai/openai-agents-python) +- **Semantic Kernel** -- [Website](https://learn.microsoft.com/en-us/semantic-kernel/) -- [Repository](https://github.com/microsoft/semantic-kernel) +- **Google ADK** -- [Website](https://google.github.io/adk-docs/) -- [Repository](https://github.com/google/adk-python) +- **PydanticAI** -- [Website](https://ai.pydantic.dev/) -- [Repository](https://github.com/pydantic/pydantic-ai) +- **CAMEL** -- [Website](https://www.camel-ai.org) -- [Repository](https://github.com/camel-ai/camel) +- **smolagents** -- [Website](https://huggingface.co/docs/smolagents) -- [Repository](https://github.com/huggingface/smolagents) +- **AG2** -- [Website](https://ag2.ai) -- [Repository](https://github.com/ag2ai/ag2) +- **Agno** -- [Website](https://www.agno.com) -- [Repository](https://github.com/agno-agi/agno) +- **Letta** -- [Website](https://www.letta.com) -- [Repository](https://github.com/letta-ai/letta) +- **Agency Swarm** -- [Website](https://agency-swarm.ai/) -- [Repository](https://github.com/VRSEN/agency-swarm) +- **Mastra** -- [Website](https://mastra.ai) -- [Repository](https://github.com/mastra-ai/mastra) +- **Langroid** -- [Website](https://langroid.github.io/langroid/) -- [Repository](https://github.com/langroid/langroid) +- **Strands Agents** -- [Website](https://strandsagents.com/) -- [Repository](https://github.com/strands-agents/sdk-python) +- **BeeAI Framework** -- [Website](https://beeai.dev) -- [Repository](https://github.com/i-am-bee/beeai-framework) +- **Haystack** -- [Website](https://haystack.deepset.ai/) -- [Repository](https://github.com/deepset-ai/haystack) +- **ChatDev** -- [Website](https://chatdev.ai/) -- [Repository](https://github.com/OpenBMB/ChatDev) +- **MetaGPT** -- [Website](https://docs.deepwisdom.ai/) -- [Repository](https://github.com/FoundationAgents/MetaGPT) +- **Inngest AgentKit** -- [Website](https://www.inngest.com) -- [Repository](https://github.com/inngest/agent-kit) +- **Temporal** -- [Website](https://temporal.io) -- [Repository](https://github.com/temporalio/temporal) +- **n8n** -- [Website](https://n8n.io) -- [Repository](https://github.com/n8n-io/n8n) +- **Dify** -- [Website](https://dify.ai) -- [Repository](https://github.com/langgenius/dify) +- **Restack** -- [Website](https://www.restack.io/) -- [Repository](https://github.com/restackio) +- **Flyte** -- [Website](https://flyte.org) -- [Repository](https://github.com/flyteorg/flyte) +- **Flowise** -- [Website](https://flowiseai.com) -- [Repository](https://github.com/FlowiseAI/Flowise) +- **Langflow** -- [Website](https://www.langflow.org) -- [Repository](https://github.com/langflow-ai/langflow) +- **AutoGPT** -- [Website](https://agpt.co) -- [Repository](https://github.com/Significant-Gravitas/AutoGPT) +- **Motia** -- [Website](https://motia.dev) -- [Repository](https://github.com/MotiaDev/motia) +- **LangSmith** -- [Website](https://smith.langchain.com) +- **Amazon Bedrock Agents** -- [Website](https://aws.amazon.com/bedrock/agents/) +- **Vertex AI Agent Builder** -- [Website](https://cloud.google.com/products/agent-builder) -- [Repository](https://github.com/google/adk-python) +- **Agentforce** -- [Website](https://www.salesforce.com/agentforce/) -- [Repository](https://github.com/salesforce/agent-sdk) +- **Relevance AI** -- [Website](https://relevanceai.com) +- **Julep** -- [Website](https://julep.ai) -- [Repository](https://github.com/julep-ai/julep) +- **LlamaIndex Workflows** -- [Website](https://www.llamaindex.ai/) -- [Repository](https://github.com/run-llama/llama_index) +- **Composio** -- [Website](https://composio.dev) -- [Repository](https://github.com/ComposioHQ/composio) +- **E2B** -- [Website](https://e2b.dev) -- [Repository](https://github.com/e2b-dev/E2B) +- **OpenHands** -- [Website](https://openhands.dev) -- [Repository](https://github.com/OpenHands/OpenHands) +- **Rivet** -- [Website](https://rivet.ironcladapp.com) -- [Repository](https://github.com/Ironclad/rivet) +- **Vercel AI SDK** -- [Website](https://ai-sdk.dev) -- [Repository](https://github.com/vercel/ai) +- **Llama Stack** -- [Website](https://llamastack.github.io/) -- [Repository](https://github.com/meta-llama/llama-stack) +- **Atomic Agents** -- [Website](https://github.com/BrainBlend-AI/atomic-agents) -- [Repository](https://github.com/BrainBlend-AI/atomic-agents) diff --git a/docs/reference/pluggable-subsystems.md b/docs/reference/pluggable-subsystems.md index 8849d72643..23ecf77286 100644 --- a/docs/reference/pluggable-subsystems.md +++ b/docs/reference/pluggable-subsystems.md @@ -47,12 +47,11 @@ Each subsystem still owns its config discriminator; the registries replace only ### Rollout strategies -- `meta/rollout/clock.py`: `Clock`. - `meta/rollout/roster.py`: `OrgRoster`. - `meta/rollout/group_aggregator.py`: `GroupSignalAggregator`. - `meta/rollout/inverse_dispatch.py`: `RollbackHandler` + 4 mutator protocols. - `meta/factory.py::build_rollout_strategies()` + `build_rollback_executor()`. -- All plumbed through frozen `SelfImprovementConfig`, with safe defaults (`RealClock`, `NoOpOrgRoster`, null aggregator) so the behaviour is opt-in. +- All plumbed through frozen `SelfImprovementConfig`, with safe defaults (`SystemClock` from `synthorg.core.clock`, `NoOpOrgRoster`, null aggregator) so the behaviour is opt-in. ### API rate limits diff --git a/docs/roadmap/index.md b/docs/roadmap/index.md index 0ceea20f5e..f245459b17 100644 --- a/docs/roadmap/index.md +++ b/docs/roadmap/index.md @@ -2,7 +2,7 @@ ## Current Status -SynthOrg is in **active development**. The core subsystems are built, tested (25,000+ unit tests, 80%+ coverage), and integrated through a REST + WebSocket API, React 19 dashboard, and Go CLI. See the [releases page](https://github.com/Aureliolo/synthorg/releases) for the latest tagged build. +SynthOrg is in **active development**. The core subsystems are built, tested (27,000+ tests, 80%+ coverage), and integrated through a REST + WebSocket API, React 19 dashboard, and Go CLI. See the [releases page](https://github.com/Aureliolo/synthorg/releases) for the latest tagged build. What works today: diff --git a/src/synthorg/core/evidence.py b/src/synthorg/core/evidence.py index 1c58468e11..8d9078576e 100644 --- a/src/synthorg/core/evidence.py +++ b/src/synthorg/core/evidence.py @@ -48,8 +48,8 @@ class RecommendedAction(BaseModel): class EvidencePackageSignature(BaseModel): """Approver signature over an EvidencePackage. - Produced by the audit chain (new module in #1268) using - ML-DSA-65 (FIPS 204) or equivalent quantum-safe signature scheme. + Produced by the audit chain using ML-DSA-65 (FIPS 204) or equivalent + quantum-safe signature scheme. Attributes: approver_id: Identity of the approver. diff --git a/src/synthorg/engine/middleware/coordination_protocol.py b/src/synthorg/engine/middleware/coordination_protocol.py index 59a3aec3f2..524013ac4b 100644 --- a/src/synthorg/engine/middleware/coordination_protocol.py +++ b/src/synthorg/engine/middleware/coordination_protocol.py @@ -45,8 +45,8 @@ class CoordinationMiddlewareContext(BaseModel): dispatch_result: Result of dispatch execution (set after phase 5). status_rollup: Aggregated subtask status rollup (set after phase 6). phases: Phase results accumulated so far. - task_ledger: TaskLedger populated by #1257 middleware. - progress_ledger: ProgressLedger populated by #1257 middleware. + task_ledger: TaskLedger populated by TaskLedgerMiddleware. + progress_ledger: ProgressLedger populated by ProgressLedgerMiddleware. metadata: Middleware-to-middleware data pass-through. """ @@ -77,11 +77,11 @@ class CoordinationMiddlewareContext(BaseModel): ) task_ledger: TaskLedger | None = Field( default=None, - description="TaskLedger from #1257 middleware", + description="TaskLedger from TaskLedgerMiddleware", ) progress_ledger: ProgressLedger | None = Field( default=None, - description="ProgressLedger from #1257 middleware", + description="ProgressLedger from ProgressLedgerMiddleware", ) metadata: dict[str, Any] = Field( default_factory=dict, diff --git a/src/synthorg/engine/trajectory/efficiency_ratios.py b/src/synthorg/engine/trajectory/efficiency_ratios.py index 4b98974814..8c64eea072 100644 --- a/src/synthorg/engine/trajectory/efficiency_ratios.py +++ b/src/synthorg/engine/trajectory/efficiency_ratios.py @@ -2,8 +2,7 @@ Defines ``IdealTrajectoryBaseline`` (per-task-type ideal reference) and ``EfficiencyRatios`` (per-run ratios against that baseline), -including verbosity, structural erosion, and PTE metrics folded -from #1244 (SlopCodeBench) and #1248 (PTE). +including verbosity, structural erosion, and Program Test Efficiency (PTE) metrics. """ from typing import Literal diff --git a/src/synthorg/hr/evaluation/extractors/efficiency.py b/src/synthorg/hr/evaluation/extractors/efficiency.py index 2a39771b6c..25f291c187 100644 --- a/src/synthorg/hr/evaluation/extractors/efficiency.py +++ b/src/synthorg/hr/evaluation/extractors/efficiency.py @@ -2,8 +2,7 @@ Lifts cost / time / token efficiency sub-metrics from the performance snapshot's 30d window (or 7d fallback). Honors the -runtime kill-switch resolver added by #1648 for the cost and -latency sub-metrics: +runtime kill-switch resolver for the cost and latency sub-metrics: - ``hr.evaluation_cost_enabled`` gates the ``cost`` sub-metric. - ``hr.evaluation_latency_enabled`` gates the ``time`` sub-metric. @@ -35,8 +34,7 @@ class EfficiencyMetricExtractor: """Extract cost / time / tokens efficiency sub-metrics. Accepts an optional ``ConfigResolver`` so the operator-facing - cost/latency kill switches added in #1648 keep working after - the inline-Efficiency block is hoisted out of ``EvaluationService``. + cost/latency kill switches stay reachable from the extractor seam. """ __slots__ = ("_config_resolver",) diff --git a/src/synthorg/hr/evaluation/metric_extractor_protocol.py b/src/synthorg/hr/evaluation/metric_extractor_protocol.py index 49999222b7..6f2fb2245d 100644 --- a/src/synthorg/hr/evaluation/metric_extractor_protocol.py +++ b/src/synthorg/hr/evaluation/metric_extractor_protocol.py @@ -161,10 +161,9 @@ class MetricExtractor(Protocol): records, feedback, audit counts, trust level, ...) and return an ``ExtractedMetrics``. The composite scorer handles the rest. - ``extract`` is async because some extractors (notably - Efficiency, after rebase against the kill-switch resolver - landed in #1648) need to await ``ConfigResolver`` calls. CPU-bound - extractors simply ``return`` directly. + ``extract`` is async because some extractors (notably Efficiency) + need to await ``ConfigResolver`` calls for dynamic configuration. + CPU-bound extractors simply ``return`` directly. """ @property diff --git a/src/synthorg/providers/management/preset_override_service.py b/src/synthorg/providers/management/preset_override_service.py index 517eb5b76b..9f2c60bb98 100644 --- a/src/synthorg/providers/management/preset_override_service.py +++ b/src/synthorg/providers/management/preset_override_service.py @@ -15,7 +15,7 @@ ``synthorg.providers.presets.get_preset(name)`` entry point still serves the in-code preset directly; consumers that need overrides applied call this service. Migrating every caller to consult the -override at read time is a follow-on task tracked in #1642. +override at read time is a planned enhancement. """ from datetime import UTC, datetime diff --git a/tests/unit/api/test_exception_handlers.py b/tests/unit/api/test_exception_handlers.py index 2cc2e3c3e1..37818ec9b9 100644 --- a/tests/unit/api/test_exception_handlers.py +++ b/tests/unit/api/test_exception_handlers.py @@ -821,7 +821,7 @@ def test_http_exception_non_string_detail_is_coerced(self) -> None: class TestNormalizeStatusCode: - """``_normalize_status_code`` warning paths added in #1659.""" + """``_normalize_status_code`` warning path validation tests.""" def test_invalid_type_logs_warning(self) -> None: """Non-numeric, non-string types (e.g. ``object``) emit a warning.""" diff --git a/tests/unit/budget/test_cost_record.py b/tests/unit/budget/test_cost_record.py index e40b5ba2e2..d4f47ee010 100644 --- a/tests/unit/budget/test_cost_record.py +++ b/tests/unit/budget/test_cost_record.py @@ -314,7 +314,7 @@ def test_factory(self) -> None: @pytest.mark.unit class TestCostRecordAnalyticsFields: - """New per-call analytics fields added in #227.""" + """Per-call analytics fields validation.""" def _base(self) -> CostRecord: return CostRecord( diff --git a/tests/unit/engine/artifacts/test_service.py b/tests/unit/engine/artifacts/test_service.py index bdf4511e8a..b3e596a84b 100644 --- a/tests/unit/engine/artifacts/test_service.py +++ b/tests/unit/engine/artifacts/test_service.py @@ -1,8 +1,8 @@ """Unit tests for :class:`ArtifactService`. -Verifies that the audit events emitted by the service are the new -``API_ARTIFACT_*`` constants (renamed from ``PERSISTENCE_ARTIFACT_*`` -in #1562) and that mutations only emit when they actually mutate. +Verifies that the audit events emitted by the service use the +``API_ARTIFACT_*`` constants and that mutations only emit when they +actually mutate. """ import pytest diff --git a/tests/unit/observability/test_events.py b/tests/unit/observability/test_events.py index 597de9e840..ff075528da 100644 --- a/tests/unit/observability/test_events.py +++ b/tests/unit/observability/test_events.py @@ -312,12 +312,11 @@ def test_all_domain_modules_discovered(self) -> None: "verification", "versioning", "web", - # Added in #1395 -- logging convention cleanup: + # Logging convention namespaces: "capture", "fine_tune", "propagation", - # Added in #1436 to keep sink internal errors out of - # the audited "security.*" namespace. + # Separate from the audited "security.*" namespace. "audit_chain", # Trace handler lifecycle + config validation events. "tracing", @@ -328,8 +327,7 @@ def test_all_domain_modules_discovered(self) -> None: # StrategyRegistry and the persistence/memory backend # registries that consolidate factory dispatch. "registry", - # Added in #1688 for ``GeneralRetryHandler`` constructor - # validation logs (max_attempts / base / cap rejection). + # Resilience handler validation and event logging. "resilience", } discovered = {info.name for info in pkgutil.iter_modules(events.__path__)} diff --git a/tests/unit/settings/test_definitions_config_bridge.py b/tests/unit/settings/test_definitions_config_bridge.py index 00480f258e..e479dd03f7 100644 --- a/tests/unit/settings/test_definitions_config_bridge.py +++ b/tests/unit/settings/test_definitions_config_bridge.py @@ -1,11 +1,9 @@ """Parametrized coverage for the config-bridge settings. -This is a single-file replacement for eleven per-namespace files -that would have contained near-identical assertions. Each row of -``_EXPECTED`` describes one newly-added setting from #1398/#1400: -expected namespace, key, type, default, numeric bounds, restart -requirement, and required presence of ``yaml_path`` / ``group`` / -``description``. +Each row of ``_EXPECTED`` describes one settings entry covering the +namespace bridge: expected namespace, key, type, default, numeric +bounds, restart requirement, and required presence of ``yaml_path`` / +``group`` / ``description``. Behaviour is asserted against the live registry after triggering registration via ``import synthorg.settings.definitions``.