From 3e00022dacef44ba6b2481d0fe2a17f5679322af Mon Sep 17 00:00:00 2001 From: Aurelio <19254254+Aureliolo@users.noreply.github.com> Date: Sat, 2 May 2026 11:28:48 +0200 Subject: [PATCH 1/2] docs: audit cleanup D -- public-facing & docs sync (#1709) Bundle D from the 2026-05-01 codebase audit. Public-facing pages and design docs drifted from the actual codebase: - README.md and docs/roadmap/index.md: replace "25,000+ unit tests" with "27,000+ tests". Live count is 27,566 collected (uv run python -m pytest tests/ --collect-only -q | tail -1 reports "27566 tests collected"). - docs/architecture/tech-stack.md: "57 Latin-script locales across 12 world regions" becomes 11 world regions (data/locales.yaml has 11 region groupings). - docs/reference/comparison.md: drop the .gitignore line and commit the generator output. The script and YAML are correct; the local file was generated 2026-04-02 against a YAML state where SynthOrg features were classified as planned across the board, then YAML was updated 2026-04-26 in #1595 without rerunning the generator. CI regenerates on every build, but tracking the artefact surfaces drift in PR diffs. - docs/architecture/decisions.md: rewrite D16 to describe the actual layered SandboxBackend strategy (subprocess for low-risk file_system and git, Docker for high-risk code_execution / terminal / database / web). Code (src/synthorg/tools/sandbox/factory.py) and design (docs/design/tools.md) already align; only the decision row drifted. - docs/design/tools.md: drop the "no unsafe subprocess fallback for code execution" phrasing; low-risk categories continue via subprocess even when Docker is unavailable. - docs/design/communication.md: drop *(proposed)* on the REJECTED A2A task-state row. TaskStatus.REJECTED is implemented at src/synthorg/core/enums.py:256 with the CREATED..REJECTED transition documented in the enum docstring. - docs/design/self-improvement.md, docs/reference/pluggable-subsystems.md and docs/reference/claude-reference.md: drop references to deleted modules. Clock and SystemClock now live at src/synthorg/core/clock.py (not src/synthorg/meta/rollout/clock.py). Auth lockout / refresh / session storage now lives under persistence/{sqlite,postgres}/ (not src/synthorg/api/auth/{lockout_store,refresh_store,session_store}.py). - docs/DESIGN_SPEC.md: add the Internationalization design page row; internationalization.md exists but was unlisted. - CLAUDE.md: drop the dangling docs/design/operations.md reference from the vendor-name carve-out list. src/synthorg/providers/presets.py is already item (3) of the same list and plays that role; renumber. Migration-framing rot (agent 155): strip 11 issue-number / migration narratives back to present-tense descriptions across src/ and tests/, plus 2 adjacent drive-bys in files already touched (coordination_protocol.py:48-49, efficiency.py:5). tests/unit/architecture/test_layering.py keeps its #1610 reference; the test exists to enforce the deletion of synthorg.api.errors, so the historical anchor is load-bearing. Closes #1709 --- .gitignore | 3 - CLAUDE.md | 2 +- README.md | 2 +- docs/DESIGN_SPEC.md | 1 + docs/architecture/decisions.md | 2 +- docs/architecture/tech-stack.md | 2 +- docs/design/communication.md | 2 +- docs/design/self-improvement.md | 1 - docs/design/tools.md | 3 +- docs/reference/claude-reference.md | 2 +- docs/reference/comparison.md | 268 ++++++++++++++++++ docs/reference/pluggable-subsystems.md | 3 +- docs/roadmap/index.md | 2 +- src/synthorg/core/evidence.py | 4 +- .../middleware/coordination_protocol.py | 8 +- .../engine/trajectory/efficiency_ratios.py | 3 +- .../hr/evaluation/extractors/efficiency.py | 6 +- .../evaluation/metric_extractor_protocol.py | 7 +- .../management/preset_override_service.py | 2 +- tests/unit/api/test_exception_handlers.py | 2 +- tests/unit/budget/test_cost_record.py | 2 +- tests/unit/engine/artifacts/test_service.py | 6 +- tests/unit/observability/test_events.py | 8 +- .../test_definitions_config_bridge.py | 10 +- 24 files changed, 304 insertions(+), 47 deletions(-) create mode 100644 docs/reference/comparison.md diff --git a/.gitignore b/.gitignore index a4c65e95d9..1ac4f8ed5f 100644 --- a/.gitignore +++ b/.gitignore @@ -93,9 +93,6 @@ web/src/api/types/generated.d.ts # (hand-off artefact for issue #1417; regenerate locally as needed). web/src/i18n/_extracted_catalog.json -# Generated comparison page (built by scripts/generate_comparison.py) -docs/reference/comparison.md - # Astro / Node.js (landing page) site/node_modules/ site/dist/ diff --git a/CLAUDE.md b/CLAUDE.md index 043c7a2380..e0822014a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -210,7 +210,7 @@ When tests fail due to timeout, slowness, or xdist resource contention: - **Timeout**: 30 seconds per test (global in `pyproject.toml`; do not add per-file `pytest.mark.timeout(30)` markers; non-default overrides like `timeout(60)` are allowed) - **Parallelism**: `pytest-xdist` via `-n 8`. **ALWAYS** include `-n 8` when running pytest locally, never run tests sequentially. CI uses `-n auto` (fewer cores on runners). - **Parametrize**: Prefer `@pytest.mark.parametrize` for testing similar cases -- **Vendor-agnostic everywhere**: NEVER use real vendor names (Anthropic, OpenAI, Claude, GPT, etc.) in project-owned code, docstrings, comments, tests, or config examples. Use generic names: `example-provider`, `example-large-001`, `example-medium-001`, `example-small-001`, `large`/`medium`/`small` as aliases. Vendor names may only appear in: (1) Operations design page provider list (`docs/design/operations.md`), (2) `.claude/` skill/agent files, (3) third-party import paths/module names (e.g. `litellm.types.llms.openai`), (4) provider presets (`src/synthorg/providers/presets.py`) which are user-facing runtime data, (5) provider logo assets (`web/public/provider-logos/*.svg` plus the matching README) -- the SVG filenames mirror the preset names by necessity, and `` falls back to a generic icon if a file is absent. Tests must use `test-provider`, `test-small-001`, etc. +- **Vendor-agnostic everywhere**: NEVER use real vendor names (Anthropic, OpenAI, Claude, GPT, etc.) in project-owned code, docstrings, comments, tests, or config examples. Use generic names: `example-provider`, `example-large-001`, `example-medium-001`, `example-small-001`, `large`/`medium`/`small` as aliases. Vendor names may only appear in: (1) `.claude/` skill/agent files, (2) third-party import paths/module names (e.g. `litellm.types.llms.openai`), (3) provider presets (`src/synthorg/providers/presets.py`) which are user-facing runtime data, (4) provider logo assets (`web/public/provider-logos/*.svg` plus the matching README) -- the SVG filenames mirror the preset names by necessity, and `` falls back to a generic icon if a file is absent. Tests must use `test-provider`, `test-small-001`, etc. - **Property-based testing**: Python uses [Hypothesis](https://hypothesis.readthedocs.io/), React uses [fast-check](https://fast-check.dev/), Go uses `testing.F` fuzz functions. CI runs 10 deterministic examples per property test (`derandomize=True`, no flakes). When Hypothesis finds a failure, it is a **real bug**: read the shrunk example, fix the underlying bug, and add an explicit `@example(...)` decorator so the case is permanently covered. See [docs/reference/claude-reference.md](docs/reference/claude-reference.md) §"Property-based Testing (Hypothesis): Deep Dive" for profile catalog, local fuzzing commands, and failure-handling workflow. - **Flaky tests**: NEVER skip, dismiss, or ignore flaky tests; always fix them fully and fundamentally. For timing-sensitive tests, the **FakeClock-first** rule applies: if the class under test accepts a `clock=` parameter, inject `FakeClock` from `tests._shared.fake_clock` and drive virtual time via `clock.advance(...)` / `await clock.advance_async(...)`. Patch `time.monotonic()` / `asyncio.sleep()` at the module level only for legacy code paths that don't yet have a `Clock` seam. For tasks that must block indefinitely until cancelled (e.g. simulating a slow provider or stubborn coroutine), use `asyncio.Event().wait()` instead of `asyncio.sleep(large_number)`; it is cancellation-safe and carries no timing assumptions. diff --git a/README.md b/README.md index f3b019b71b..baf667e93c 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ SynthOrg is a Python framework for building **synthetic organizations**, autonom Define your company in YAML. Agents collaborate through a message bus, follow workflows (Kanban, Agile sprints, or custom), track costs against budgets, and produce real artifacts. The framework is provider-agnostic (100+ LLMs via [LiteLLM](https://github.com/BerriAI/litellm)), configuration-driven ([Pydantic v2](https://docs.pydantic.dev/) models), and designed for the full autonomy spectrum, from human approval on every action to fully autonomous operation. -> **Early access.** Core subsystems are built and tested (25,000+ unit tests, 80%+ coverage). APIs may change between releases. See the [roadmap](https://synthorg.io/docs/roadmap/) for what's next. +> **Early access.** Core subsystems are built and tested (27,000+ tests, 80%+ coverage). APIs may change between releases. See the [roadmap](https://synthorg.io/docs/roadmap/) for what's next. ## Why SynthOrg? diff --git a/docs/DESIGN_SPEC.md b/docs/DESIGN_SPEC.md index 6317255e03..f1f2a465ed 100644 --- a/docs/DESIGN_SPEC.md +++ b/docs/DESIGN_SPEC.md @@ -43,6 +43,7 @@ The design specification has been split into focused documentation pages for bet | [Client Simulation](design/client-simulation.md) | Client Types, Intake, Review Pipeline, Simulation | Synthetic client framework for workload generation and evaluation | | [Strategy & Trendslop Mitigation](design/strategy.md) | Lenses, Principles, Confidence, Impact | Anti-trendslop mitigation for strategic agents | | [Self-Improvement](design/self-improvement.md) | Meta-Loop, Signals, Rules, Proposals, Rollout | Self-improving company: signal aggregation, rule engine, improvement proposals, staged rollout | +| [Internationalization](design/internationalization.md) | Locale Resolution, UI Text, Translation Scope | British English UI default, locale-aware Intl-based display formatting, no planned translation framework | ## Supporting Pages diff --git a/docs/architecture/decisions.md b/docs/architecture/decisions.md index 7166415914..d2c19058ba 100644 --- a/docs/architecture/decisions.md +++ b/docs/architecture/decisions.md @@ -61,7 +61,7 @@ All significant design and architecture decisions in force today, organized by d | ID | Decision | Rationale | Alternatives considered | |----|----------|-----------|------------------------| -| D16 | Docker MVP via `aiodocker`; `SandboxBackend` protocol for future backends | Docker cold start (1-2s) invisible against LLM latency (2-30s). Pre-built image + user config. Fail if Docker unavailable; no unsafe subprocess fallback. gVisor as config-level hardening upgrade | Docker + WASM (CPython can't run pip packages in WASM), Docker + Firecracker (Linux-only, requires KVM), docker-py (sync, no 3.14 support). Precedents: E2B, major cloud providers, Daytona (none offer unsandboxed fallback) | +| D16 | Layered `SandboxBackend` protocol via `aiodocker`. Subprocess default for low-risk categories (file_system, git); Docker required for high-risk (code_execution, terminal, database, web) | Subprocess is genuinely safe for read-only / workspace-scoped categories: env filtering (allowlist + denylist), restricted PATH, workspace-scoped cwd, timeout + process-group kill, library-injection-var blocking. Docker is required where arbitrary code or network egress can land. Layered design preserves the local-first quickstart (file/git tools work without Docker) without weakening isolation where it matters. Docker cold start (1-2s) is invisible against LLM latency (2-30s). gVisor remains a config-level hardening upgrade for the Docker tier | Docker + WASM (CPython can't run pip packages in WASM), Docker + Firecracker (Linux-only, requires KVM), docker-py (sync, no 3.14 support), Docker-only for every category (rejected: forces a running container for trivial file reads, breaks local-first quickstart, no isolation gain over subprocess for read-only file_system tools). Precedents: E2B, major cloud providers, Daytona | | D17 | Official `mcp` Python SDK, exact-pinned (`==`), updated via Renovate; `MCPBridgeTool` adapter | Used by every major framework (LangChain, CrewAI, major agent SDKs, Pydantic AI). Python 3.14 compatible. Pydantic v2 compatible. Thin adapter isolates codebase from SDK changes | Custom MCP client (must implement protocol handshake, track spec changes manually) | | D18 | MCP result mapping via adapter in `MCPBridgeTool` | Keep `ToolResult` as-is. Text concatenation for LLM path. Rich content in metadata. Zero disruption to existing codebase | Extend ToolResult for multi-modal (cascading changes across codebase; LLM providers consume as text anyway) | diff --git a/docs/architecture/tech-stack.md b/docs/architecture/tech-stack.md index 27e5aa46b9..28157b83fe 100644 --- a/docs/architecture/tech-stack.md +++ b/docs/architecture/tech-stack.md @@ -87,7 +87,7 @@ The SynthOrg engine is structured as a set of loosely coupled subsystems. Each b | **Product Telemetry** | Optional: Logfire (via `logfire` SDK), NoopReporter (default) | Opt-in anonymous product telemetry (disabled by default). Pluggable `TelemetryReporter` protocol with `PrivacyScrubber` (allowlist validation). Optional dependency: `telemetry = ["logfire"]`. | | **Agent Communication** | A2A Protocol compatible | Future-proof inter-agent communication. See [Industry Standards](../reference/standards.md). | | **Authentication** | PyJWT + argon2-cffi | JWT (HMAC HS256/384/512) for session tokens, Argon2id for password hashing, HMAC-SHA256 for API key storage (keyed with server secret). | -| **Name Generation** | Faker | Multi-locale agent name generation for templates and setup wizard. 57 Latin-script locales across 12 world regions, cached Faker instances, deterministic seeding for reproducible names. | +| **Name Generation** | Faker | Multi-locale agent name generation for templates and setup wizard. 57 Latin-script locales across 11 world regions, cached Faker instances, deterministic seeding for reproducible names. | | **Config Format** | YAML + Pydantic validation | Human-readable config with strict validation. | | **CLI** | Go (Cobra + charm.land/huh/v2, charm.land/lipgloss/v2) | Cross-platform binary for Docker lifecycle management: `init`, `start`, `stop`, `status`, `logs`, `update`, `doctor`, `uninstall`, `version`, `cleanup`, `backup`, `wipe`, `config`, `completion-install`. Update channel (stable/dev) selectable via `synthorg config set channel dev`. Distributed via GoReleaser + install scripts (`curl \| bash`, `irm \| iex`). Syft generates CycloneDX JSON SBOMs per archive (via GoReleaser `sboms:` stanza). Cosign keyless signing of checksums file (`.sig` + `.pem`). SLSA Level 3 provenance attestations on all release archives. Sigstore provenance bundle (`.sigstore.json`) attached to releases. | diff --git a/docs/design/communication.md b/docs/design/communication.md index 76c15eaa6a..096cb01088 100644 --- a/docs/design/communication.md +++ b/docs/design/communication.md @@ -196,7 +196,7 @@ a bidirectional reference for the gateway translation layer. | `COMPLETED` | `completed` | Bidirectional | Successful completion | | `FAILED` | `failed` | Bidirectional | Unrecoverable failure | | `CANCELLED` | `canceled` | Bidirectional | Client-initiated cancellation | -| `REJECTED` *(proposed)* | `rejected` | Bidirectional | Task refused by agent or guard (requires new TaskStatus value) | +| `REJECTED` | `rejected` | Bidirectional | Task refused by agent or guard | **Gate verdict mapping** (not a task state): diff --git a/docs/design/self-improvement.md b/docs/design/self-improvement.md index 72e03d463a..76b62a1788 100644 --- a/docs/design/self-improvement.md +++ b/docs/design/self-improvement.md @@ -75,7 +75,6 @@ src/synthorg/meta/ ab_test.py -- A/B test group assignment and observation loop ab_comparator.py -- Control vs treatment comparison (Welch-backed) ab_models.py -- GroupAssignment, ABTestVerdict, GroupMetrics (sample-backed) - clock.py -- Clock protocol + RealClock (wall-time abstraction for tests) roster.py -- OrgRoster protocol + CallableOrgRoster / NoOpOrgRoster group_aggregator.py -- GroupSignalAggregator protocol + TrackerGroupAggregator inverse_dispatch.py -- RollbackHandler protocol + 4 mutator protocols + default handlers diff --git a/docs/design/tools.md b/docs/design/tools.md index 6c441f127d..1abb302e0a 100644 --- a/docs/design/tools.md +++ b/docs/design/tools.md @@ -120,7 +120,8 @@ the local-first experience lightweight while providing strong isolation where it Docker MVP uses `aiodocker` (async-native) with a pre-built image (Python 3.14 + Node.js LTS + basic utils, <500MB). If Docker is unavailable, the framework -fails with a clear error; no unsafe subprocess fallback for code execution +fails with a clear error for any tool category whose configured backend is Docker; +low-risk categories (file_system, git) continue to run via subprocess ([Decision Log](../architecture/decisions.md) D16). ### Container Log Shipping diff --git a/docs/reference/claude-reference.md b/docs/reference/claude-reference.md index 66b0bc4e16..aba2e78a0d 100644 --- a/docs/reference/claude-reference.md +++ b/docs/reference/claude-reference.md @@ -37,7 +37,7 @@ curl http://localhost:3000/api/v1/readyz # backend (via web proxy) ```text src/synthorg/ - api/ # Litestar REST + WebSocket API, RFC 9457 errors, setup wizard, personality presets, auth/ (role-based access control, HttpOnly cookie sessions, CSRF double-submit, lockout_store, refresh_store, concurrent session enforcement, session store, user presence, OrgRole enum for org config permissions), guards (HumanRole-based + OrgRole-based with department scoping via require_org_mutation), user management (CRUD + org-role grant/revoke), dto_org (request DTOs for company/department/agent mutations), dto_workflow (request/response DTOs for workflow definition and execution operations), services/org_mutations (read-modify-write config mutation service), auto-wiring, lifecycle (auto-promote first owner), bootstrap (agent registry init from config), template packs (list + live-apply), memory admin (fine-tuning pipeline with orchestrator, checkpoint management, preflight checks, run history, embedder queries), optimistic concurrency (ETag/If-Match), TLS config, tiered rate limiting (unauth by IP, auth by user ID), rate_limits/policies (RATE_LIMIT_POLICIES canonical registry of (max_requests, window_seconds) defaults per operation id + per_op_rate_limit_from_policy helper; operator overrides flow through PerOpRateLimitConfig.overrides separately), workflows (visual workflow definition CRUD, validation, YAML export, blueprint listing, blueprint instantiation, version history, diff, rollback), workflow executions (activate, list, get, cancel), ceremony policy (project + per-department query/override, resolved policy with field origins), quality overrides (per-agent quality score override CRUD), reports (on-demand report generation, period listing), notification_dispatcher (fan-out notification sink), training (training plan CRUD, execution, preview, overrides) + api/ # Litestar REST + WebSocket API, RFC 9457 errors, setup wizard, personality presets, auth/ (role-based access control, HttpOnly cookie sessions, CSRF double-submit, lockout / refresh-token / session repositories under persistence/{sqlite,postgres}/, concurrent session enforcement, user presence, OrgRole enum for org config permissions), guards (HumanRole-based + OrgRole-based with department scoping via require_org_mutation), user management (CRUD + org-role grant/revoke), dto_org (request DTOs for company/department/agent mutations), dto_workflow (request/response DTOs for workflow definition and execution operations), services/org_mutations (read-modify-write config mutation service), auto-wiring, lifecycle (auto-promote first owner), bootstrap (agent registry init from config), template packs (list + live-apply), memory admin (fine-tuning pipeline with orchestrator, checkpoint management, preflight checks, run history, embedder queries), optimistic concurrency (ETag/If-Match), TLS config, tiered rate limiting (unauth by IP, auth by user ID), rate_limits/policies (RATE_LIMIT_POLICIES canonical registry of (max_requests, window_seconds) defaults per operation id + per_op_rate_limit_from_policy helper; operator overrides flow through PerOpRateLimitConfig.overrides separately), workflows (visual workflow definition CRUD, validation, YAML export, blueprint listing, blueprint instantiation, version history, diff, rollback), workflow executions (activate, list, get, cancel), ceremony policy (project + per-department query/override, resolved policy with field origins), quality overrides (per-agent quality score override CRUD), reports (on-demand report generation, period listing), notification_dispatcher (fan-out notification sink), training (training plan CRUD, execution, preview, overrides) backup/ # Backup/restore orchestrator, scheduler, retention, handlers/ budget/ # Cost tracking, budget enforcement, quota degradation (including synchronous peek for routing-time selector hints), CFO optimization, trend analysis, budget forecasting, configurable currency formatting, risk budget (cumulative risk-unit tracking, risk scoring integration, risk check, risk records), automated reporting (periodic comprehensive reports, spending/performance/task-completion/risk-trends templates, report scheduling config), coordination metrics (9 empirical metrics: efficiency, overhead, error amplification, message density, redundancy, Amdahl ceiling, straggler gap, token/speedup ratio, message overhead), project cost aggregates (durable per-project lifetime cost totals surviving retention pruning) cli/ # Python CLI module (superseded by top-level cli/ Go binary) diff --git a/docs/reference/comparison.md b/docs/reference/comparison.md new file mode 100644 index 0000000000..aeb0a713ed --- /dev/null +++ b/docs/reference/comparison.md @@ -0,0 +1,268 @@ +--- +title: Framework Comparison +description: >- + How SynthOrg compares to every notable agent orchestration + framework, platform, and research project. +--- + + + +# Framework Comparison + +How SynthOrg compares to agent orchestration frameworks, platforms, and research projects. + +Last updated: 2026-05-01 + +**Legend:** +✔ Full support | ~ Partial support | - Not supported | ⏲ Planned + +!!! tip "Interactive Version" + For a filterable, sortable version of this comparison, visit the [interactive comparison page](https://synthorg.io/compare/). + +## Organization & Coordination + +| Framework | Category | License | Pricing | Self-Hosted | Org Structure | Multi-Agent | Task Delegation | Human-in-the-Loop | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ✔ | ~ | ✔ | ✔ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ✔ | ✔ | ✔ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ✔ | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | - | ✔ | ✔ | ✔ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ✔ | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ✔ | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | - | ~ | ~ | ✔ | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ✔ | ~ | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | ~ | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | ~ | ✔ | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ✔ | ✔ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ✔ | ~ | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | ✔ | ✔ | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | ✔ | ~ | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | ~ | ✔ | ✔ | ~ | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ~ | ~ | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | - | ~ | ✔ | ✔ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | - | ~ | ~ | ✔ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | - | ~ | ~ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | - | ~ | ~ | ~ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | - | ~ | ~ | ✔ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | - | ✔ | ✔ | ✔ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | - | ✔ | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | - | ~ | ~ | ~ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | - | ✔ | ~ | ✔ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | - | - | - | - | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | - | - | - | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | - | ~ | ~ | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | - | - | - | ~ | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | - | ~ | ~ | ~ | + +## Technical Capabilities + +| Framework | Category | License | Pricing | Self-Hosted | Memory | Tool Use | Security | Workflows | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ✔ | ✔ | ~ | ~ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ✔ | ✔ | ~ | ~ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | ✔ | ✔ | ~ | ✔ | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | ✔ | ✔ | - | ✔ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | ✔ | ✔ | ~ | ✔ | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | - | ✔ | ~ | ~ | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ✔ | ✔ | ~ | ~ | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | ~ | ✔ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | ~ | - | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | ~ | ✔ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ✔ | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ~ | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ✔ | ✔ | - | ✔ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | ~ | ~ | - | ~ | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | ~ | ~ | ✔ | ✔ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | ~ | ✔ | ~ | ~ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ✔ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | ~ | ~ | ~ | ✔ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | ~ | ✔ | ✔ | ✔ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | ✔ | ~ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | ✔ | ✔ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ~ | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | ~ | ✔ | ~ | ✔ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | ~ | ✔ | - | ✔ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | - | ✔ | ~ | - | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | - | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | ~ | ✔ | - | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | ~ | ✔ | ~ | - | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | ~ | ✔ | ~ | ~ | + +## Operations & Tooling + +| Framework | Category | License | Pricing | Self-Hosted | Budget / Cost | Observability | Dashboard | CLI | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ~ | ✔ | ✔ | ~ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | ~ | ✔ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | - | ~ | ~ | - | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | - | ~ | ~ | ✔ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | - | ✔ | ~ | - | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ✔ | - | - | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ~ | ✔ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | - | ~ | - | - | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | - | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | ~ | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | ~ | - | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ~ | ~ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ~ | ✔ | ✔ | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ~ | ~ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | ~ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ~ | - | - | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ✔ | - | ~ | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | - | ~ | - | - | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ~ | - | ~ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ~ | ~ | ✔ | ~ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | - | - | - | ~ | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | ~ | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | - | ✔ | ✔ | ✔ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | - | ~ | ✔ | ~ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ✔ | ~ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ✔ | ✔ | - | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | ✔ | ✔ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | - | ~ | ✔ | ~ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | - | ~ | ✔ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | - | ~ | ✔ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | - | ✔ | ~ | ✔ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | ~ | ✔ | ✔ | ✔ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | - | ✔ | ✔ | ✔ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ~ | ✔ | ✔ | - | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | - | ~ | - | ✔ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | ~ | ✔ | - | ~ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | - | ~ | ~ | ~ | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ~ | ~ | ✔ | ✔ | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | ✔ | ~ | ✔ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | - | ~ | - | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | - | ~ | ~ | - | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | - | ~ | ~ | ~ | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | - | ~ | - | ~ | + +## Maturity + +| Framework | Category | License | Pricing | Self-Hosted | Production Ready | Templates | +|:----------|:---------|:--------|:--------|:-----------:|:---:|:---:| +| [**SynthOrg**](https://synthorg.io) | Virtual Org Simulator | BUSL-1.1 | Depends | ✔ | ~ | ✔ | +| [CrewAI](https://crewai.com) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | +| [AutoGen](https://microsoft.github.io/autogen/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | ~ | +| [LangGraph](https://www.langchain.com/langgraph) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | +| [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | - | +| [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/) | Multi-Agent Framework | MIT | Free | ✔ | ✔ | ✔ | +| [Google ADK](https://google.github.io/adk-docs/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ~ | +| [PydanticAI](https://ai.pydantic.dev/) | Multi-Agent Framework | MIT | Open-core | ~ | ~ | ~ | +| [CAMEL](https://www.camel-ai.org) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | - | +| [smolagents](https://huggingface.co/docs/smolagents) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | - | +| [AG2](https://ag2.ai) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | - | +| [Agno](https://www.agno.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ~ | +| [Letta](https://www.letta.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ✔ | +| [Agency Swarm](https://agency-swarm.ai/) | Multi-Agent Framework | MIT | Open-core | ✔ | ~ | ~ | +| [Mastra](https://mastra.ai) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | ~ | +| [Langroid](https://langroid.github.io/langroid/) | Multi-Agent Framework | MIT | Free | ✔ | ~ | - | +| [Strands Agents](https://strandsagents.com/) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ✔ | - | +| [BeeAI Framework](https://beeai.dev) | Multi-Agent Framework | Apache-2.0 | Free | ✔ | ~ | ~ | +| [Haystack](https://haystack.deepset.ai/) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ✔ | ~ | +| [ChatDev](https://chatdev.ai/) | Virtual Org Simulator | Apache-2.0 | Free | ✔ | ~ | ✔ | +| [MetaGPT](https://docs.deepwisdom.ai/) | Virtual Org Simulator | MIT | Open-core | ✔ | - | - | +| [Inngest AgentKit](https://www.inngest.com) | Multi-Agent Framework | Apache-2.0 | Open-core | ✔ | ~ | - | +| [Temporal](https://temporal.io) | Workflow Engine | MIT | Open-core | ✔ | ✔ | ~ | +| [n8n](https://n8n.io) | Workflow Engine | SUL-1.0 | Depends | ✔ | ✔ | ✔ | +| [Dify](https://dify.ai) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | +| [Restack](https://www.restack.io/) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ~ | +| [Flyte](https://flyte.org) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ✔ | ~ | +| [Flowise](https://flowiseai.com) | Workflow Engine | Apache-2.0 | Open-core | ✔ | ~ | ✔ | +| [Langflow](https://www.langflow.org) | Workflow Engine | MIT | Open-core | ✔ | ~ | ~ | +| [AutoGPT](https://agpt.co) | Workflow Engine | PolyForm-Shield-1.0.0 | Open-core | ✔ | ~ | ~ | +| [Motia](https://motia.dev) | Workflow Engine | ELv2 | Depends | ✔ | ~ | ~ | +| [LangSmith](https://smith.langchain.com) | Commercial Platform | Proprietary | Paid | ~ | ✔ | ~ | +| [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) | Commercial Platform | Proprietary | Paid | - | ✔ | ~ | +| [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) | Commercial Platform | Proprietary | Paid | - | ✔ | ~ | +| [Agentforce](https://www.salesforce.com/agentforce/) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | +| [Relevance AI](https://relevanceai.com) | Commercial Platform | Proprietary | Paid | - | ✔ | ✔ | +| [Julep](https://julep.ai) | Workflow Engine | Apache-2.0 | Free | ✔ | ~ | ~ | +| [LlamaIndex Workflows](https://www.llamaindex.ai/) | Developer Tool | MIT | Open-core | ~ | ~ | ~ | +| [Composio](https://composio.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | - | +| [E2B](https://e2b.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ✔ | ✔ | +| [OpenHands](https://openhands.dev) | Developer Tool | MIT | Open-core | ✔ | ~ | ~ | +| [Rivet](https://rivet.ironcladapp.com) | Developer Tool | MIT | Free | ✔ | ~ | ~ | +| [Vercel AI SDK](https://ai-sdk.dev) | Developer Tool | Apache-2.0 | Open-core | ✔ | ~ | - | +| [Llama Stack](https://llamastack.github.io/) | Developer Tool | MIT | Free | ✔ | ~ | ~ | +| [Atomic Agents](https://github.com/BrainBlend-AI/atomic-agents) | Developer Tool | MIT | Free | ✔ | ~ | - | + +## Project Links + +- **SynthOrg** -- [Website](https://synthorg.io) -- [Repository](https://github.com/Aureliolo/synthorg) +- **CrewAI** -- [Website](https://crewai.com) -- [Repository](https://github.com/crewAIInc/crewAI) +- **AutoGen** -- [Website](https://microsoft.github.io/autogen/) -- [Repository](https://github.com/microsoft/autogen) +- **LangGraph** -- [Website](https://www.langchain.com/langgraph) -- [Repository](https://github.com/langchain-ai/langgraph) +- **OpenAI Agents SDK** -- [Website](https://openai.github.io/openai-agents-python/) -- [Repository](https://github.com/openai/openai-agents-python) +- **Semantic Kernel** -- [Website](https://learn.microsoft.com/en-us/semantic-kernel/) -- [Repository](https://github.com/microsoft/semantic-kernel) +- **Google ADK** -- [Website](https://google.github.io/adk-docs/) -- [Repository](https://github.com/google/adk-python) +- **PydanticAI** -- [Website](https://ai.pydantic.dev/) -- [Repository](https://github.com/pydantic/pydantic-ai) +- **CAMEL** -- [Website](https://www.camel-ai.org) -- [Repository](https://github.com/camel-ai/camel) +- **smolagents** -- [Website](https://huggingface.co/docs/smolagents) -- [Repository](https://github.com/huggingface/smolagents) +- **AG2** -- [Website](https://ag2.ai) -- [Repository](https://github.com/ag2ai/ag2) +- **Agno** -- [Website](https://www.agno.com) -- [Repository](https://github.com/agno-agi/agno) +- **Letta** -- [Website](https://www.letta.com) -- [Repository](https://github.com/letta-ai/letta) +- **Agency Swarm** -- [Website](https://agency-swarm.ai/) -- [Repository](https://github.com/VRSEN/agency-swarm) +- **Mastra** -- [Website](https://mastra.ai) -- [Repository](https://github.com/mastra-ai/mastra) +- **Langroid** -- [Website](https://langroid.github.io/langroid/) -- [Repository](https://github.com/langroid/langroid) +- **Strands Agents** -- [Website](https://strandsagents.com/) -- [Repository](https://github.com/strands-agents/sdk-python) +- **BeeAI Framework** -- [Website](https://beeai.dev) -- [Repository](https://github.com/i-am-bee/beeai-framework) +- **Haystack** -- [Website](https://haystack.deepset.ai/) -- [Repository](https://github.com/deepset-ai/haystack) +- **ChatDev** -- [Website](https://chatdev.ai/) -- [Repository](https://github.com/OpenBMB/ChatDev) +- **MetaGPT** -- [Website](https://docs.deepwisdom.ai/) -- [Repository](https://github.com/FoundationAgents/MetaGPT) +- **Inngest AgentKit** -- [Website](https://www.inngest.com) -- [Repository](https://github.com/inngest/agent-kit) +- **Temporal** -- [Website](https://temporal.io) -- [Repository](https://github.com/temporalio/temporal) +- **n8n** -- [Website](https://n8n.io) -- [Repository](https://github.com/n8n-io/n8n) +- **Dify** -- [Website](https://dify.ai) -- [Repository](https://github.com/langgenius/dify) +- **Restack** -- [Website](https://www.restack.io/) -- [Repository](https://github.com/restackio) +- **Flyte** -- [Website](https://flyte.org) -- [Repository](https://github.com/flyteorg/flyte) +- **Flowise** -- [Website](https://flowiseai.com) -- [Repository](https://github.com/FlowiseAI/Flowise) +- **Langflow** -- [Website](https://www.langflow.org) -- [Repository](https://github.com/langflow-ai/langflow) +- **AutoGPT** -- [Website](https://agpt.co) -- [Repository](https://github.com/Significant-Gravitas/AutoGPT) +- **Motia** -- [Website](https://motia.dev) -- [Repository](https://github.com/MotiaDev/motia) +- **LangSmith** -- [Website](https://smith.langchain.com) +- **Amazon Bedrock Agents** -- [Website](https://aws.amazon.com/bedrock/agents/) +- **Vertex AI Agent Builder** -- [Website](https://cloud.google.com/products/agent-builder) -- [Repository](https://github.com/google/adk-python) +- **Agentforce** -- [Website](https://www.salesforce.com/agentforce/) -- [Repository](https://github.com/salesforce/agent-sdk) +- **Relevance AI** -- [Website](https://relevanceai.com) +- **Julep** -- [Website](https://julep.ai) -- [Repository](https://github.com/julep-ai/julep) +- **LlamaIndex Workflows** -- [Website](https://www.llamaindex.ai/) -- [Repository](https://github.com/run-llama/llama_index) +- **Composio** -- [Website](https://composio.dev) -- [Repository](https://github.com/ComposioHQ/composio) +- **E2B** -- [Website](https://e2b.dev) -- [Repository](https://github.com/e2b-dev/E2B) +- **OpenHands** -- [Website](https://openhands.dev) -- [Repository](https://github.com/OpenHands/OpenHands) +- **Rivet** -- [Website](https://rivet.ironcladapp.com) -- [Repository](https://github.com/Ironclad/rivet) +- **Vercel AI SDK** -- [Website](https://ai-sdk.dev) -- [Repository](https://github.com/vercel/ai) +- **Llama Stack** -- [Website](https://llamastack.github.io/) -- [Repository](https://github.com/meta-llama/llama-stack) +- **Atomic Agents** -- [Website](https://github.com/BrainBlend-AI/atomic-agents) -- [Repository](https://github.com/BrainBlend-AI/atomic-agents) diff --git a/docs/reference/pluggable-subsystems.md b/docs/reference/pluggable-subsystems.md index 8849d72643..23ecf77286 100644 --- a/docs/reference/pluggable-subsystems.md +++ b/docs/reference/pluggable-subsystems.md @@ -47,12 +47,11 @@ Each subsystem still owns its config discriminator; the registries replace only ### Rollout strategies -- `meta/rollout/clock.py`: `Clock`. - `meta/rollout/roster.py`: `OrgRoster`. - `meta/rollout/group_aggregator.py`: `GroupSignalAggregator`. - `meta/rollout/inverse_dispatch.py`: `RollbackHandler` + 4 mutator protocols. - `meta/factory.py::build_rollout_strategies()` + `build_rollback_executor()`. -- All plumbed through frozen `SelfImprovementConfig`, with safe defaults (`RealClock`, `NoOpOrgRoster`, null aggregator) so the behaviour is opt-in. +- All plumbed through frozen `SelfImprovementConfig`, with safe defaults (`SystemClock` from `synthorg.core.clock`, `NoOpOrgRoster`, null aggregator) so the behaviour is opt-in. ### API rate limits diff --git a/docs/roadmap/index.md b/docs/roadmap/index.md index 0ceea20f5e..f245459b17 100644 --- a/docs/roadmap/index.md +++ b/docs/roadmap/index.md @@ -2,7 +2,7 @@ ## Current Status -SynthOrg is in **active development**. The core subsystems are built, tested (25,000+ unit tests, 80%+ coverage), and integrated through a REST + WebSocket API, React 19 dashboard, and Go CLI. See the [releases page](https://github.com/Aureliolo/synthorg/releases) for the latest tagged build. +SynthOrg is in **active development**. The core subsystems are built, tested (27,000+ tests, 80%+ coverage), and integrated through a REST + WebSocket API, React 19 dashboard, and Go CLI. See the [releases page](https://github.com/Aureliolo/synthorg/releases) for the latest tagged build. What works today: diff --git a/src/synthorg/core/evidence.py b/src/synthorg/core/evidence.py index 1c58468e11..8d9078576e 100644 --- a/src/synthorg/core/evidence.py +++ b/src/synthorg/core/evidence.py @@ -48,8 +48,8 @@ class RecommendedAction(BaseModel): class EvidencePackageSignature(BaseModel): """Approver signature over an EvidencePackage. - Produced by the audit chain (new module in #1268) using - ML-DSA-65 (FIPS 204) or equivalent quantum-safe signature scheme. + Produced by the audit chain using ML-DSA-65 (FIPS 204) or equivalent + quantum-safe signature scheme. Attributes: approver_id: Identity of the approver. diff --git a/src/synthorg/engine/middleware/coordination_protocol.py b/src/synthorg/engine/middleware/coordination_protocol.py index 59a3aec3f2..ada2b2281c 100644 --- a/src/synthorg/engine/middleware/coordination_protocol.py +++ b/src/synthorg/engine/middleware/coordination_protocol.py @@ -45,8 +45,8 @@ class CoordinationMiddlewareContext(BaseModel): dispatch_result: Result of dispatch execution (set after phase 5). status_rollup: Aggregated subtask status rollup (set after phase 6). phases: Phase results accumulated so far. - task_ledger: TaskLedger populated by #1257 middleware. - progress_ledger: ProgressLedger populated by #1257 middleware. + task_ledger: TaskLedger populated by coordination middleware. + progress_ledger: ProgressLedger populated by coordination middleware. metadata: Middleware-to-middleware data pass-through. """ @@ -77,11 +77,11 @@ class CoordinationMiddlewareContext(BaseModel): ) task_ledger: TaskLedger | None = Field( default=None, - description="TaskLedger from #1257 middleware", + description="TaskLedger from coordination middleware", ) progress_ledger: ProgressLedger | None = Field( default=None, - description="ProgressLedger from #1257 middleware", + description="ProgressLedger from coordination middleware", ) metadata: dict[str, Any] = Field( default_factory=dict, diff --git a/src/synthorg/engine/trajectory/efficiency_ratios.py b/src/synthorg/engine/trajectory/efficiency_ratios.py index 4b98974814..8c64eea072 100644 --- a/src/synthorg/engine/trajectory/efficiency_ratios.py +++ b/src/synthorg/engine/trajectory/efficiency_ratios.py @@ -2,8 +2,7 @@ Defines ``IdealTrajectoryBaseline`` (per-task-type ideal reference) and ``EfficiencyRatios`` (per-run ratios against that baseline), -including verbosity, structural erosion, and PTE metrics folded -from #1244 (SlopCodeBench) and #1248 (PTE). +including verbosity, structural erosion, and Program Test Efficiency (PTE) metrics. """ from typing import Literal diff --git a/src/synthorg/hr/evaluation/extractors/efficiency.py b/src/synthorg/hr/evaluation/extractors/efficiency.py index 2a39771b6c..25f291c187 100644 --- a/src/synthorg/hr/evaluation/extractors/efficiency.py +++ b/src/synthorg/hr/evaluation/extractors/efficiency.py @@ -2,8 +2,7 @@ Lifts cost / time / token efficiency sub-metrics from the performance snapshot's 30d window (or 7d fallback). Honors the -runtime kill-switch resolver added by #1648 for the cost and -latency sub-metrics: +runtime kill-switch resolver for the cost and latency sub-metrics: - ``hr.evaluation_cost_enabled`` gates the ``cost`` sub-metric. - ``hr.evaluation_latency_enabled`` gates the ``time`` sub-metric. @@ -35,8 +34,7 @@ class EfficiencyMetricExtractor: """Extract cost / time / tokens efficiency sub-metrics. Accepts an optional ``ConfigResolver`` so the operator-facing - cost/latency kill switches added in #1648 keep working after - the inline-Efficiency block is hoisted out of ``EvaluationService``. + cost/latency kill switches stay reachable from the extractor seam. """ __slots__ = ("_config_resolver",) diff --git a/src/synthorg/hr/evaluation/metric_extractor_protocol.py b/src/synthorg/hr/evaluation/metric_extractor_protocol.py index 49999222b7..6f2fb2245d 100644 --- a/src/synthorg/hr/evaluation/metric_extractor_protocol.py +++ b/src/synthorg/hr/evaluation/metric_extractor_protocol.py @@ -161,10 +161,9 @@ class MetricExtractor(Protocol): records, feedback, audit counts, trust level, ...) and return an ``ExtractedMetrics``. The composite scorer handles the rest. - ``extract`` is async because some extractors (notably - Efficiency, after rebase against the kill-switch resolver - landed in #1648) need to await ``ConfigResolver`` calls. CPU-bound - extractors simply ``return`` directly. + ``extract`` is async because some extractors (notably Efficiency) + need to await ``ConfigResolver`` calls for dynamic configuration. + CPU-bound extractors simply ``return`` directly. """ @property diff --git a/src/synthorg/providers/management/preset_override_service.py b/src/synthorg/providers/management/preset_override_service.py index 517eb5b76b..9f2c60bb98 100644 --- a/src/synthorg/providers/management/preset_override_service.py +++ b/src/synthorg/providers/management/preset_override_service.py @@ -15,7 +15,7 @@ ``synthorg.providers.presets.get_preset(name)`` entry point still serves the in-code preset directly; consumers that need overrides applied call this service. Migrating every caller to consult the -override at read time is a follow-on task tracked in #1642. +override at read time is a planned enhancement. """ from datetime import UTC, datetime diff --git a/tests/unit/api/test_exception_handlers.py b/tests/unit/api/test_exception_handlers.py index 2cc2e3c3e1..37818ec9b9 100644 --- a/tests/unit/api/test_exception_handlers.py +++ b/tests/unit/api/test_exception_handlers.py @@ -821,7 +821,7 @@ def test_http_exception_non_string_detail_is_coerced(self) -> None: class TestNormalizeStatusCode: - """``_normalize_status_code`` warning paths added in #1659.""" + """``_normalize_status_code`` warning path validation tests.""" def test_invalid_type_logs_warning(self) -> None: """Non-numeric, non-string types (e.g. ``object``) emit a warning.""" diff --git a/tests/unit/budget/test_cost_record.py b/tests/unit/budget/test_cost_record.py index e40b5ba2e2..d4f47ee010 100644 --- a/tests/unit/budget/test_cost_record.py +++ b/tests/unit/budget/test_cost_record.py @@ -314,7 +314,7 @@ def test_factory(self) -> None: @pytest.mark.unit class TestCostRecordAnalyticsFields: - """New per-call analytics fields added in #227.""" + """Per-call analytics fields validation.""" def _base(self) -> CostRecord: return CostRecord( diff --git a/tests/unit/engine/artifacts/test_service.py b/tests/unit/engine/artifacts/test_service.py index bdf4511e8a..b3e596a84b 100644 --- a/tests/unit/engine/artifacts/test_service.py +++ b/tests/unit/engine/artifacts/test_service.py @@ -1,8 +1,8 @@ """Unit tests for :class:`ArtifactService`. -Verifies that the audit events emitted by the service are the new -``API_ARTIFACT_*`` constants (renamed from ``PERSISTENCE_ARTIFACT_*`` -in #1562) and that mutations only emit when they actually mutate. +Verifies that the audit events emitted by the service use the +``API_ARTIFACT_*`` constants and that mutations only emit when they +actually mutate. """ import pytest diff --git a/tests/unit/observability/test_events.py b/tests/unit/observability/test_events.py index 597de9e840..ff075528da 100644 --- a/tests/unit/observability/test_events.py +++ b/tests/unit/observability/test_events.py @@ -312,12 +312,11 @@ def test_all_domain_modules_discovered(self) -> None: "verification", "versioning", "web", - # Added in #1395 -- logging convention cleanup: + # Logging convention namespaces: "capture", "fine_tune", "propagation", - # Added in #1436 to keep sink internal errors out of - # the audited "security.*" namespace. + # Separate from the audited "security.*" namespace. "audit_chain", # Trace handler lifecycle + config validation events. "tracing", @@ -328,8 +327,7 @@ def test_all_domain_modules_discovered(self) -> None: # StrategyRegistry and the persistence/memory backend # registries that consolidate factory dispatch. "registry", - # Added in #1688 for ``GeneralRetryHandler`` constructor - # validation logs (max_attempts / base / cap rejection). + # Resilience handler validation and event logging. "resilience", } discovered = {info.name for info in pkgutil.iter_modules(events.__path__)} diff --git a/tests/unit/settings/test_definitions_config_bridge.py b/tests/unit/settings/test_definitions_config_bridge.py index 00480f258e..e479dd03f7 100644 --- a/tests/unit/settings/test_definitions_config_bridge.py +++ b/tests/unit/settings/test_definitions_config_bridge.py @@ -1,11 +1,9 @@ """Parametrized coverage for the config-bridge settings. -This is a single-file replacement for eleven per-namespace files -that would have contained near-identical assertions. Each row of -``_EXPECTED`` describes one newly-added setting from #1398/#1400: -expected namespace, key, type, default, numeric bounds, restart -requirement, and required presence of ``yaml_path`` / ``group`` / -``description``. +Each row of ``_EXPECTED`` describes one settings entry covering the +namespace bridge: expected namespace, key, type, default, numeric +bounds, restart requirement, and required presence of ``yaml_path`` / +``group`` / ``description``. Behaviour is asserted against the live registry after triggering registration via ``import synthorg.settings.definitions``. From 695ab2b9702f68d1039d3fab4cce712f05e37388 Mon Sep 17 00:00:00 2001 From: Aurelio <19254254+Aureliolo@users.noreply.github.com> Date: Sat, 2 May 2026 11:50:56 +0200 Subject: [PATCH 2/2] fix: babysit round 1, 1 finding (1 gemini) Coordination protocol docstring uses specific middleware class names: - src/synthorg/engine/middleware/coordination_protocol.py:48-49,80-84: "TaskLedger populated by coordination middleware" becomes "TaskLedger populated by TaskLedgerMiddleware"; same shape for ProgressLedger -> ProgressLedgerMiddleware. The specific classes are registered at engine/middleware/_defaults.py:80,82 and live in engine/middleware/coordination_constraints.py:57,133. Naming the concrete middleware is more informative than the generic category. Skipped findings (logged for audit): - coderabbitai/inline at docs/DESIGN_SPEC.md:46 ("British English UI default" vs APP_LOCALE_FALLBACK = 'en'). The DESIGN_SPEC row already separates UI text spelling (British English; colour, behaviour, ...) from runtime Intl formatting (locale-resolved with 'en' fallback). docs/design/internationalization.md:7 explicitly distinguishes the two; the runtime constant is for number/date formatting, not UI text. - coderabbitai/outside-diff at docs/reference/claude-reference.md:38-69 (replace text-fenced package tree with Mermaid). The CLAUDE.md rule is "Never use text blocks with ASCII/Unicode box-drawing characters for diagrams"; both conditions are conjunctive. The block contains zero box-drawing characters; it is a structured description list with multi-line per-directory descriptions, which Mermaid would not render legibly. --- src/synthorg/engine/middleware/coordination_protocol.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/synthorg/engine/middleware/coordination_protocol.py b/src/synthorg/engine/middleware/coordination_protocol.py index ada2b2281c..524013ac4b 100644 --- a/src/synthorg/engine/middleware/coordination_protocol.py +++ b/src/synthorg/engine/middleware/coordination_protocol.py @@ -45,8 +45,8 @@ class CoordinationMiddlewareContext(BaseModel): dispatch_result: Result of dispatch execution (set after phase 5). status_rollup: Aggregated subtask status rollup (set after phase 6). phases: Phase results accumulated so far. - task_ledger: TaskLedger populated by coordination middleware. - progress_ledger: ProgressLedger populated by coordination middleware. + task_ledger: TaskLedger populated by TaskLedgerMiddleware. + progress_ledger: ProgressLedger populated by ProgressLedgerMiddleware. metadata: Middleware-to-middleware data pass-through. """ @@ -77,11 +77,11 @@ class CoordinationMiddlewareContext(BaseModel): ) task_ledger: TaskLedger | None = Field( default=None, - description="TaskLedger from coordination middleware", + description="TaskLedger from TaskLedgerMiddleware", ) progress_ledger: ProgressLedger | None = Field( default=None, - description="ProgressLedger from coordination middleware", + description="ProgressLedger from ProgressLedgerMiddleware", ) metadata: dict[str, Any] = Field( default_factory=dict,