Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/design/coordination.md
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,40 @@ worktrees directly; signals the `WorkspaceManager` to act.

**Module**: `src/synthorg/engine/workspace/disk_quota.py`

### Persistent Per-Project Workspace and Push Queue Serialisation

Each project gets a 1:1 persistent git-backed working tree on the
runtime volume. `ProjectWorkspaceService.get_or_provision(project_id)`
materialises the working tree under
`<base_root>/projects/<project_id>/` on first touch via the configured
`GitBackend` (`embedded` default; `local_path` / `external_remote` are
opt-in via config). The tree survives across agents, tasks, and
sessions. `GitBackendConfig.kind` is authoritative: a persisted row
whose kind differs from the live config triggers a re-provision under
the new backend and a `WORKSPACE_BACKEND_KIND_CHANGED` log event.

When N agents finish concurrently on one project, their
merge-to-default-branch + push-to-backend operations route through a
per-project FIFO serial queue (`PushQueueCoordinator`) so concurrent
pushes never collide at the git backend. The queue sits in front of
the `WorkspaceIsolationStrategy` seam, so a future virtual-branch
strategy supplies its own `merge_workspace` without changing the
queue. A conflicted merge resolves the caller future without pushing
(the queue refuses to push a broken default branch). `stop()` drains
in flight then exits cleanly; `WorkspacePushError` distinguishes a
forge-rejection push failure from a local `WorkspaceMergeError`.

Events emitted: `PROJECT_WORKSPACE_PROVISIONED`,
`PROJECT_WORKSPACE_REUSED`, `WORKSPACE_BACKEND_KIND_CHANGED`,
`WORKSPACE_PUSH_QUEUE_ENQUEUED`, `WORKSPACE_PUSH_QUEUE_MERGED`,
`WORKSPACE_PUSH_QUEUE_FAILED`, `WORKSPACE_PUSH_QUEUE_WORKER_FAILED`.

**Modules**:
- `src/synthorg/engine/workspace/project_workspace_service.py`
- `src/synthorg/engine/workspace/git_backend/` (protocol + 3 strategies + factory)
- `src/synthorg/engine/workspace/push_queue.py`
- `src/synthorg/engine/workspace/service.py` (per-project queue cache + `merge_workspace_with_push`)

## Task Decomposability & Coordination Topology

Empirical research on agent scaling
Expand Down
3 changes: 3 additions & 0 deletions docs/design/integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ and optional rate limiting and health check configuration.
| Type | Auth Fields | Health Check |
|------|------------|--------------|
| `github` | `token`, `api_url` | `GET /user` |
| `gitlab` | `token`, `api_url` | `GET /user` |
| `gitea` | `token`, `api_url` | `GET /api/v1/user` |
| `forgejo` | `token`, `api_url` | `GET /api/v1/user` |
| `slack` | `token`, `signing_secret` | `POST auth.test` |
| `smtp` | `host`, `port`, `username`, `password` | SMTP EHLO |
| `database` | `dialect`, `host`, `port`, `username`, `password`, `database` | `SELECT 1` |
Expand Down
5 changes: 5 additions & 0 deletions docs/reference/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ The NotFound hierarchy is driven by a single `NotFoundError` class with domain-s
| 3014 | `AB_TEST_NOT_FOUND` | A/B test record for a proposal |
| 3015 | `BACKUP_NOT_FOUND` | Backup archive |
| 3016 | `MEMORY_ENTRY_NOT_FOUND` | Agent memory entry |
| 3017 | `CONVERSATION_NOT_FOUND` | Conversation record |

All share the same `type` URI; the numeric code is the discriminator.

Expand All @@ -98,6 +99,9 @@ All share the same `type` URI; the numeric code is the discriminator.
| 4011 | `FINE_TUNE_RUN_ACTIVE` | A fine-tune run is already active (start/resume blocked) |
| 4012 | `TRAINING_PLAN_NOT_MODIFIABLE` | Training plan cannot be modified after execution or failure |
| 4013 | `BACKUP_UNRESTARTABLE` | Backup service stopped in an unrestartable state |
| 4014 | `AGENT_RUNTIME_NOT_CONFIGURED` | No LLM provider configured; agent runtime cannot execute |
| 4015 | `CONVERSATION_CLOSED` | Conversation is closed; no further messages or actions accepted |
| 4016 | `PROJECT_WORKSPACE_NOT_PROVISIONED` | Project workspace required but never provisioned by the git backend |
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## Rate Limit (5xxx)

Expand Down Expand Up @@ -131,6 +135,7 @@ All share the same `type` URI; the numeric code is the discriminator.
| 7007 | `INTEGRATION_ERROR` | Non-LLM integration failure |
| 7008 | `OAUTH_ERROR` | OAuth exchange failed |
| 7009 | `WEBHOOK_ERROR` | Webhook receive/replay failure |
| 7010 | `CONVERSATIONAL_PROPOSE_RESPONSE_INVALID` | Chief-of-Staff proposer returned an invalid response |

## Internal (8xxx)

Expand Down
9 changes: 9 additions & 0 deletions docs/reference/pluggable-subsystems.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,15 @@ Domain errors live at `meta/errors.py::RollbackMutationDeniedError` (409) and `U
- `backup/registry.py::PERSISTENCE_BACKUP_HANDLER_REGISTRY`: `StrategyRegistry` keyed on `config.persistence.backend` ("sqlite" / "postgres").
- `backup/factory.py::build_backup_handlers()`: dispatches per `BackupComponent` and uses the registry for the persistence handler.

### Git backend storage strategy

- `engine/workspace/git_backend/protocol.py`: `GitBackend` `@runtime_checkable` Protocol, with `ProvisionResult`/`PushResult`/`FetchResult` frozen result models.
- `engine/workspace/git_backend/config.py`: `GitBackendConfig` (frozen) with `kind: GitBackendType` discriminator and `GitBackendDeps` (collaborators not safe in frozen config: `workspace_base_root`, `connection_catalog`, `secret_backend`, `clock`).
- `engine/workspace/git_backend/embedded.py::EmbeddedGitBackend` (safe default: bare repo self-hosted on the persistent volume, no external dependency).
- `engine/workspace/git_backend/local_path.py::LocalPathGitBackend` (bring-your-own on-disk git repository, push/fetch are no-ops because the on-disk repo is the durable store).
- `engine/workspace/git_backend/external_remote.py::ExternalRemoteGitBackend` (GitHub / GitLab / Gitea / Forgejo resolved via the connection catalog; ships protocol + thin clone/push/fetch glue; deep OAuth hardening is a tracked follow-up).
- `engine/workspace/git_backend/factory.py::build_git_backend()`: `StrategyRegistry[GitBackend]` keyed on `GitBackendType`. Missing required deps fail fast at construction with `GitBackendConfigError`. Wired at boot in `api/app.py::_install_runtime_services` under the `has_persistence` gate, alongside `ProjectWorkspaceService`.

## Services are a distinct pattern (not pluggable subsystems)

A **service** wraps one or more repositories to keep controllers thin and centralise audit logging, and MAY orchestrate multiple repositories (e.g. `WorkflowService` spans `workflow_definitions` + `workflow_versions`; `MemoryService` spans fine-tune checkpoints + runs + settings).
Expand Down
6 changes: 6 additions & 0 deletions scripts/_ghost_wiring_manifest.txt
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,9 @@ ENFORCED build_work_pipeline #1960 -- called by workers.runtime_builder._build_r
ENFORCED build_chief_of_staff_proposer #1968 -- called by api.app._wire_chief_of_staff_proposer behind propose_enabled + provider switch; constructs ChiefOfStaffProposer which parks approval-gated WorkItems for the conversational interface
ENFORCED TaskBoardEntryAdapter #1963 -- constructed by engine.pipeline.entry.factory.build_work_entry_adapter on the TASK_BOARD arm; wired at boot by engine.pipeline.entry.boot.wire_real_task_board_entry; drives the spine for human-filed board tasks (POST /tasks)
ENFORCED ObjectiveEntryAdapter #1964 -- built at boot by engine.pipeline.entry.boot.wire_real_objective_entry; fed by POST /objectives
ENFORCED ProjectWorkspaceService #1974 -- constructed in api/app.py _install_runtime_services; per-project persistent git-backed workspace provisioning
ENFORCED build_git_backend #1974 -- called in api/app.py _install_runtime_services; builds the configured GitBackend strategy (embedded default)
ENFORCED EmbeddedGitBackend #1974 -- default GitBackend strategy built by engine/workspace/git_backend/factory.py::_build_embedded
ENFORCED LocalPathGitBackend #1974 -- GitBackend strategy built by engine/workspace/git_backend/factory.py::_build_local_path for the LOCAL_PATH discriminator
ENFORCED ExternalRemoteGitBackend #1974 -- GitBackend strategy built by engine/workspace/git_backend/factory.py::_build_external_remote for the EXTERNAL_REMOTE discriminator
ENFORCED PushQueueCoordinator #1974 -- constructed per-project by engine/workspace/service.py::WorkspaceIsolationService._get_or_create_queue; serialises merge+push to the backend
2 changes: 2 additions & 0 deletions scripts/schema_drift_baseline.txt
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ column:preset_overrides:updated_at:TEXT:TIMESTAMPTZ:SQLite has no TIMESTAMPTZ; c
column:principle_overrides:created_at:TEXT:TIMESTAMPTZ:SQLite has no TIMESTAMPTZ; column stores TEXT carrying ISO-8601 with explicit +00:00/Z suffix, normalised to UTC at write time
column:principle_overrides:updated_at:TEXT:TIMESTAMPTZ:SQLite has no TIMESTAMPTZ; column stores TEXT carrying ISO-8601 with explicit +00:00/Z suffix, normalised to UTC at write time
column:project_cost_aggregates:last_updated:TEXT:TIMESTAMPTZ:SQLite has no TIMESTAMPTZ; column stores TEXT carrying ISO-8601 with explicit +00:00/Z suffix, normalised to UTC at write time
column:project_workspaces:created_at:TEXT:TIMESTAMPTZ:SQLite has no TIMESTAMPTZ; column stores TEXT carrying ISO-8601 with explicit +00:00/Z suffix, normalised to UTC at write time
column:project_workspaces:updated_at:TEXT:TIMESTAMPTZ:SQLite has no TIMESTAMPTZ; column stores TEXT carrying ISO-8601 with explicit +00:00/Z suffix, normalised to UTC at write time
column:projects:deadline:TEXT:TIMESTAMPTZ:SQLite has no TIMESTAMPTZ; column stores TEXT carrying ISO-8601 with explicit +00:00/Z suffix, normalised to UTC at write time
column:projects:task_ids:TEXT:JSONB:SQLite has no JSONB type; column stores TEXT carrying json.dumps(...) (see schema header column inventory)
column:projects:team:TEXT:JSONB:SQLite has no JSONB type; column stores TEXT carrying json.dumps(...) (see schema header column inventory)
Expand Down
41 changes: 41 additions & 0 deletions src/synthorg/api/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -1052,6 +1052,47 @@ async def _install_runtime_services() -> None:
if env_workspace_root is not None:
app_state.set_agent_workspace_root(env_workspace_root)

# Per-project persistent workspace substrate. The git backend is
# config-selected (embedded default, no external dep);
# ProjectWorkspaceService provisions one persistent git-backed
# tree per project under the workspace base. Persistence-less
# boots (test fixtures, dev apps with no DB) skip wiring -- the
# service is optional and gates on ``has_project_workspace_service``.
if app_state.has_persistence and app_state.project_workspace_service is None:
# Guard against partial-startup retry: this hook fires once
# the persistence layer is connected, but ``build_runtime_services``
# below is fallible and a re-entry after its failure would
# otherwise hit the ``_set_once`` guard inside
# ``set_project_workspace_service`` and fail with
# "already configured" instead of cleanly retrying the
# runtime-services build.
from synthorg.engine.workspace.git_backend import ( # noqa: PLC0415
GitBackendConfig,
GitBackendDeps,
build_git_backend,
)
from synthorg.engine.workspace.project_workspace_service import ( # noqa: PLC0415
ProjectWorkspaceService,
)

git_backend_config = GitBackendConfig()
git_backend = build_git_backend(
git_backend_config,
GitBackendDeps(
workspace_base_root=app_state.agent_workspace_root,
clock=app_state.clock,
),
)
app_state.set_project_workspace_service(
ProjectWorkspaceService(
base_root=app_state.agent_workspace_root,
repo=app_state.persistence.project_workspaces,
git_backend=git_backend,
config=git_backend_config,
clock=app_state.clock,
),
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.

try:
services = await build_runtime_services(
app_state,
Expand Down
28 changes: 28 additions & 0 deletions src/synthorg/api/state.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,9 @@
WorkflowRollbackService,
)
from synthorg.engine.workflow.webhook_bridge import WebhookEventBridge
from synthorg.engine.workspace.project_workspace_service import (
ProjectWorkspaceService,
)
from synthorg.integrations.connections.catalog import ConnectionCatalog
from synthorg.integrations.health.prober import HealthProberService
from synthorg.integrations.mcp_catalog.installations import (
Expand Down Expand Up @@ -270,6 +273,7 @@ class AppState(AppStateServicesMixin):
"_personality_service",
"_preset_override_service",
"_project_facade_service",
"_project_workspace_service",
"_prometheus_collector",
"_provider_audit_service",
"_provider_health_tracker",
Expand Down Expand Up @@ -463,6 +467,10 @@ def __init__( # noqa: PLR0913, PLR0915
# process-stable temp directory so dev / empty-company runs
# still have a valid absolute workspace.
self._agent_workspace_root: Path | None = None
# Per-project persistent workspace provisioning service. Wired
# at boot behind the provider switch; ``None`` for empty-company
# / dev apps with no runtime services installed.
self._project_workspace_service: ProjectWorkspaceService | None = None
# Guards the double-checked locking on first-access lazy wiring
# of worker_execution_service / experiment_service. Both
# properties may be invoked from concurrent request handlers
Expand Down Expand Up @@ -978,6 +986,26 @@ def set_agent_workspace_root(self, path: Path) -> None:
"Agent workspace root",
)

@property
def project_workspace_service(self) -> ProjectWorkspaceService | None:
"""Per-project persistent workspace provisioner, or ``None``.

Wired at boot behind the provider-present switch; ``None`` for
empty-company / dev apps where no runtime services are installed.
"""
return self._project_workspace_service

def set_project_workspace_service(
self,
service: ProjectWorkspaceService,
) -> None:
"""Attach the project workspace service (once-only, startup)."""
self._set_once(
"_project_workspace_service",
service,
"Project workspace service",
)

@property
def experiment_service(self) -> ExperimentService:
"""Return the A/B experiment service, auto-wiring the default.
Expand Down
15 changes: 15 additions & 0 deletions src/synthorg/core/enums.py
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,21 @@ class ProjectStatus(StrEnum):
CANCELLED = "cancelled"


class GitBackendType(StrEnum):
"""Discriminator selecting how a project's git repository is stored.

``EMBEDDED`` is the safe default: the product self-hosts a bare repo
on the persistent volume, with no external dependency. ``LOCAL_PATH``
targets a caller-supplied repository on disk. ``EXTERNAL_REMOTE``
delegates to an external forge remote resolved via the connection
catalogue.
"""

EMBEDDED = "embedded"
EXTERNAL_REMOTE = "external_remote"
LOCAL_PATH = "local_path"


class ToolAccessLevel(StrEnum):
"""Access level for tool permissions.

Expand Down
1 change: 1 addition & 0 deletions src/synthorg/core/error_taxonomy.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ class ErrorCode(IntEnum):
BACKUP_UNRESTARTABLE = 4013
AGENT_RUNTIME_NOT_CONFIGURED = 4014
CONVERSATION_CLOSED = 4015
PROJECT_WORKSPACE_NOT_PROVISIONED = 4016

# 5xxx -- rate_limit
RATE_LIMITED = 5000
Expand Down
59 changes: 59 additions & 0 deletions src/synthorg/core/project_workspace.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
"""Persistent per-project workspace domain model.

A :class:`ProjectWorkspace` is the 1:1 mapping between a
:class:`~synthorg.core.project.Project` and the persistent, git-backed
working tree that survives across agents, tasks and sessions. The row
records where the workspace lives on the persistent volume and which git
backend provisioned it, so a session restart re-locates the same
directory without re-deriving environment precedence and a configured
backend switch can be detected against the persisted kind.
"""

from typing import Final

from pydantic import AwareDatetime, BaseModel, ConfigDict, Field

from synthorg.core.enums import GitBackendType # noqa: TC001
from synthorg.core.types import NotBlankStr

_DEFAULT_BRANCH: Final[str] = "main"


class ProjectWorkspace(BaseModel):
"""Persistent git-backed workspace bound to a single project.

Attributes:
project_id: Owning project identifier (primary key, 1:1 with
``Project.id``).
workspace_path: Absolute on-volume path of the project working
tree (``<base>/projects/<project_id>``). Persisted so a
restart re-locates the same directory deterministically.
git_backend_kind: Which backend provisioned the repository; lets
a config switch detect a mismatch against the live config.
remote_ref: External remote URL or connection-catalog name for
the ``EXTERNAL_REMOTE`` backend; ``None`` for embedded/local.
default_branch: Default branch the backend provisions and the
merge/push queue targets.
created_at: Provisioning timestamp (timezone-aware, UTC).
updated_at: Last mutation timestamp (timezone-aware, UTC).
"""

model_config = ConfigDict(frozen=True, allow_inf_nan=False, extra="forbid")

project_id: NotBlankStr = Field(description="Owning project identifier (PK)")
workspace_path: NotBlankStr = Field(
description="Absolute on-volume path of the project working tree",
)
git_backend_kind: GitBackendType = Field(
description="Backend that provisioned this workspace",
)
remote_ref: NotBlankStr | None = Field(
default=None,
description="External remote URL / connection name (external backend)",
)
default_branch: NotBlankStr = Field(
default=NotBlankStr(_DEFAULT_BRANCH),
description="Default branch the backend provisions",
)
created_at: AwareDatetime = Field(description="Provisioning timestamp (UTC)")
updated_at: AwareDatetime = Field(description="Last mutation timestamp (UTC)")
Loading
Loading