test(sdk-e2e): self-contained embedded-PG ICU + connector-sync & watcher-reaction gates#1021
Closed
buremba wants to merge 66 commits into
Closed
test(sdk-e2e): self-contained embedded-PG ICU + connector-sync & watcher-reaction gates#1021buremba wants to merge 66 commits into
buremba wants to merge 66 commits into
Conversation
Functional sugar over ConnectorRuntime: declare a connector as a spec with per-feed sync and per-action execute handlers (feed/action keys derived from the record keys). Lowers to a ConnectorRuntime subclass so connector-worker's child-runner runs it unchanged; handler closures are stripped from the serializable definition. Unit test mirrors child-runner's findRuntimeClass detection to prove a bundled default export is picked up unchanged.
…ector Adds an optional authenticate(ctx) hook to the functional spec, lowered to ConnectorRuntime.authenticate. When omitted, the connector inherits the base behavior (throws). Closes the gap where interactive-auth connectors required the class form, making defineConnector feature-complete vs the class.
Scaffolds @lobu/sdk (Apache-2.0): defineConfig/defineAgent/defineEntityType/ defineRelationshipType/defineWatcher/defineConnection/defineAuthProfile + secret(), and re-exports defineConnector + TypeBox Type from connector-sdk so a project imports its whole authoring surface from one package. Producers are pure branded data with typed handles (EntityType -> relationship rules, Agent -> watcher); the CLI loader (next slice) maps them to DesiredState, which stays CLI-private per the apply-IR boundary. Excluded from root tsconfig like the other workspace packages (own tsconfig); wired into build:packages + root test.
Adds mapProjectToDesiredState — the single place that translates the public @lobu/sdk authoring objects (defineConfig default export) into the apply-private DesiredState IR. Maps agents (providers -> installedProviders/modelSelection, network, resolved provider keys), entity/relationship types (typed handles -> slugs), watchers (agent handle -> id, sources record -> array, notification), and connections/auth profiles (connector class -> key, secret() -> $VAR + required-secrets). The esbuild entrypoint loader + apply wiring follow next.
Adds loadDesiredStateFromConfig: esbuild-bundles lobu.config.ts (relative imports inlined; node_modules externalized so @lobu/sdk + @lobu/connector-sdk resolve from the project), imports the bundle to read the defineConfig() default export, and maps it via mapProjectToDesiredState. Dynamic imports are allow-listed in desired-state.ts (esbuild loaded lazily; bundle imported by URL). E2E test bundles a real fixture config end-to-end. Not yet wired into the apply command (next).
lobu apply now loads DesiredState from the TypeScript entrypoint when a lobu.config.ts is present, falling back to lobu.toml otherwise. Downstream apply logic (required-secrets gate, org resolution, diff, mutations) is source-agnostic — it operates on DesiredState.
- BLOCKER (#976): @lobu/sdk re-exported defineConnector/Type through connector-sdk's barrel, which flakily fails under bun's ESM linker and broke the cli unit suite. Source Type/Static from @sinclair/typebox directly and deep-import defineConnector via a new connector-sdk '/define-connector' subpath export (bypasses the barrel). Full cli suite now 320 pass, 0 fail. - Thread --only into the TS loader/mapper so 'apply --only agents' doesn't demand connector secrets (matches the TOML loader). - Port the TOML structural validations into the mapper: connection/auth-profile slug patterns, cron schedules, duplicate feed keys, and forbidding credentials on interactive (oauth_account/browser_session) auth kinds — fail loud in the CLI before any remote mutation. - Register @lobu/sdk in release-please-config, bump-version, and publish-packages so it versions and publishes. Deferred (task #7): local defineConnector definitions authored in lobu.config.ts are not yet uploaded (connectors.definitions stays []); needs connector file-discovery wired into the TS loader.
…arity) cronError now rejects schedules firing more than once a minute, matching the TOML loader + server validation, so a sub-minute cron fails loud in the CLI instead of at server mutation. Closes the last codex parity gap (slice 3 now 93%).
…ig.ts
The TypeScript apply path (loadDesiredStateFromConfig) always set
connectors.definitions to []. A connector authored in ./connectors and
referenced by a connection resolved to a key but its source was never
uploaded, so apply failed "not installed".
Discover ./connectors/*.connector.ts (non-recursive, sorted, files only)
and ship each as a DesiredConnectorDefinition{ key: null, sourcePath,
sourceCode } — the same key-null contract the YAML loader uses for
auto-discovered connector files. apply-cmd then compiles each sourcePath on
the CLI and uploads it via install_connector; the server resolves the real
key. Skipped under --only agents|memory, matching the mapper.
Key resolution is intentionally deferred to the server (no eager
compile/instantiate at load time, which would force esbuild + installed
deps + module side effects on every load, including --dry-run).
…onfig The TS authoring path (defineAgent + mapProjectToDesiredState) only covered providers + network allowed/denied + the memory schema, while the TOML loader's buildAgentSettings lifts much more. Applying a migrated example would silently drop config (office-bot's egress judges, org metadata, etc.), blocking the TOML-deletion slice. Close that gap. defineAgent gains: network.judged + judges, egress, tools (preApproved/allowed/denied/strict), guardrails, nixPackages, mcpServers (typed type/authScope unions), plus preview and dir (consumed by a later `lobu run`/loader slice — not mapped into cloud settings; a test guards the preview non-leak). defineConfig gains orgName/orgDescription/organizationId. mapProjectToDesiredState now produces the same AgentSettings shape as buildAgentSettings: judgedDomains deduped by domain (last-wins), egressConfig, preApprovedTools + toolsConfig, guardrails, nixConfig, mcpServers (with the same loose cast for authScope/oauth), and collects $VAR/secret() refs from mcp headers/env + oauth clientId/clientSecret into the apply secrets gate. Org metadata maps into DesiredState.memory. Deferred to follow-ups: agent-dir SOUL/IDENTITY/USER markdown + skills/ loading (file IO + skill merge), platforms, dev.ts preview/dir wiring, example migration, TOML deletion.
buildAgentSettings lifts SOUL.md/IDENTITY.md/USER.md prompt markdown and local skills (project ./skills + per-agent <dir>/skills, with their network/nix/mcp declarations merged into agent settings); the TS config path did neither, so a migrated agent would lose its prompt files and skills. loadDesiredStateFromConfig now reads each agent's dir (defaulting to ./agents/<id>, overridable via defineAgent.dir) using the existing readMarkdown/loadSkillFiles/buildLocalSkills helpers, and merges the result via a new pure mergeAgentDirArtifacts(settings, markdown, skills). The mapper stays file-IO-free (unit-testable); the merge mirrors buildAgentSettings exactly: agent-level network/nix/mcp first, skills on top — allowed/denied/nix unioned+deduped (skill "*" dropped), judged-domains + judges agent-wins on conflict, skill MCP servers add only ids the agent didn't define.
These two DesiredWatcher fields are used by example watchers (8 and 2 uses respectively) but had no defineWatcher equivalent, so a migrated watcher would drop them. Add the SDK fields + mapper passthrough. The executable `reaction` (reaction_script) field still lands in the reactions slice.
defineAgent() accepted `connections` and `schema` but mapProjectToDesiredState ignored them — a silent config drop. Connections and the memory schema are declared at the project level (defineConfig), matching the apply model (there is no agent-scoped association in DesiredState). Remove the dead fields rather than leave them silently ignored; project-level remains the wired path.
The generated client vendors its fetch runtime into src/generated/client/ — no runtime code imports the @hey-api/client-fetch npm package; it is used only by @hey-api/openapi-ts at generation time (openapi-ts.config.ts plugin). Move it to devDependencies so consumers of the published @lobu/client don't install it.
defineWatcher gains `reaction?: string` — a relative POSIX path to a sibling .ts reaction script. loadDesiredStateFromConfig validates + reads it (raw source) and attaches it to DesiredWatcher.reactionScript; apply ships it via the existing setReactionScript and the server compiles it. Mirrors the TOML loader's parseWatcher exactly: relative-POSIX/.ts/no-`..`/under-config-dir/ 256KB checks, present-but-empty rejected (gate on absence, not truthiness), raw-source contract. mapWatcher stays pure; the loader zips project.watchers[i].reaction -> state.watchers[i].reactionScript. Unblocks the 6 example watchers that use reaction_script. (The runAction / operations typing on ReactionClient is a separate enhancement — no example reaction calls connector actions; they only use client.knowledge.)
mapAuthProfile shipped the literal `$VAR` placeholder for env/oauth_app credentials, whereas the TOML loader (loadConnectors) resolves them to the real env value before pushing to the DB — so a TS-config auth profile would write "$GITHUB_CLIENT_SECRET" instead of the secret. Add resolveCredentialValue (mirrors loadConnectors): secret()/$VAR creds resolve against env, the ref is still collected for the apply secrets gate. mcp oauth client_secret keeps the literal pass-through (matching buildAgentSettings). Found migrating lobu-crm.
Migrate every example from lobu.toml + models/*.yaml + connectors/*.yaml to a single TypeScript lobu.config.ts using the @lobu/sdk authoring API (define*). Each was verified to produce a DesiredState byte-identical to the legacy TOML loader (modulo object key order, installedAt timestamps, and the connector sourceFile error-label). Agent dirs, *.connector.ts, and *.reaction.ts files are unchanged (still file-based, referenced from the config). The old toml/yaml files remain for now; they are removed when the TOML loader is deleted in the next commit.
…ig.ts Extract loadProjectConfig (bundle+import lobu.config.ts → SDK Project) and route all consumers through it / the TS loader: apply drops the lobu.toml fallback (always loadDesiredStateFromConfig); doctor, chat, validate, and dev's preview-bot registration read the SDK Project; dev's auto-apply gate keys on lobu.config.ts. Drop writeMemoryOrganizationId — TS projects carry org/ organizationId in defineConfig + the .lobu/project.json link, so apply never rewrites the config file. Update the dryrun + dev tests to lobu.config.ts fixtures (under the worktree so the externalized @lobu/sdk import resolves). The TOML loader (loadDesiredState/loadConnectors/parse*, config/loader, lobu-toml-schema) is now dead and removed in the next commit.
… YAML `lobu memory seed` read [memory] from lobu.toml and entity/relationship/watcher schema from models/*.yaml. Migrate it onto loadDesiredStateFromConfig: org + entity/relationship types now come from lobu.config.ts (same source apply uses, seeding stays idempotent). Drop watcher seeding — watchers are agent-scoped and provisioned by `lobu apply`, not the old entity-scoped models path. The ./data instance seeding (entities + relationships) is unchanged. This removes seed's last dependency on the TOML/YAML loader, unblocking its deletion.
With every consumer (apply, dev, doctor, chat, validate, seed) on
lobu.config.ts, the TOML/YAML loader is dead code. Remove it:
- packages/cli: config/loader.ts + config/schema.ts; the TOML functions in
desired-state.ts (loadDesiredState/loadConfig use, collectEnvRefs,
buildAgentSettings, buildPlatforms, parse{Entity,Relationship,Watcher}Type,
parse*Doc, loadMemoryModels, loadConnectors, rejectUnsupportedAgentShapes,
+ their TOML-only consts). Kept the TS-path + shared helpers (readMarkdown,
loadSkillFiles, buildLocalSkills, isRecord/asString, the connector-config
validators, loadProjectConfig/loadDesiredStateFromConfig). Drop smol-toml.
- packages/core: lobu-toml-schema.ts + its index.ts export block (no server
importers).
- Delete the 2 TOML-loader test files (coverage replaced by map-config.test.ts
+ load-config.test.ts) and the 2 core schema tests.
- Delete the examples' lobu.toml + models/*.yaml + connectors/*.yaml
(lobu.config.ts + *.connector.ts + *.reaction.ts + agent dirs remain).
8339 lines removed. cli/core/server typecheck clean; cli suite green.
gen-landing-snippets read the deleted example lobu.toml + models/*.yaml. Rework it to slice examples/<slug>/lobu.config.ts as text (829 → 472 lines): drop the TOML/YAML parsers + compressors, add a string-literal-aware defineX(...) slicer, and source agentConfig/memorySchema/watcher from the config. examples list + useCases now scan lobu.config.ts. Frontend: agentToml → agentConfig, copy lobu.toml/YAML → lobu.config.ts/TypeScript. Landing build green.
Replace the hand-rolled esbuild-bundle-to-temp-file loader with jiti — the
runtime TS loader Next.js/Nuxt use for *.config.ts. It transpiles on import and
resolves the config's imports (@lobu/sdk, relative reaction/connector files)
from the project, with no bundling and no temp file written into the user's
cwd. Verified jiti produces byte-identical DesiredState to the esbuild loader
across all 12 examples. The dynamic import("jiti") stays lazy + allow-listed.
…-finder, help) The reader side was migrated earlier; this finishes the producer side so the init → apply flow works end to end: - lobu init scaffolds lobu.config.ts (defineConfig/defineAgent via @lobu/sdk) instead of lobu.toml; drops the models/ + data/ YAML scaffolding (schema lives in the config). Round-trip tested through loadDesiredStateFromConfig. - lobu agent add scaffolds the agent dir + prints a defineAgent snippet to paste (no fragile mutation of the user's typed config) instead of appending TOML. - ensure-deps-installed anchors the connector project root on lobu.config.ts. - index.ts help text + the seed description say lobu.config.ts. - Delete init-memory.test.ts (tested the removed TOML scaffolder); cli-ux.test.ts asserts lobu.config.ts. (lobu export still emits YAML — its single-file design is an open question, handled separately.)
…ve lobu export
Replace the YAML-era `lobu export` (partial, agent-less) with
`lobu init --from-org <slug>`: the inverse of `lobu apply`. It fetches the org's
full declared state and emits a clean, runnable project — one lean lobu.config.ts
(handle consts, real object literals, short arrays inlined, empty/default fields
omitted, resources sorted) plus the files it references (agents/<id>/{SOUL,
IDENTITY,USER}.md, skills/<name>/SKILL.md, reactions/<slug>.reaction.ts). Now
includes agents (the gap export punted on). Write-only secrets become
secret("ENV_VAR") placeholders + a .env.example; never real values. Runs after
init's empty-dir guard, so it never overwrites an existing project.
Round-trip gated: the test bootstraps stubbed cloud state, loads the generated
lobu.config.ts via loadDesiredStateFromConfig, and asserts the DesiredState
matches the input. cli 282 pass, tsc clean.
- memory/_lib/schema.ts: drop the now-dead model-file parsing (parseModelYamlFile/expandModelDefinition/validateModel/expandModelSection + the model schema types + their AutoCreateWhenRule/TypeCompiler/yaml imports); only the ./data seed-record schema + validateDataRecord (used by `lobu memory seed`) remain. 487 → 124 lines. - Fix the user-facing "referenced in lobu.toml" message (render.ts) → lobu.config.ts, and stale lobu.toml comments in apply-cmd/dev/diff. No lobu.toml/YAML-config reference remains in cli source.
…locker) A generated lobu.config.ts imports @lobu/sdk, but `lobu init` only added @lobu/connector-sdk to the scaffolded package.json, and the tsconfig only included connectors/** — so a fresh project couldn't resolve @lobu/sdk (jiti `lobu apply` fails) or type-check its config outside this monorepo. Extract a shared scaffoldProjectPackaging() (adds both SDK devDeps + a tsconfig that includes lobu.config.ts/reactions/agents), and call it from BOTH `lobu init` and `lobu init --from-org` (bootstrap wrote no package.json at all). Regression test asserts the scaffolded package.json + tsconfig.
…m lobu.config.ts
Adds a 'delete' diff verb. When the target org is code-managed
(organization.managed_by='code', opted in via 'lobu apply --manage'),
computeDiff({ codeManaged: true }) emits delete rows for entity types,
relationship types, watchers, and connector definitions that exist
remotely but are absent from the config. Data (entity/relationship
instances), connections, auth profiles, feeds, agents, and platforms are
never pruned — they stay 'drift' as before. Connectors still wired to a
surviving remote connection/auth-profile are spared. The unnamed-local-
connector guard suppresses connector prune when keys can't be matched.
Execution runs in reverse-dependency order (watcher -> relationship-type
-> entity-type -> connector) and the server refuses an entity-type delete
while instances exist, so data stays safe. A blast-radius confirm gates
applies that would delete more than 3 definitions; --dry-run never
deletes. UI-managed orgs (default) are unchanged: no delete rows, summary
omits the delete count.
Server managed_by exposure + migration land in the next commit; until
then managed_by is absent and every org stays UI-managed (no prune).
…managed prune Slice 5b — the server half of code-managed prune (CLI half in the prior commit). Adds: - migration 20260522120000: organization.managed_by text NOT NULL DEFAULT 'ui' (CHECK ui|code). The default backfills every existing org to 'ui' so none starts prunable — the 2026-05-20 safety lesson. - /oauth/userinfo organizations[].managed_by, the field the CLI's listOrgs reads to decide codeManaged. - PATCH /api/:orgSlug/organization/managed-by (owner/admin + mcp:admin, same gate as visibility) — the one-time 'lobu apply --manage' opt-in target. Client points setOrgManagedBy at it. Verified against a real ephemeral embedded Postgres (PG18) in managed-by-prune.test.ts (7 tests): migration default + CHECK constraint, userinfo exposure, definition deletes (entity/relationship type, watcher), and that an entity-type delete REFUSES while instances exist — so prune can never cascade into data. CLI client wire contract pinned in client.test.ts.
Final pi sweep: requireRelationshipType (write mode) resolved by global slug with no org preference, so when an org owned a rel-type AND a public type from another org shared the slug, LIMIT 1 could grab the foreign row and the access-denied guard then blocked the org from updating/deleting its OWN type (and broke code-managed prune). Now tenant-first ordered, mirroring read mode. Also org-scoped rtHandleCreate's duplicate check (the unique index is (organization_id, slug), and entity-type create was already org-scoped) so a same-slug foreign public type can't block this org's create. Not a cross-org deletion path (the access-denied guard held) — a correctness/availability fix. Cross-org integration test added (9 prune tests, 7 entity-schema tests green).
…g-scoping Final convergence pi findings on this PR's surface: - init-from-org (bootstrap.ts): an auth profile with no connector_key was emitted as `connector: null`, which crashed the next `lobu apply` (connectorKey(null)). Now skipped with a surfaced warning. Also flagged the credential placeholders with a TODO — keys must be renamed to the connector's auth-schema fields (values are write-only, unrecoverable on bootstrap). - manage_entity_schema: the inverse_type_slug lookups in relationship-type create/update resolved by global slug; made them tenant-first (ORDER BY org-owned), completing the org-scoping hardening of the rel-type handlers this PR already touches. Out of scope (pre-existing, NOT exercised by this PR — documented for a separate hardening pass): manage_feeds delete_feed cancels runs before the ownership check (prune never deletes feeds). CLI 296 tests + 16 schema/prune integration tests green; server typecheck clean.
…rms)
Final pi review caught that the migration silently dropped config-authored
platforms: the SDK had no platforms field, mapAgent hardcoded platforms:[],
yet 'lobu init --platform' still prompted for one + wrote bot-token env
placeholders that 'apply' never consumed (on main, lobu.toml emitted
[[agents.platforms]] which apply created). Per the user, restore them:
- @lobu/sdk: defineAgent({ platforms: [{ type, name?, config, channels? }] }).
- mapAgent: maps platforms to DesiredPlatform with a deterministic stable id
(agentId-type[-name]) so apply matches (noop) instead of recreating; config
/secret() values kept as placeholders + collected into requiredSecrets.
- lobu init: emits a platforms block from the chosen --platform + its config.
- lobu init --from-org: round-trips live platform bindings back to config
(strips the route's embedded key; config -> secret() refs).
Unit-tested (stable-id derivation + secret collection + literal passthrough);
full CLI suite 297 green; SDK + CLI typecheck clean.
…empotent apply) A config-authored chat platform restarted on EVERY `lobu apply`: the diff compared the desired config (`botToken: "$TELEGRAM_BOT_TOKEN"`) against the GET round-trip, where the server redacts the secret (`"***oken"`) and rewrites the $VAR into an internal `secret://…` reference. Neither round-trips, so the config never matched and the platform was needlessly updated + restarted, dropping in-flight messages. Treat a key as unchanged when the desired value is a $VAR placeholder and the remote value is opaque (redacted or secret://), mirroring the auth-profile credentials rule; real (non-secret) config changes still surface as updates.
…noop Locks the idempotent-apply behavior: a $VAR secret placeholder matching a redacted (***) or secret:// remote value is a noop, while a real non-secret config change still surfaces as an update.
…alues
Bootstrap assumed stored platform config kept $VAR placeholders, but the server
rewrites $VAR into an internal secret:// reference and the GET masks it
(***-suffixed). Neither matched the $VAR regex, so a redacted botToken was
emitted as the literal "***oken" — re-applying the generated config would push
that broken value. Recognize redacted (***) and secret:// values and emit a
deterministic secret("<AGENT>_<PLATFORM>_<KEY>") placeholder (collected into
.env.example), mirroring how provider keys round-trip. Non-secret config fields
stay literals.
…lder Final ship-verdict pi caught that config-authored platform secrets were broken: the server's normalizeConfigForStorage persists whatever plaintext the CLI sends as the secret (then swaps it for a secret:// ref), so sending the $VAR placeholder stored a literal '$TELEGRAM_BOT_TOKEN' as the bot token — a dead connection. Fixes: - mapAgent resolves platform config secret()/$VAR to the REAL env value (resolveCredentialValue), mirroring provider keys + auth-profile credentials; the config row still never holds cleartext at rest (server-side secret store). - diffPlatform now treats a config key as unchanged when the REMOTE value is opaque (***/secret://) regardless of the desired form — desired is now the resolved value, so the prior $VAR-keyed check would restart every apply. - mapAgent rejects two platforms whose names slugify to the same stable id. - init-from-org recovers the name from the stable id so NAMED platforms re-derive the same id (no drift/duplicate on re-apply). Unit-tested (resolved values + collision + named round-trip); the prior round-trip test updated to assert the resolved value. 302 CLI tests green.
…authored
The prior comment ('Chat platforms ... are NOT authored here') was added in this
same migration to rationalize dropping platforms; it was never an external
product decision, and platforms were config-driven on main via lobu.toml
[[agents.platforms]]. Now that defineAgent({ platforms }) is restored, the
comment was both stale and the source of a wrong 'platforms are UI-only'
assumption. Fix it to reflect reality.
emitAuthProfile derived the credential KEY from the env-var name
(<SLUG>_VALUE / _CLIENT_SECRET), but lobu apply sends
credentials: { <key>: <value> } and the server validates <key> against
the connector's auth_schema — so a non-schema key is rejected.
Plumb listConnectorDefinitions(true) through fetchOrgState and emit
credentials keyed by each connector's real required auth-schema field
(or all properties when none are required), with a deterministic
<SLUG>_<FIELD> env-var placeholder. Falls back to the prior single
placeholder + TODO when the connector/auth_schema isn't found.
…ing runs handleDeleteFeed cancelled active runs by feed_id BEFORE the org-scoped feed delete confirmed ownership. The run-cancel UPDATE is not org-scoped (runs reach their org only through the feed), so a guessed foreign feed_id could cancel another org's active runs even though the delete then no-ops. Reorder: delete the org-owned feed first, bail on no match, then cancel runs — no cross-org side effect before the ownership check. Adds a cross-org isolation integration test (red before the reorder).
The desired-channels Map in the Slack sync-channels route is keyed by
`${teamId} ${channelId}`. Document why the single-space separator is
collision-safe: the entry regex (`[^/\s]+`) forbids whitespace and
slashes in either component, so neither can contain the delimiter and
distinct (team, channel) pairs always map to distinct keys. The Map is
in-memory and request-scoped (never persisted — the DB keeps team_id and
channel_id as separate columns), so there are no stored keys to migrate.
Comment-only; no behavior change.
The generated hey-api SSE client retries forever by default: a non-OK response (401/404/5xx) or network failure throws inside its read loop, fires onSseError, then sleeps and reconnects with no attempt cap — so a failed stream never terminated and streamEvents' async iterator hung indefinitely. Cap attempts (sseMaxRetryAttempts default 1, overridable via StreamEventsOptions.maxRetryAttempts) and capture the stream error via onSseError into the iterator's error slot so the consumer rejects rather than hanging. Caller-initiated aborts are treated as a clean shutdown, not an error. Adds tests for the 401-rejects and abort-terminates paths (both would hang before this fix).
Replace the persistent per-org `managed_by` provenance flag with a
config-declared `defineConfig({ prune: true })`. When prune is on,
`lobu apply` deletes any org-owned definition (entity/relationship type,
watcher, connector definition) absent from the config — including ones
created via the dashboard/API. Data, connections, auth profiles, and
agents stay exempt. Safety is the existing blast-radius confirm; there is
no applied-set tracking and no server-side state.
Removed:
- migration 20260522120000_organization_managed_by.sql (never shipped)
- PATCH /api/:orgSlug/organization/managed-by endpoint
- managed_by from getUserInfo organizations[]
- RemoteOrg.managed_by, listOrgs parse, setOrgManagedBy
- --manage flag + willManage/flipToCodeManaged opt-in flow
Kept: org/organizationId consistency hard-stop, confirmDeletions
blast-radius gate, the org-owned/public-type filter in computeDiff, and
the entity/relationship-type instance-refusal server gate.
computeDiff: codeManaged -> prune (logic unchanged). SDK + mapper carry
prune into DesiredState.
# Conflicts: # AGENTS.md # packages/cli/src/commands/_lib/apply/__tests__/desired-state-extra.test.ts # packages/cli/src/commands/_lib/apply/apply-cmd.ts # packages/cli/src/commands/_lib/apply/desired-state.ts
CI's typecheck job (make build-packages) and unit job (inline build step) never built packages/sdk, so the CLI typecheck and bun test packages/cli both failed to resolve @lobu/sdk — a cascade of TS2307 + implicit-any + $secret errors and unhandled import errors in unit. Add packages/sdk to both build paths and align the Makefile list with root build:packages (which already builds client + sdk).
apply-cmd.ts destructures `warnings` from loadDesiredStateFromConfig, but
the committed desired-state.ts return type was { state, configPath } only —
a real TS2339 (Property 'warnings' does not exist) that the merge resolution
left uncommitted in the working tree. Restores the field + warnings channel.
…nit path
From a fresh full-diff pi review (bug_free_confidence 22):
- B1 init --from-org emitted secret("…") on the MCP-oauth clientSecret path
without registering the `secret` import → generated config didn't compile.
Couple SecretCollector to ImportTracker so every secret() ref auto-imports;
drop the 4 now-redundant manual imports.use("secret") calls.
- B2 platform secret rotation/removal was a silent noop (opaque remote can't be
diffed). Apply now upserts EVERY desired platform idempotently (mirrors the
provider-key push); the server's PUT decides noop vs restart. Diff also flags
a removed key (present in remote, absent in desired) as a change.
- B3 init --from-org read auth_schema as JSON Schema (.properties), but it's a
ConnectorAuthSchema (.methods). Rewrite authSchemaFields to read env_keys
fields[].key and oauth clientIdKey/clientSecretKey per profile kind.
- B4 relationship-type delete left status='active', so the (org,slug) partial
unique index (WHERE status='active') blocked re-creating the same slug
(prune → re-add). Delete now also sets status='archived' to vacate the index.
- B5 inverse-type lookup wasn't org/public scoped: it could link to and write
the reciprocal back-link onto another tenant's private rel-type. Scope the
lookup to own-org-or-public and only write the back-link when caller owns it.
- B6 init --from-org dropped per-skill network.judge / judges from SKILL.md.
Emit them as the YAML the frontmatter loader reads back.
Tests: CLI unit (MCP-oauth import, oauth_app methods keys, skill judged
domains, platform-key removal) + server integration (rel-type re-create after
delete, inverse tenant isolation x2). Full CLI 314 / server unit 201 green;
prune integration 9/9 against embedded PG.
Found via full lifecycle E2E (init → connector → watcher → apply → run → init-from-org), re-applying a stable config repeatedly: - I1 relationship-type rules churned a perpetual '~ rules' update because the rel-type `list` action omits rules — the diff compared desired rules against an always-empty remote. Hydrate remote rules (new client.listRelationshipTypeRules via the existing list_rules action) for the types the config declares with rules, so a matching set is a true noop. - I2 an omitted optional display name (feeds, connections) churned a perpetual '~ name' update: stringChanged(undefined, 'Ticks') is always true. Add optionalNameChanged — an omitted name means 'no opinion', so the server keeps its derived/stored name and it doesn't diff. - B2 follow-through: the earlier 'upsert every platform every apply' restored rotation but made the server restart the platform on EVERY apply (it can't tell the resolved secret matches the stored secret://), dropping the bot connection each deploy. Revert to upserting only diff-flagged platforms (idempotent — no restart churn); KEEP the diff removal-detection (a removed key is still applied). In-place opaque-secret rotation needs a secret-aware compare on the server upsert (owletto) — tracked as a follow-up. Verified: 7 consecutive applies of a stable config converge to '0 update, 9 noop' (connector re-push is by-design idempotent). CLI 316 green.
…arative rules) Fresh full-diff pi review (bug_free 58) — 2 blockers + 2 bugs, all fixed: - P1 write-mode requireRelationshipType did an unscoped slug lookup and threw 'Access denied' for a slug owned only by another (private) org — an existence oracle. Scope the lookup to ctx.organizationId; a missing own row is now 'not found'. A tenant can only update/delete its OWN types anyway (public foreign types stay referenceable-but-read-only). - P2 relationship-type rules are now fully declarative: upsertRelationshipType reconciles (add_rule for missing, remove_rule by id for extras) against the current remote set, and the snapshot hydrates rules for EVERY config-declared rel-type (incl. those declared with no rules) so dropping all rules is detected. Was add-only → removing a rule never took effect and churned a perpetual 'rules changed' update. - P3 migration archives pre-fix rel-type tombstones (deleted_at set but status='active') so they leave the WHERE status='active' partial unique index and re-create can't collide. - P4 init-from-org platform config emission uses emitKey() so a non-identifier config key (e.g. hyphenated) generates valid lobu.config.ts. Tests: prune integration 10/10 (embedded PG, +foreign-private 'not found'); CLI 316. Live E2E proved the rule reconcile: stable=noop, removal removes + converges, change add+removes + converges.
…UL delimiter Final pi review (bug_free 82, 0 blockers) — 1 medium bug + 1 hygiene: - envVarFor only normalized the slug, not the suffix, so a non-identifier platform config key (e.g. `bot-token`) produced an invalid POSIX env-var name (`BOT_TELEGRAM_BOT-TOKEN`) — the .env key would be rejected and apply's required-secret check would never pass. Normalize both parts. Added a regression test (quoted key + sanitized env var + round-trip). - The rules reconcile rule-key delimiter was an actual NUL byte, which made grep/rg treat client.ts as binary. Replaced with a tab (slugs never contain whitespace, so it's still an unambiguous composite key). CLI 317 green; no NUL bytes remain in cli/src.
Found via live prune E2E (the gap I flagged). With prune:true, apply marked the per-org system entity type $member 'will be deleted' (it's absent from config and can't be declared — the server reserves $ slugs). The delete is then refused because member rows exist, which HALTS the entire apply on first failure — so prune:true apply failed on every run for any org with members (i.e. every real org). If an org somehow had no member rows, it would instead DELETE the system type and corrupt the org. computeDiff now exempts $-prefixed entity/relationship/watcher definitions from prune: they stay ignorable drift in both modes, never delete. Regression test added (prune.test covers the server refusal; diff.test covers the verb). E2E: prune now creates all defs (no halt), $member stays drift, removing lead/knows/w-drop deletes exactly those, kept defs + $member survive, re-apply is a clean noop.
Closes the coverage gap: unit/integration prove config MAPS correctly, but nothing proved the whole SDK path RUNS. scripts/sdk-e2e.sh boots `lobu run` (embedded Postgres — the linux binary ships as an @embedded-postgres optional dep, the engine prod uses), auto-applies a prune:true fixture, and drives a REAL agent turn through a spawned worker against a deterministic mock OpenAI-compatible provider (scripts/sdk-e2e/, no provider key → reproducible). Asserts (non-zero exit → red CI): auto-apply completes (not halted — guards the $member prune-halt class), every definition is created, $member is never pruned, the agent turn returns the mock reply via the worker→secret-proxy→ upstream path, and a stable re-apply is idempotent (0 deletes). Wired as the CI `sdk-e2e` job (Node 22 for isolated-vm; failure fails the PR) and `make test-e2e-sdk`. Verified locally: 5/5 assertions pass.
The sdk-e2e gate boots `lobu run`'s embedded Postgres, whose PG18 binary is dynamically linked against ICU 60 (libicuuc.so.60). ubuntu-latest ships a newer ICU, so initdb failed to load the shared lib. Install the 18.04-era libicu60 from the Ubuntu archive (with a security-mirror fallback) before the gate, and ldd the initdb binary to surface any further missing libs. Prod is unaffected (external Postgres; embedded-postgres is pruned from the app image).
…pawns Embedded PG now boots on CI (libicu60), but the agent turn failed: the orchestrator wraps Linux workers in `systemd-run --user --scope`, which needs a user systemd/dbus session the CI runner doesn't have → worker exited 1. Set LOBU_DISABLE_SYSTEMD_RUN=1 for the gate (it only talks to the loopback mock; not testing the prod network sandbox) and install bubblewrap + enable unprivileged userns for the worker's exec-sandbox. No-op on macOS.
Clears the recurring biome useOptionalChain warning (p.id && p.id.startsWith → p.id?.startsWith) on this PR's platform round-trip code.
…her-reaction gates
Embedded Postgres portability (Task 1): the @embedded-postgres PG18 binaries
are NEEDED-linked against ICU 60 with an rpath of $ORIGIN/../lib, and that lib
dir already ships libicu{uc,i18n,data}.so.60.2 — it was only missing the .so.60
SONAME symlinks the loader resolves. scripts/sdk-e2e/fix-embedded-pg-icu.mjs
creates them (idempotent, Linux-only no-op on macOS), so initdb loads its
bundled ICU with zero system deps. Drops the fragile archive .deb download from
CI; now works identically on a local Linux dev box.
Expanded gate (Task 2):
- Connector sync: a local zero-dep ./connectors/pulse.connector.ts whose sync()
emits one event; a defineConnection wires its feed. The gate triggers an
immediate sync via manage_feeds(trigger_feed), polls the run to completed, and
asserts items_collected>=1 and feed event_count>=1 — proving the compiled
connector actually RUNS and persists, not just that apply mapped it.
- Watcher reaction: a ./reactions/digest.reaction.ts that saves an assertable
SDKE2E_REACTION_OK knowledge event. The gate triggers the watcher (asserts the
run row is enqueued) then deterministically drives read_knowledge ->
complete_window (the fixed-reply mock never produces the complete_window
tool-call) so the reaction fires on the connector-emitted window, and asserts
the side effect via query_sql.
API assertions use an mcp:admin PAT minted with lobu token create against the
local-install org. Gate stays deterministic with generous polling + clear
failures; teardown unchanged.
|
Important Review skippedToo many files! This PR contains 192 files, which is 42 over the limit of 150. To get a review, narrow the scope: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (19)
📒 Files selected for processing (192)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the SDK lifecycle e2e gate (`scripts/sdk-e2e.sh`). Targets `main` to exercise CI; the lead will merge it into `feat/client-sdk` (PR #1015).
Task 1 — portable embedded Postgres (drops the fragile libicu60 .deb)
The @embedded-postgres PG18 binaries (initdb/postgres) are NEEDED-linked against ICU 60 with an rpath of `$ORIGIN/../lib`. That lib dir already ships `libicu{uc,i18n,data}.so.60.2` — it was only missing the `.so.60` SONAME symlinks the loader resolves. New helper `scripts/sdk-e2e/fix-embedded-pg-icu.mjs` creates them (idempotent; Linux-only, a no-op on macOS), so initdb loads its bundled ICU with zero system deps, no LD_LIBRARY_PATH, no .deb download — identical in CI and on a local Linux dev box. Removed the archive `.deb` step from the `sdk-e2e` CI job. Prod is unaffected (external Postgres; embedded-postgres pruned from the app image).
Task 2 — gate now also asserts connector sync + watcher reaction
API assertions use an `mcp:admin` PAT minted via `lobu token create` against the `local-install` org. Stays deterministic (generous polling, clear failures); teardown unchanged. Existing assertions (apply/prune/$member-exemption/turn/idempotency) preserved.
Verified locally on macOS (Node 22, embedded PG): full gate green across repeated runs.
🤖 Generated with Claude Code