Skip to content
Merged
44 changes: 44 additions & 0 deletions docs/contributing/architecture/data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,50 @@ Operational notes:
arbitrary shell syntax, and package runtime bundles are loaded from published
artifacts rather than a mounted checkout.

### Direct Artifacts git publishes

Saved package source can also be edited through Artifacts git remotes directly.
`package_get_git_remote` resolves package identity to `entity_sources`, mints a
short-lived Artifacts repo token, and returns both a plain remote and setup
commands that pass the token through `http.extraHeader`.

After an external `git push`, `package_publish_external_push` reconciles the
current Artifacts default-branch HEAD with `entity_sources.published_commit`.
The RepoSession Durable Object clones that commit, checks that it is a
fast-forward unless `allow_force` is set, runs `runRepoChecks(...)`, and then
calls `publishFromExternalRef(...)`.

`publishFromExternalRef(...)` owns the post-receive publish transaction:

- run manifest, dependency, bundle, typecheck, and lint checks before mutation
- advance `entity_sources.published_commit`
- write the `PublishedSourceSnapshot` and manifest snapshot to
`BUNDLE_ARTIFACTS_KV`
- roll the D1 commit pointer back if KV snapshot persistence fails
- rebuild saved package projections, bundle artifacts, vector search entries,
retriever manifests, package jobs, and services through
`refreshSavedPackageProjection(...)`

The same helper is used by the existing repo-session publish path after it has
pushed the session commit to the source Artifacts repo.

### Reconcile cron

`packages/worker/src/jobs/reconcile-artifacts-pushes.ts` is a safety net for
external pushes that were not followed by an explicit
`package_publish_external_push` call. The Worker scheduled handler runs every
five minutes (`wrangler.jsonc` `*/5 * * * *`), selects a small batch of stale
`entity_sources` rows by `last_external_check_at`, resolves the Artifacts
default-branch HEAD, and calls the same external publish path when HEAD differs
from `published_commit`.

The reconcile loop is idempotent: if another caller publishes the same commit
first, the publish path returns `already_published`. Check failures and
non-fast-forward results leave D1/KV untouched and are counted in the one-line
metrics log. Once per day during the 03:00 UTC cron window, reconcile also calls
`revokeStaleArtifactsTokens(...)` for checked repos to clean up expired
Artifacts tokens.

Production note:

- Released `wrangler` `4.83.0` warns that the documented Artifacts Worker
Expand Down
29 changes: 28 additions & 1 deletion docs/contributing/packages-and-manifests.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,16 +188,43 @@ Retriever implementations should truncate or paginate before returning.

## Repo-backed workflow

Package source is edited and published through the repo session capabilities.
Package source is edited and published through repo-backed flows.

- prefer `repo_run_commands` for package changes
- `repo_run_commands` accepts a newline-separated parsed git-command string, not
arbitrary shell; keep agent-facing guidance aligned with the deployed
capability schema
- use `package_get_git_remote` and `package_publish_external_push` when a human
or autonomous agent should drive a normal git client directly against the
package's Cloudflare Artifacts repo
- open repo sessions by package identity when possible
- for an existing package, treat the repo snapshot as the durable source of
truth

## External Artifacts pushes

Saved package source repos are real Cloudflare Artifacts git repositories.
`package_get_git_remote` mints a short-lived read or write token for the
canonical source repo and returns both a plain remote URL and setup commands
that use `http.extraHeader` for secret-bearing credentials.

After a direct `git push`, `package_publish_external_push` resolves the
package's default-branch HEAD, opens a transient repo session checkout at that
commit, and uses `publishFromExternalRef` to run the same package checks before
advancing `entity_sources.published_commit`. Check failures return the failed
checks and do not mutate D1, KV snapshots, published bundle artifacts, package
projections, or vectors. Non-fast-forward external heads are refused unless the
caller passes `allow_force: true`.

The scheduled reconcile job in
`packages/worker/src/jobs/reconcile-artifacts-pushes.ts` is a safety net for
pushed-but-unpublished commits. Every five minutes it scans a small batch of
stale `entity_sources` rows, compares Artifacts HEAD with `published_commit`,
and calls the same external publish path when they differ.
`entity_sources.last_external_check_at` throttles the scan. At 03:00 UTC the job
also asks each checked repo to revoke expired Artifacts tokens through
`revokeStaleArtifactsTokens`.

## Search and discovery

Search returns packages as the saved-entity unit.
Expand Down
61 changes: 61 additions & 0 deletions docs/use/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,67 @@ Use:
- `package_get` and `package_list` to inspect saved packages
- `repo_run_commands` to edit, check, and publish repo-backed package source
after it exists using parsed, git-only command forms rather than shell
- `package_get_git_remote` and `package_publish_external_push` when you want a
normal git client to clone, edit, push, and then ask Kody to reconcile the
pushed Artifacts HEAD

## Edit a saved package via direct git push

Saved package source is backed by a Cloudflare Artifacts git repository. You can
edit it with a normal git client without round-tripping each file change through
`package_save` or `repo_run_commands`.

1. Mint a short-lived remote credential:

```json
{
"package_id": "pkg_123",
"scope": "write",
"ttl_seconds": 1800
}
```

Call `package_get_git_remote` with either `package_id` or `kody_id`. The
result includes the plain remote URL, an authenticated one-line clone URL, an
`Authorization: Bearer ...` extra header, and setup commands that use
`git -c http.extraHeader=...` so the token does not need to be saved in shell
history or `.git/config`.

2. Clone and edit:

```bash
git -c http.extraHeader='Authorization: Bearer art_v1_...' clone \
https://<account>.artifacts.cloudflare.net/git/default/<repo>.git \
my-package
cd my-package
# edit files
git add .
git commit -m "fix: update package behavior"
git -c http.extraHeader='Authorization: Bearer art_v1_...' push origin HEAD:<defaultBranch>
```

Use the default branch returned by `package_get_git_remote` for
`<defaultBranch>`.

3. Publish the pushed Artifacts HEAD:

```json
{
"package_id": "pkg_123"
}
```

Call `package_publish_external_push`. Kody checks the pushed tree server-side
before recording the new published version, writing the published source
snapshot, rebuilding package bundle artifacts, and refreshing search
projections. If the pushed HEAD is already current, the tool returns
`already_published`. If checks fail, it returns `checks_failed` with the
failed check entries and leaves the underlying storage state unchanged.

Choose the narrowest token scope that fits the task. Use `read` for inspection
or local diffing, and `write` only when the git client needs to push. Keep TTLs
short for autonomous agents and CI-style helpers; the tool accepts 60 seconds to
24 hours and defaults to 30 minutes.

## Search and discovery

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
ALTER TABLE entity_sources
ADD COLUMN last_external_check_at TEXT;

CREATE INDEX IF NOT EXISTS idx_entity_sources_external_check
ON entity_sources(last_external_check_at);
13 changes: 13 additions & 0 deletions packages/worker/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ import { getRequestIp } from '#app/audit-log.ts'
import { handleCapabilityReindexRequest } from './capability-maintenance.ts'
import { handleJobReindexRequest } from './job-maintenance.ts'
import { handleMemoryReindexRequest } from './memory-maintenance.ts'
import { reconcileArtifactsPushes } from './jobs/reconcile-artifacts-pushes.ts'
import { CodemodeFetchGateway } from '#mcp/fetch-gateway.ts'
import {
connectorSessionKey,
Expand Down Expand Up @@ -360,6 +361,18 @@ const workerHandler = {
) {
await handleInboundEmail(message, env, ctx)
},
async scheduled(
controller: ScheduledController,
env: Env,
_ctx: ExecutionContext,
) {
const baseUrl = env.APP_BASE_URL ?? 'https://kody.local'
await reconcileArtifactsPushes({
env,
baseUrl,
now: new Date(controller.scheduledTime),
})
},
Comment thread
cursor[bot] marked this conversation as resolved.
} satisfies ExportedHandler<Env>

export default Sentry.withSentry(
Expand Down
Loading
Loading