Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 18 additions & 11 deletions .claude/skills/orchestrate/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -598,9 +598,11 @@ So, on hitting a harness defect mid-run:
from inside a run.
3. At the end of the run, if the ledger is non-empty, create ONE Linear ticket in the
Backlog project (`757e1ced-43e9-4459-b7eb-3ade25dc1919`, team ORB) carrying EVERY harness
defect that run found, each with its evidence and its workaround. One ticket per RUN,
never one per defect; it must pass `node tools/check-ticket.mjs --issue ORB-N` like any
other ticket. An empty ledger creates nothing: silence is correct when nothing broke.
defect that run found, each with its evidence and its workaround. Its title must contain
the fixed marker `Harness defect ledger`, which lets `check-ticket.mjs` distinguish its
children from ordinary child tickets. One ticket per RUN, never one per defect; it must
pass `node tools/check-ticket.mjs --issue ORB-N` like any other ticket. An empty ledger
creates nothing: silence is correct when nothing broke.
4. Name that ticket in the closing report alongside the per-ticket ledger, so a run ends as
a work list rather than as silence.

Expand All @@ -617,14 +619,19 @@ the same defect then taxes every later run until somebody schedules it.

So once every ticket in the run is merged or stopped, and NOTHING is still executing:

1. **The run ticket stays the record; each defect gets its own CHILD ticket to be fixed
from.** The aggregate ticket above is one per RUN and cannot carry N repair PRs: it would
close on the first merge, leaving the rest with no live issue to launch from, attach to or
close, and `launch-worker.mjs` derives the worktree name and branch from the issue, so two
PRs off one issue collide before that even matters. So file one child ticket per ledger
entry (`orca linear create --parent <run ticket>`), each passing `check-ticket.mjs`, each
describing ONE defect. A ledger with exactly one entry needs no child: the run ticket is
already that shape.
1. **The run ticket stays the durable record of every ledger entry; only recurring or blocking
defects get a CHILD ticket to be fixed from.** Create a child
(`orca linear create --parent <run ticket>`) when the defect occurred on at least three
tickets or blocked the run. Record every entry below that threshold in the parent ticket's
body with its evidence. If later evidence raises a recorded entry to three occurrences or
shows that it blocked a run, create its child then from the accumulated evidence.
Each child describes ONE defect, passes `check-ticket.mjs`, and contains this fixed line:
`Ledger occurrence: <count>; blocked: no|<what it blocked>`. Use `no` only for a
non-blocking defect; a blocking value must affirmatively name the outcome using
`blocked the <operation>` or `the <operation> was blocked`. The aggregate ticket above
cannot carry N repair PRs: it would close on the first merge, leaving the rest with no live
issue to launch from, attach to or close, and `launch-worker.mjs` derives the worktree name
and branch from the issue, so two PRs off one issue collide before that even matters.
2. Work the children one PR per ticket, through the same worker machinery as any ticket. The
run is over, so editing `tools/`, this skill or `.claude/orchestrator.json` no longer
changes a contract anything is executing.
Expand Down
102 changes: 102 additions & 0 deletions tools/check-ticket.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@

import { execFileSync } from "node:child_process"
import { existsSync, readFileSync } from "node:fs"
import { request } from "node:https"
import { homedir } from "node:os"
import { join } from "node:path"

const USAGE = `usage: check-ticket.mjs --issue ORB-12 | --file body.md

Expand Down Expand Up @@ -38,6 +41,12 @@ const REQUIRED_SECTIONS = [

/** D4: one ticket = one repo. repo:both is a defect, not a label. */
const REPO_LABELS = ["repo:ui", "repo:api", "repo:landing"]
const LEDGER_OCCURRENCE_THRESHOLD = 3
const LEDGER_OCCURRENCE_FORMAT = "Ledger occurrence: <count>; blocked: no|<what it blocked>"
const LEDGER_PARENT_MARKER = /\bHarness defect ledger\b/i
const LINEAR_PARENT_TIMEOUT_MS = 5_000
const AFFIRMATIVE_BLOCKING_CLAIM =
/^(?:blocked|halted|stopped|prevented)\s+(?:the|a|an)\s+\S(?:.*\S)?$|^(?:the|a|an)\s+\S(?:.*\S)?\s+(?:was|were)\s+(?:blocked|halted|stopped|prevented)(?:\s+.*\S)?$/i
Comment thread
thomasluizon marked this conversation as resolved.
Resolved

/**
* A criterion that quantifies over an OPEN set has no provable finish line, so review can never
Expand Down Expand Up @@ -136,6 +145,86 @@ const validateLabels = (labels) => {
}
}

const parentFromOrca = (issue, relations) => {
if (Object.hasOwn(issue, "parent")) return issue.parent
const parentRelation = relations.find((relation) => {
const relationship = relation.relationship ?? relation.type
return relationship === "parent" || relationship === "childOf"
})
return parentRelation
? parentRelation.relatedIssue ?? parentRelation.issue ?? parentRelation
: null
}

const isLedgerParent = (parent) => {
return LEDGER_PARENT_MARKER.test(`${parent?.title ?? ""}\n${parent?.description ?? ""}`)
}

const readLinearParent = async (issue) => {
if (!issue.id) return null
const keyPath = join(process.env.USERPROFILE || homedir(), ".linear-api-key")
if (!existsSync(keyPath)) throw new Error(`missing ${keyPath}`)
const apiKey = readFileSync(keyPath, "utf8").trim()
if (!apiKey) throw new Error(`${keyPath} is empty`)
const requestBody = JSON.stringify({
query: "query($id: String!) { issue(id: $id) { parent { id identifier title description } } }",
variables: { id: issue.id },
})
const response = await new Promise((resolve, reject) => {
const linearRequest = request("https://api.linear.app/graphql", {
method: "POST",
agent: false,
headers: {
Authorization: apiKey,
"Content-Type": "application/json",
"Content-Length": Buffer.byteLength(requestBody),
},
timeout: LINEAR_PARENT_TIMEOUT_MS,
}, (linearResponse) => {
const chunks = []
linearResponse.on("data", (chunk) => chunks.push(chunk))
linearResponse.on("end", () => resolve({
status: linearResponse.statusCode ?? 0,
body: Buffer.concat(chunks).toString("utf8"),
}))
})
linearRequest.on("error", reject)
linearRequest.on("timeout", () => {
linearRequest.destroy(new Error(`Linear parent lookup timed out after ${LINEAR_PARENT_TIMEOUT_MS}ms`))
})
linearRequest.end(requestBody)
})
const payload = JSON.parse(response.body)
if (response.status < 200 || response.status >= 300 || payload.errors) {
throw new Error(payload.errors?.[0]?.message ?? `HTTP ${response.status}`)
}
return payload.data?.issue?.parent ?? null
}

const validateLedgerOccurrence = (body) => {
const occurrenceLine = body.split(/\r?\n/).find((line) => line.startsWith("Ledger occurrence:"))
if (!occurrenceLine) {
problems.push(`ledger child is missing the fixed line: ${LEDGER_OCCURRENCE_FORMAT}`)
return
}
const parsedLine = occurrenceLine.match(/^Ledger occurrence: (\d+); blocked: ([^\r\n]+)$/)
if (!parsedLine) {
problems.push(`ledger occurrence line is unparseable; expected: ${LEDGER_OCCURRENCE_FORMAT}`)
return
}
const occurrenceCount = Number(parsedLine[1])
const blockingClaim = parsedLine[2].trim()
if (blockingClaim === "no") {
if (occurrenceCount < LEDGER_OCCURRENCE_THRESHOLD) {
problems.push(`ledger child states ${occurrenceCount} occurrences, below the threshold of ${LEDGER_OCCURRENCE_THRESHOLD}, without an affirmative blocking claim naming what it blocked`)
}
return
}
if (!AFFIRMATIVE_BLOCKING_CLAIM.test(blockingClaim)) {
problems.push(`ledger child blocking value must be literal no or an affirmative claim naming what it blocked`)
}
}

const mode = process.argv[2]
const target = process.argv[3]

Expand Down Expand Up @@ -164,6 +253,19 @@ if (mode === "--file") {
validateBody(body)
validateLabels(labels)
const relations = parsedResult.relations ?? issue.relations ?? []
let parent = parentFromOrca(issue, relations)
const parentIsClassifiable = parent === null ||
typeof parent?.title === "string" ||
typeof parent?.description === "string"
if (!parentIsClassifiable) {
try {
parent = await readLinearParent(issue)
} catch (error) {
console.error(`check-ticket: could not read the Linear parent relation: ${error instanceof Error ? error.message : String(error)}`)
process.exit(2)
}
}
if (isLedgerParent(parent)) validateLedgerOccurrence(body)
const blockedBy = relations.filter((r) => r.relationship === "blockedBy" || r.type === "blockedBy")
if (mentionsIssueDependency(body) && blockedBy.length === 0) {
problems.push("body PROSE mentions a dependency but the issue has no blockedBy relation; the DAG is explicit, never inferred from titles (6.2)")
Expand Down
180 changes: 180 additions & 0 deletions tools/test-tools.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,31 @@ const BASH = resolveBash()
const ORCA_SHIM = stage(
"orca-shim.cjs",
`const { spawnSync } = require("node:child_process")
const { EventEmitter } = require("node:events")
const { appendFileSync, existsSync, readFileSync, rmSync } = require("node:fs")
if (process.env.ORBIT_LINEAR_PARENT_STUB) {
const https = require("node:https")
const { syncBuiltinESMExports } = require("node:module")
const stub = JSON.parse(process.env.ORBIT_LINEAR_PARENT_STUB)
https.request = (_url, options, callback) => {
const linearRequest = new EventEmitter()
linearRequest.end = () => {
process.nextTick(() => {
const linearResponse = new EventEmitter()
linearResponse.statusCode = stub.status ?? 200
callback(linearResponse)
linearResponse.emit("data", Buffer.from(JSON.stringify(stub.body)))
linearResponse.emit("end")
})
}
linearRequest.destroy = (error) => process.nextTick(() => linearRequest.emit("error", error))
if (stub.requireTimeout && options.timeout !== 5000) {
process.nextTick(() => linearRequest.emit("error", new Error("missing Linear parent timeout")))
}
return linearRequest
}
syncBuiltinESMExports()
}
const argv = process.argv.slice(1)
if (argv[0] && existsSync(argv[0])) return
const line = argv.join(" ")
Expand Down Expand Up @@ -4739,6 +4763,162 @@ Not run.`,
),
},
)
const ledgerIssue = (line) => ({
...VALID_ISSUE,
description: `${VALID_TICKET_BODY}\n\n${line}`,
parent: { identifier: "ORB-140", title: "Harness defect ledger from the recorded run" },
})
const checkIssue = (name, issue, expect, relations = [], env = {}) =>
check(
"check-ticket.mjs",
name,
["--issue", issue.identifier],
expect,
{
env: {
...orcaEnv([{ match: `linear issue ${issue.identifier}`, stdout: JSON.stringify({ ok: true, result: { issue, relations } }) }]),
...env,
},
},
)

checkIssue("a ledger child with 7 occurrences passes", ledgerIssue("Ledger occurrence: 7; blocked: no"), { status: 0, stdout: /ticket ok/ })
checkIssue("a ledger child at the threshold of 3 occurrences passes", ledgerIssue("Ledger occurrence: 3; blocked: no"), { status: 0, stdout: /ticket ok/ })
checkIssue(
"a non-blocking ledger child below the threshold fails with the count and threshold",
ledgerIssue("Ledger occurrence: 2; blocked: no"),
{ status: 1, stderr: /2[\s\S]*threshold of 3/i },
)
for (const alias of [
"false",
"none",
"n/a",
"no.",
"nothing",
"did not block the run",
"the defect could not block the run",
"the defect could not have blocked the run",
]) {
checkIssue(
`a non-blocking ${alias} alias cannot bypass the threshold`,
ledgerIssue(`Ledger occurrence: 2; blocked: ${alias}`),
{ status: 1, stderr: /literal no or an affirmative claim naming what it blocked/i },
)
}
checkIssue(
"a below-threshold ledger child passes when it names what blocked the run",
ledgerIssue("Ledger occurrence: 2; blocked: the merge sweep was blocked from merging qualifying PRs"),
{ status: 0, stdout: /ticket ok/ },
)
checkIssue(
"a bare blocking claim does not bypass the threshold",
ledgerIssue("Ledger occurrence: 2; blocked: yes"),
{ status: 1, stderr: /literal no or an affirmative claim naming what it blocked/i },
)
checkIssue(
"a bare blocking claim is rejected above the occurrence threshold",
ledgerIssue("Ledger occurrence: 5; blocked: true"),
{ status: 1, stderr: /literal no or an affirmative claim naming what it blocked/i },
)
for (const claim of [
"blocked the merge sweep",
"the merge sweep was blocked by unavailable authentication",
]) {
checkIssue(
`an affirmative ${claim} claim passes below the threshold`,
ledgerIssue(`Ledger occurrence: 2; blocked: ${claim}`),
{ status: 0, stdout: /ticket ok/ },
)
}
checkIssue(
"a ledger child with no occurrence line fails",
{ ...VALID_ISSUE, parent: { identifier: "ORB-140", title: "Harness defect ledger from the recorded run" } },
{ status: 1, stderr: /missing[\s\S]*Ledger occurrence/i },
)
checkIssue(
"a recorded non-ledger child ticket is unaffected",
{ ...VALID_ISSUE, parent: { identifier: "ORB-88", title: "Ordinary project parent" } },
{ status: 0, stdout: /ticket ok/ },
)
checkIssue(
"an unparseable ledger occurrence line fails",
ledgerIssue("Ledger occurrence: several; blocked: no"),
{ status: 1, stderr: /ledger occurrence line is unparseable/i },
)
const noLinearKeyHome = join(root, "check-ticket-no-linear-key")
mkdirSync(noLinearKeyHome, { recursive: true })
const ledgerParentRelation = [{
relationship: "parent",
relatedIssue: { identifier: "ORB-140", title: "Harness defect ledger from the recorded run" },
}]
checkIssue(
"an Orca parent relation validates without a separate Linear key",
{
...VALID_ISSUE,
id: "linear-child-id",
description: `${VALID_TICKET_BODY}\n\nLedger occurrence: 3; blocked: no`,
},
{ status: 0, stdout: /ticket ok/ },
ledgerParentRelation,
{ USERPROFILE: noLinearKeyHome },
)
checkIssue(
"an Orca ledger parent relation still requires the occurrence line",
{ ...VALID_ISSUE, id: "linear-child-without-line" },
{ status: 1, stderr: /missing[\s\S]*Ledger occurrence/i },
ledgerParentRelation,
{ USERPROFILE: noLinearKeyHome },
)
checkIssue(
"a standalone Orca issue validates without a separate Linear key",
{ ...VALID_ISSUE, id: "linear-standalone-id" },
{ status: 0, stdout: /ticket ok/ },
[],
{ USERPROFILE: noLinearKeyHome },
)
const linearKeyHome = join(root, "check-ticket-linear-key")
mkdirSync(linearKeyHome, { recursive: true })
writeFileSync(join(linearKeyHome, ".linear-api-key"), "fixture-key")
const partialParentRelation = [{
relationship: "parent",
relatedIssue: { identifier: "ORB-140" },
}]
checkIssue(
"a partial Orca parent relation uses the bounded Linear fallback",
{
...VALID_ISSUE,
id: "linear-partial-parent",
description: `${VALID_TICKET_BODY}\n\nLedger occurrence: 3; blocked: no`,
},
{ status: 0, stdout: /ticket ok/ },
partialParentRelation,
{
USERPROFILE: linearKeyHome,
ORBIT_LINEAR_PARENT_STUB: JSON.stringify({
requireTimeout: true,
body: {
data: {
issue: {
parent: { identifier: "ORB-140", title: "Harness defect ledger from the recorded run" },
},
},
},
}),
},
)
checkIssue(
"a Linear parent GraphQL error exits with a tool error",
{ ...VALID_ISSUE, id: "linear-parent-error" },
{ status: 2, stderr: /could not read the Linear parent relation[\s\S]*fixture GraphQL failure/i },
partialParentRelation,
{
USERPROFILE: linearKeyHome,
ORBIT_LINEAR_PARENT_STUB: JSON.stringify({
status: 200,
body: { errors: [{ message: "fixture GraphQL failure" }] },
}),
},
)
},
"check-push-target.mjs": () => {
check("check-push-target.mjs", "a push to main is blocked", [], { status: 1, stderr: /BLOCKED/ }, { input: "refs/heads/main abc refs/heads/main def\n" })
Expand Down
Loading