Skip to content

docs(#5426): add CEL trigger and dispatch guidance to BYOA guide - #5532

Merged
ifireball merged 5 commits into
mainfrom
agent/5426-byoa-cel-dispatch-docs
Jul 27, 2026
Merged

docs(#5426): add CEL trigger and dispatch guidance to BYOA guide#5532
ifireball merged 5 commits into
mainfrom
agent/5426-byoa-cel-dispatch-docs

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

Restructures the Bring Your Own Agent guide so the default mental model is a custom agent that runs automatically when matching GitHub events arrive, with fullsend run repositioned as the local development/debug path.

Related Issue

Closes #5426

Changes

  • Restructured intro and diagram — replaced fullsend run-centric flow with dispatch-first flow showing how events reach agents through fullsend dispatch
  • New "How custom agents are dispatched" section — explains the end-to-end dispatch flow (normalize → authorize → enumerate → evaluate → launch) so users understand how registered agents are discovered without per-agent workflow files
  • New "Writing CEL triggers" section — documents NormalizedEvent fields, transition kinds, common trigger patterns (label-based, slash-command, PR events, review events, actor permissions), fork safety, label checking, and local verification with fullsend trigger validate/eval
  • Updated minimum viable agent example — added a trigger field to the harness example so users see triggers from the start
  • New troubleshooting entry — added guidance for debugging trigger expressions that don't match events
  • Updated "See also" — added link to NormalizedEvent v1 spec

Testing

  • lychee markdown link check passes (44 total links, 0 errors)
  • Secret scan passes
  • gitlint passes

Checklist

  • PR title follows Conventional Commits (correct type, ! for breaking changes)
  • I wrote this contribution myself and can explain all changes in it

Closes #5426

Post-script verification

  • Branch is not main/master (agent/5426-byoa-cel-dispatch-docs)
  • Secret scan passed (gitleaks — e45db7d74f778894558047adfaef5c2d52adcbb9..HEAD)
  • PR body secret scan passed (gitleaks — no-git)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

Restructure the Bring Your Own Agent guide so the default mental
model is a custom agent triggered by GitHub events, not one invoked
locally with `fullsend run`. Add three new sections:

- "How custom agents are dispatched" — explains the end-to-end
  dispatch flow (normalize → authorize → enumerate → evaluate →
  launch) so users understand how registered agents are discovered
  and selected without per-agent workflow files.

- "Writing CEL triggers" — covers NormalizedEvent fields, transition
  kinds, common trigger patterns (label-based, slash-command,
  PR events, review events), fork safety, and local verification
  with `fullsend trigger validate/eval`.

- Updated intro and diagram — leads with the dispatch-first model
  and repositions `fullsend run` as the local development/debug
  path.

Also adds a trigger to the minimum viable agent example, a new
troubleshooting entry for trigger debugging, and a link to the
NormalizedEvent v1 spec in "See also".

Note: pre-commit could not run (sandbox network restriction).
The post-script runs an authoritative check on the runner.

Closes #5426
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner July 23, 2026 14:15
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Jul 23, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 23, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:20 PM UTC · Completed 2:35 PM UTC
Commit: 5ecb4e8 · View workflow run →

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown

Site preview

Preview: https://864ea953-site.fullsend-ai.workers.dev

Commit: 6e105dcd5964710113fcb5f3a775c97f956cb98b

@rh-hemartin rh-hemartin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have this division consolidated anywhere as far as I know, but I think you need to split the changes in two files: the current file and a reference one. The current file would be lightweight and have two examples not explaining much. Then you point to the reference document and have all the details on there.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review

Findings

Low

  • [edge-case] docs/guides/user/cel-triggers-reference.md:104 — The slash-command trigger pattern accesses event.transition.comment.command without guarding for its absence with has(). The command field is optional in the NormalizedEvent schema (tagged omitempty in Go), so it is absent from the CEL map for non-command comments. When a comment_added event without a command reaches this trigger, CEL produces a missing-key error. Dispatch handles this gracefully (logs and skips), so the functional outcome is correct — the only impact is spurious error log entries. The same unguarded pattern also appears at line 143 in the label-checking example. Note: this pattern is consistent with existing normative docs elsewhere in the repo.
    Remediation: Guard the command access with has(): has(event.transition.comment.command) && event.transition.comment.command == "/my-command". Consider adding a brief note about has() for optional fields.
Previous run

Review

Findings

Low

  • [Incomplete documentation] docs/guides/user/cel-triggers-reference.md:58 — The NormalizedEvent fields table omits event.actor.is_entity_author (boolean, required per schema). The table documents the other three actor fields (id, kind, role) but omits this one, which is useful for CEL triggers (e.g., skipping events where the actor is the entity author).
    Remediation: Add a row: event.actor.is_entity_author | boolean | True when the actor is the author of the work item or change proposal.

  • [Incomplete API surface documentation] docs/guides/user/cel-triggers-reference.md:56 — Multiple schema fields usable in CEL triggers are omitted from the reference table: event.entity.url (required), event.entity.key (for Jira), event.source.raw_type (required), event.source.raw_action (optional). The event.state.change_proposal row lists only 3 of 7 required sub-fields. The doc links to the full schema but a reference table that omits accessible fields may cause users to believe those fields are unavailable.
    Remediation: Consider adding a note directly below the table like "This table covers the most common trigger fields. For the complete field list, see the NormalizedEvent v1 schema."

Previous run (2)

Review

Findings

Medium

  • [API contract violation] docs/guides/user/cel-triggers-reference.md:154 — The "Verifying your trigger" section documents two CLI commands — fullsend trigger validate and fullsend trigger eval — that do not exist in the codebase. The CLI has no trigger subcommand; users following these instructions will get an "unknown command" error.
    Remediation: Either remove the commands and replace with an accurate verification workflow (e.g., using fullsend dispatch with a test fixture), or implement the trigger validate and trigger eval subcommands before merging this doc.
Previous run (3)

Review

Findings

Medium

  • [internal-consistency] docs/guides/user/bring-your-own-agent.md:363 — The harness field reference example uses event.type == 'issue' in the trigger expression, but event.type is not a valid NormalizedEvent field. The NormalizedEvent schema defines root properties as repo, entity, transition, actor, state, and source — no type field exists. This PR updates the comment on this line to reference the new "Writing CEL triggers" section and NormalizedEvent, making the invalid expression more conspicuous to readers who will see valid field paths like event.entity.kind throughout the new trigger documentation.
    Remediation: Change the trigger example to use a valid NormalizedEvent field path, e.g. trigger: "event.entity.kind == 'work_item'".

Low

  • [edge-case-completeness] docs/guides/user/bring-your-own-agent.md — The "Transition kinds" table omits the updated kind, which is a valid enum value in the NormalizedEvent JSON schema. The normative README documents it as a legacy umbrella that adapters may still emit. Users whose adapters emit updated transitions would not find it in this guide.
    Remediation: Add updated to the transition kinds table with a note: "Legacy umbrella — prefer edited or synchronized for new triggers."

Labels: PR modifies documentation under docs/guides/ covering dispatch and CEL triggers

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment component/docs User-facing documentation component/dispatch Workflow dispatch and triggers labels Jul 23, 2026
@ifireball

Copy link
Copy Markdown
Member

/fs-fix

Implement the review feedback on this PR as follows.

Primary: rh-hemartin's split (guide vs reference)

Split the new dispatch/CEL material so BYOA stays a lightweight how-to and the detail lives in a reference doc.

Keep in docs/guides/user/bring-your-own-agent.md

  • Dispatch-first intro + short diagram (already present)
  • Minimum viable agent with one concrete trigger example
  • Short pointer: register → write trigger → link to the new reference
  • Real-world triage example (or a second minimal example) without field tables / pattern catalogs
  • Existing harness / register / troubleshooting flow

Move to a new reference doc

Create something like docs/guides/user/cel-triggers.md (or agent-dispatch-triggers.md) and move into it almost as-is:

  • "How custom agents are dispatched" (normalize → authorize → enumerate → evaluate → launch)
  • You vs dispatch table + coexistence note
  • "Writing CEL triggers": NormalizedEvent fields, transition kinds, common patterns, label checks, fork safety, fullsend trigger validate/eval
  • Link to normative NormalizedEvent + examples for deeper detail

Wiring

  • Link from BYOA intro / MVP / troubleshooting → the new reference
  • Add the new page to docs/guides/README.md and the VitePress User Guides sidebar in website/.vitepress/config.ts
  • Replace deep #writing-cel-triggers (and similar) anchors in BYOA with links to the new page
  • Do not send users to ADRs for this; the user-facing reference is the home for detail (per docs: cover CEL triggers and custom agent dispatch in Bring Your Own Agent guide #5426)

After the move, BYOA should read as a how-to with 1–2 triggers and almost no CEL explanation — not a CEL handbook. Do not split the pre-existing harness field reference unless needed for the above.

Also fix fullsend-ai-review findings

  1. Medium: In the harness field reference example, replace invalid trigger: "event.type == 'issue'" with a valid NormalizedEvent path, e.g. trigger: "event.entity.kind == 'work_item'" (aligned with the MVP example).
  2. Low: In the transition kinds table (in the new reference), add updated with a note that it is a legacy umbrella — prefer edited or synchronized for new triggers.

Verify

  • Stage changes and run lint / markdown link checks as usual for this repo
  • Reply on rh-hemartin's review thread noting the guide/reference split
  • Resolve the bot inline thread after the event.type fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 11:01 AM UTC · Completed 11:09 AM UTC
Commit: 5ecb4e8 · View workflow run →

Move dispatch flow and CEL trigger documentation (NormalizedEvent fields,
transition kinds, common patterns, fork safety, verification) from the
Bring Your Own Agent guide into a dedicated reference page at
docs/guides/user/cel-triggers-reference.md.

BYOA now reads as a lightweight how-to (intro, MVP, triage example,
register, troubleshoot) with links to the reference for detail.

Also fixes:
- Harness field reference: invalid event.type → event.entity.kind
- Transition kinds table: adds 'updated' legacy umbrella kind

Addresses review feedback on #5532
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (human-triggered)

Split BYOA guide into a lightweight how-to and a dedicated CEL triggers reference page. BYOA keeps dispatch intro, diagram, MVP trigger, triage example, and register/troubleshoot flow. New reference page at cel-triggers-reference.md has dispatch flow, NormalizedEvent fields, transition kinds (including 'updated' legacy kind), common patterns, fork safety, and verification. Fixed invalid event.type trigger in harness field reference. Added sidebar and README entries. All internal links verified with lychee (71 OK, 6 external URL failures due to sandbox network).

Fixed (6):

  1. Split BYOA guide into how-to + reference (rh-hemartin feedback) (docs/guides/user/bring-your-own-agent.md): Removed 'How custom agents are dispatched' and 'Writing CEL triggers' sections from BYOA guide, keeping it as a lightweight how-to with dispatch intro, diagram, MVP trigger example, and pointers to the new reference
  2. Create CEL triggers reference doc (docs/guides/user/cel-triggers-reference.md): Created new reference page with dispatch flow (normalize, authorize, enumerate, evaluate, launch), you-vs-dispatch table, coexistence note, NormalizedEvent fields, transition kinds, common trigger patterns, label checks, fork safety, and trigger verification
  3. Invalid trigger expression in harness field reference (docs/guides/user/bring-your-own-agent.md): Replaced invalid event.type == 'issue' with valid event.entity.kind == 'work_item' aligned with the MVP example
  4. Add 'updated' transition kind (docs/guides/user/cel-triggers-reference.md): Added 'updated' to transition kinds table with note that it is a legacy umbrella — prefer 'edited' or 'synchronized' for new triggers
  5. Wire new reference into sidebar and README (website/.vitepress/config.ts): Added CEL Triggers Reference entry to VitePress User Guides sidebar and docs/guides/README.md
  6. Update links from BYOA to new reference (docs/guides/user/bring-your-own-agent.md): Replaced #writing-cel-triggers anchors with links to cel-triggers-reference.md, updated troubleshooting and See also sections to point to the new reference page instead of ADRs

Tests: passed

Updated by fullsend fix agent

@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ifireball

Copy link
Copy Markdown
Member

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 26, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 11:31 AM UTC · Completed 11:48 AM UTC
Commit: 67d4355 · View workflow run →

@ifireball

Copy link
Copy Markdown
Member

/fs-fix fix per-fullsend-ai-review comments, especially about non-existent CLI commands

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 12:39 PM UTC · Completed 12:43 PM UTC
Commit: 67d4355 · View workflow run →

Replace `fullsend trigger validate` and `fullsend trigger eval` commands
(which do not exist in the CLI) with actionable verification guidance:
check field paths against the schema, walk through example fixtures, and
test end-to-end in a test repository.

Addresses review feedback on #5532
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 2 (human-triggered)

Addressed 1 review finding: removed non-existent CLI commands (fullsend trigger validate/eval) from the CEL triggers reference and replaced with practical verification steps.

Fixed (1):

  1. Non-existent fullsend trigger validate/eval CLI commands (docs/guides/user/cel-triggers-reference.md): Removed non-existent fullsend trigger validate and fullsend trigger eval commands from the 'Verifying your trigger' section. Replaced with actionable verification guidance: (1) check field paths against the NormalizedEvent schema, (2) walk through example event fixtures manually, (3) test end-to-end in a test repository.

Tests: passed

Updated by fullsend fix agent

@ifireball

Copy link
Copy Markdown
Member

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 26, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:07 PM UTC · Completed 1:23 PM UTC
Commit: a28d716 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot removed the requires-manual-review Review requires human judgment label Jul 26, 2026
@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Jul 26, 2026
@ifireball

Copy link
Copy Markdown
Member

/fs-fix remaining review agent comments

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 2:02 PM UTC · Completed 2:11 PM UTC
Commit: a28d716 · View workflow run →

Add event.actor.is_entity_author (required boolean) to the NormalizedEvent
fields table in the CEL triggers reference, and add a note below the table
directing readers to the full schema for fields not covered by the table.

Addresses review feedback on #5532
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 3 (human-triggered)

Addressed both remaining review findings: added the missing event.actor.is_entity_author field to the NormalizedEvent fields table, and added a completeness note below the table pointing to the full schema for fields not individually listed.

Fixed (2):

  1. Missing event.actor.is_entity_author in NormalizedEvent fields table (docs/guides/user/cel-triggers-reference.md): Added event.actor.is_entity_author (boolean, required per schema) to the NormalizedEvent fields table between event.actor.role and event.state.labels
  2. Multiple schema fields omitted from reference table without completeness note (docs/guides/user/cel-triggers-reference.md): Added a note below the NormalizedEvent fields table directing readers to the full schema for fields not covered by the table (event.entity.url, event.entity.key, event.source.raw_type, event.state.change_proposal sub-fields)

Tests: passed

Updated by fullsend fix agent

@ifireball

Copy link
Copy Markdown
Member

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 26, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:12 PM UTC · Completed 3:32 PM UTC
Commit: 7a773e6 · View workflow run →

Comment thread docs/guides/user/cel-triggers-reference.md
@ifireball

Copy link
Copy Markdown
Member

/fs-fix the review agent comment about has().

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 5:36 AM UTC · Completed 5:41 AM UTC
Commit: 7a773e6 · View workflow run →

Add has() guards before accessing event.transition.comment.command in
the slash-command and label-checking trigger examples. The command field
is optional (omitempty in Go) and absent from the CEL map for
non-command comments, causing missing-key errors without the guard.

Add a new "Guarding optional fields with has()" section explaining when
and why has() is needed.

Addresses review feedback on #5532
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 4 (human-triggered)

Addressed all 3 aspects of the has() review finding: guarded both trigger examples that access event.transition.comment.command with has(), and added a new documentation section explaining when and why has() guards are needed for optional CEL fields.

Fixed (3):

  1. Unguarded access to optional event.transition.comment.command field in slash-command trigger pattern (line 104) (docs/guides/user/cel-triggers-reference.md): Added has(event.transition.comment.command) guard before the command comparison in the slash-command trigger example
  2. Unguarded access to optional event.transition.comment.command field in label-checking trigger example (line 143) (docs/guides/user/cel-triggers-reference.md): Added has(event.transition.comment.command) guard before the command comparison in the label-checking trigger example
  3. Missing documentation about has() for optional fields (docs/guides/user/cel-triggers-reference.md): Added new 'Guarding optional fields with has()' section explaining when has() is needed, with a standalone code example and guidance on which fields require guards

Tests: passed

Updated by fullsend fix agent

@ifireball
ifireball added this pull request to the merge queue Jul 27, 2026
Merged via the queue into main with commit 325c679 Jul 27, 2026
16 checks passed
@ifireball
ifireball deleted the agent/5426-byoa-cel-dispatch-docs branch July 27, 2026 07:04
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 27, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 7:07 AM UTC · Completed 7:27 AM UTC
Commit: 6e105dc · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #5532 — CEL trigger and dispatch documentation

Workflow: Issue #5426 → triage → code agent → PR #5532 → 4 review rounds → 4 fix iterations → human approvals → merged.

Timeline: Issue created Jul 21. Code agent opened PR Jul 23. First review same day. Three days of fix/review cycles (Jul 26–27). Two human approvals Jul 27. Merged Jul 27. Total: ~6 days open, ~3.5 days of active iteration.

What went well

  • Review agent precision was 100%. All 6 findings across 4 rounds were true positives — every one was acted on. Zero false positives. The agent caught an invalid NormalizedEvent field path (event.typeevent.entity.kind), fabricated CLI commands, a missing schema field, and missing has() guards.
  • The review agent caught a fix-agent hallucination. Fix iteration 1 fabricated two CLI commands (fullsend trigger validate and fullsend trigger eval) that don't exist in the codebase. The review agent flagged this as an API contract violation in round 2.
  • Human review provided high-value structural feedback. rh-hemartin identified that the BYOA guide was being overloaded with reference material and should be split into a lightweight how-to guide and a separate CEL triggers reference doc — the single most impactful piece of feedback on the PR.

Improvement opportunities (all covered by existing issues)

1. Review agent approved despite medium-severity finding (Round 2). The review agent found fabricated CLI commands — a medium-severity "API contract violation" — yet submitted an APPROVED review. The schema allows this because it only enforces approval constraints for protected-path findings, not severity-based constraints. The LLM instructions say a single medium finding should produce comment-only, but the agent deviated. This provides fresh evidence for fullsend#1453 (escalate medium-severity correctness findings) and fullsend#2940 (use CHANGES_REQUESTED for medium-severity findings). The review result schema (schemas/review-result.schema.json in fullsend-ai/agents) has no allOf rule preventing approve with medium+ severity findings — adding one (analogous to the existing protected-path rule) would be a concrete implementation path.

2. Review agent missed document architecture feedback. The most impactful review comment came from a human (rh-hemartin), not the review agent. The agent evaluated field-level correctness but did not assess whether cramming ~160 lines of reference material into a how-to guide was appropriate for the document's audience and purpose. This is a direct instance of agents#261 (review agent should evaluate information architecture of new user-facing documentation) and fullsend#4838 (assess documentation placement semantically).

3. Fix agent introduced a hallucination not present in the original. The initial code agent output did not contain fabricated CLI commands. Fix iteration 1 introduced them when restructuring the document — adding a "Verifying your trigger" section with invented subcommands. This is evidence for fullsend#2059 (code agent should smoke-test CLI invocations) and fullsend#1322 (validate CLI flag semantics in documentation against source code). Both apply to the fix agent since it shares the same code agent infrastructure.

4. Piecemeal findings drove 4 fix iterations. The review agent discovered findings incrementally: 2 in round 1, 1 in round 2, 2 in round 3, 1 in round 4. Had all 6 findings been surfaced in round 1, the PR could have converged in 1–2 fix iterations instead of 4. This supports fullsend#4970 (detect all valid findings in a single pass) and fullsend#1582 (catch all findings in the first pass to reduce rework cycles). Note: 3 of the 6 findings were against code introduced by fix iterations (not present in the original commit), so complete first-pass coverage would not have been possible — but the 2 findings from round 1 plus the has() guard finding from round 4 were all present in the initial commit.

Autonomy assessment

The review agent demonstrated strong competence at schema-level validation (field paths, completeness, API contract verification) but missed the highest-value feedback category for docs PRs: information architecture. Until agents#261 is addressed, docs PRs that add substantial new content should continue to require human review — the review agent cannot yet substitute for editorial judgment about document structure and audience.

No new proposals

All identified improvements are covered by existing open issues. No new proposals are warranted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/dispatch Workflow dispatch and triggers component/docs User-facing documentation ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: cover CEL triggers and custom agent dispatch in Bring Your Own Agent guide

2 participants