Conversation
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can hide the parts of a finding you never read, like the evidence or the agent prompt |
PR Summary by QodoAdopt gh-aw safe outputs as the GitHub side-effect model
AI Description
Diagram
High-Level Assessment
Files changed (4)
|
Site previewPreview: https://9117c072-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
In general sounds good to me, but a few points:
Could you bring this to the contributor's meeting for further discussion? |
See https://blog.verbum.org/2026/08/21/agentic-ai-and-software-forges/ I believe GH-AW is a reference baseline, and some aspects of how parts of agents do writes is less secure than what the GH-AW safe outputs functionality provides. We should just reuse it instead of reinventing it. Declare too that we also aim to generalize a safe-outputs style system beyond GitHub. Assisted-by: https://github.com/cgwalters/cgwalters#llms Signed-off-by: Colin Walters <walters@verbum.org>
7afad93 to
572258c
Compare
|
Thanks for the rewrite. Is |
Yes, this is an important decision as it's influenced by the whole system architecture. A thing I find extremely compelling about GH-AW is one can describe it in a single sentence: "A compiler that takes markdown agent steps with YAML frontmatter defining its event triggers and output restrictions, and generates Github Actions." (I think it'd be quite challenging to describe fullsend or various other projects in this space as comprehensively in a single sentence description) The reason the architecture matters a lot here because the safe outputs implementation is heavily influenced by that compilation phase, see e.g. But in my investigations, it should be feasible to actually reuse the implementation code (mostly in Go) and this also means we reuse the exact same schema. A huge advantage of this approach is that if e.g. my employer wanted to productize an upstream project using GH-AW today, and port them over to Fullsend, it'd be a lot more seamless if the exact same safe-outputs schema worked. |
|
Could you bring the topic to the contributors meeting? https://docs.google.com/document/d/106cDm4CXBg4wd2j9VpVhexHHylOLThdcpqe4p6gXIXQ |
While I can see that document from my work machine, I often do development from a different machine that only does personal and upstream development (as is the case currently) and is not signed into work accounts. So for reference this is now a transition point where someone outside our employer but who is interested in contributing to fullsend would get stuck. |
|
#6878 is another example of something that GH-AW is also fixing today. |
|
|
||
| For GitHub-specific agent side effects, Fullsend will migrate compatible writes | ||
| from privileged post-scripts to gh-aw safe outputs, operation by operation. | ||
| Note that [gh-aw custom safe outputs exists](https://github.github.com/gh-aw/reference/custom-safe-outputs/) |
There was a problem hiding this comment.
Hm, so - custom safe-outputs is kind of equivalent to what we have with post scripts today. The user can provide an arbitrary shell script to execute at the end of the process.
Nothing about the platform or framework ensures that the output of that is really "safe" (in both fullsend and gh-aw). It's just the flexible "escape hatch".
|
A thing I'm wrestling with right now, is over this:
This is both a blessing and a curse. It's a blessing insofar as github has thought this through pretty well. People will be familiar with it at some point and seeing it in fullsend harness config files will feel obvious to them - if we were to share the schema with gh-aw. The curse might be that the schema is github-flavoured. If we want to use the safe-outputs schema as the general schema for expressing constraints on issue-creation for generic backends, could we have an agent use this to create issues in JIRA, or in service-now? "allowed-repos" doesn't make sense in those contexts. -- It's just to say that I'm torn. If we expose safe-outputs schema to the user, we benefit from the familiarity, but we lose the control we likely need to succeed at being agnostic to the external system (be it a forge or another tracker system). |
|
That said, we could still consider using safe-outputs under the hood - by translating a fullsend-native config that looks similar to the gh-aw safe-outputs config schema down to the literal gh-aw safe-outputs config schema at runtime. |
| deduplication, and policy checks. However, we will also eventually invest | ||
| in a "generic-safe-outputs" style tool that has similar opinionated verbs | ||
| for other sources such as GitLab, Forgejo and Jira. |
There was a problem hiding this comment.
This generic-safe-outputs may be the main thing to focus on here. i.e., to wrap gh-aw's safe-outputs in a fullsend-native version that is meant to look identical but which is forge-agnostic.
I think we should support explicitly prefix the entire thing with Forge agnostic I think does really want a A problem with any abstraction layer like that is it drives to the lowest common denominator, and I don't think we want to force that. People who know they are using a specific forge should be able to do forge-native things.
I'm arguing for supporting both. BTW a previous project I generated in this space was https://github.com/LobsterTrap/service-gator and I eventually concluded it could make sense for it to grow into a generic-safe-outputs tool instead of being a MCP. At least it's not shell script... |
So I was searching the GH-AW docs for the keyword "jira" and came across this: It looks like they've been building out other direct integrations. Would they accept them for GitLab or Forgejo? Would be interesting to try 😄 |
| For GitHub-specific agent side effects, Fullsend will migrate compatible writes | ||
| from privileged post-scripts to gh-aw safe outputs, operation by operation. | ||
| Note that [gh-aw custom safe outputs exists](https://github.github.com/gh-aw/reference/custom-safe-outputs/) | ||
| as a generic escape hatch. |
There was a problem hiding this comment.
I'm on the fence about this ADR - I think its trying to go in a positive direction, but putting everything in terms of gh-aw and otherwise vague language makes it hard for me to understand what the result would look like from a Fullsend user POV.
The existence of conditional overlays in harnesses means that we can use different approaches for post run processing in different environments, so I can see us adopting a better mechanism for GH while retaining the scripts for other platforms for a while.
There was a problem hiding this comment.
The existence of conditional overlays in harnesses means that we can use different approaches for post run processing in different environments
I think this gets entirely replaced with generic-safe-outputs.
|
@cgwalters my last comment is rather vague - here is something more workable - can you consider adding some overview of gh-aw safe outputs in the ADR context, enough so the reader would be agle to understand what using it in Fullsend would look and feel like? |
I'm uncertain how I could explain it better than the docs do; and definitely not better than someone asking their own agents questions pointing them at the docs.
I think there's two parts to this:
(BTW, GH-AW has no "builtin agents" - really important to internalize that, and I think it's the right call. Fullsend is at least going that direction, GH-AW is already there) Anyways most interesting is for https://fullsend.sh/docs/guides/user/bring-your-own-agent - it would be a breaking change for people doing that today. Basically we're replacing the pre and post scripts. Or at least IMO in most cases we should replace the pre scripts with e.g. a skill the agent should execute, or a MCP tool. I mean for the triage agent is it really better to fetch all the issue content before starting the agent vs just having the agent fetch it (and btw, be able to fetch any updated content after?) Debatable. But we're definitely replacing the post scripts with GH-AW - or perhaps per above a new "generic-safe-outputs" project that has Does that help? |
|
There's some interesting subtleties we could call out; for example GH-AW is in preview, and retains the capability to make breaking changes across versions - but the intended semantic is that one pins to a version. (Heading of the "we shouldn't depend on something that can change" objection: the same is true in practice of Fullsend itself as an overall system; we're IMO pretty far from some kind of 1.0 LTS) In practice, what would probably work is for us to pin to a version of GH-AW sources in fullsend, and then bump it whenever we're making semver bumps to other parts of fullsend. |
4 participants
Fullsend needs a safer boundary than broadly privileged post-scripts without making its agent model dependent on one forge. Record gh-aw as the migration target for compatible GitHub writes and as guidance for a separate provider-neutral design covering GitLab, Jira, and Forgejo.
Assisted-by: https://github.com/cgwalters/cgwalters#llms