Skip to content

fix(#1230): run OutputPipeline on post-review before posting to forge - #2444

Merged
ben-alkov merged 3 commits into
mainfrom
agent/1230-sanitize-post-review
Jun 23, 2026
Merged

fix(#1230): run OutputPipeline on post-review before posting to forge#2444
ben-alkov merged 3 commits into
mainfrom
agent/1230-sanitize-post-review

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

The post-review command posted review content directly to the GitHub API without running it through the security output pipeline. When invoked standalone (outside fullsend run), secrets and zero-width- obfuscated tokens in agent output could reach the forge unredacted.

Call security.OutputPipeline().Scan() on the review body and finding text fields (description, remediation) before any forge API call. This matches the pattern used by fullsend scan output and the sandbox post-tool hooks.

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com


Closes #1230

Post-script verification

  • Branch is not main/master (agent/1230-sanitize-post-review)
  • Secret scan passed (gitleaks — 59159d05e2cb01f868054a4c7143303d6049214b..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown

Site preview

Preview: https://88f76682-site.fullsend-ai.workers.dev

Commit: 4cb8250e358b3b0f09cc66374a8e0d3ceaecd710

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 18, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 8:01 PM UTC · Completed 8:12 PM UTC
Commit: 6deedbe · View workflow run →

@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.42857% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/cli/postreview.go 96.42% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 18, 2026

Copy link
Copy Markdown

Looks good to me

Previous run

Review

Findings

Low

  • [logging-gap] internal/cli/postreview.go:554 — When a secret is redacted from a finding's Severity, Category, Description, or Remediation field, the sanitization is applied silently with no log output. The body sanitization path logs both a summary count and per-finding details via printer.StepWarn, but the per-finding-field loop has no logging at all. Note: Pipeline.Scan always returns a non-empty Sanitized field for non-empty input, so the logging condition should check len(result.Findings) > 0 rather than result.Sanitized != "".

  • [repetitive-pattern] internal/cli/postreview.go:554 — The scan-check-assign pattern is repeated identically for four finding fields (Severity, Category, Description, Remediation). Extracting a small helper would reduce duplication and make it harder to forget logging when a new field is added to ReviewFinding.

  • [data-exposure] internal/cli/postreview.go:540 — The sanitizeReviewResult function sanitizes Body, Severity, Category, Description, and Remediation fields but does not sanitize the File field. While File is used as a structured GitHub API path parameter (forge.ReviewComment.Path) and the API would reject invalid paths, sanitizing it would complete the defense-in-depth boundary. The remaining unsanitized fields are protected by other mechanisms: Action (switch/case allowlist), HeadSHA (hex regex), Reason (alphanumeric regex).


Security coverage verification: All five free-text ReviewFinding fields that reach the forge via formatFindingComment (Severity, Category, Description, Remediation) and sticky.Post (Body) are now sanitized through security.OutputPipeline(). The prior review's medium finding ([incomplete-sanitization]: Severity and Category fields not sanitized) has been fully resolved with a dedicated test (TestSanitizeReviewResult_RedactsSecretsInSeverityAndCategory).

Previous run (2)

Review

Findings

Low

  • [logging-pattern] internal/cli/postreview.go:546 — The warning log for body sanitization emits a summary count but does not log per-finding details (scanner name and detail). The established pattern in scan.go:194-196 and run.go:1831 logs each finding individually after the summary line, which helps operators diagnose what was sanitized.

  • [repetitive-pattern] internal/cli/postreview.go:549 — The scan-check-assign pattern is repeated identically for each of the five finding fields (Severity, Category, Description, Remediation, and Body). Extracting a sanitizeField(pipeline, field) string helper would reduce repetition and make it easier to add fields in the future.

Info

  • [prior-finding-resolved] internal/cli/postreview.go — The prior review's medium finding ([incomplete-sanitization]: Severity and Category fields not sanitized) has been fully resolved. Both fields now have pipeline.Scan() applied, with a dedicated test (TestSanitizeReviewResult_RedactsSecretsInSeverityAndCategory).

Security coverage verification: All five free-text ReviewFinding fields that reach the forge via formatFindingComment (Severity, Category, Description, Remediation) and sticky.Post (Body) are now sanitized through security.OutputPipeline(). The remaining unsanitized fields are protected by other mechanisms: File is used only as a GitHub API Path parameter (validated by GitHub against the diff), Action passes through a switch/case allowlist, HeadSHA is validated by hex regex, and Reason is validated by an alphanumeric regex.

Previous run

Review

Findings

Medium

  • [incomplete-sanitization] internal/cli/postreview.go:549sanitizeReviewResult does not sanitize the Severity and Category fields of ReviewFinding. These fields are interpolated into Markdown posted to the forge by formatFindingComment (line 427: fmt.Fprintf(&b, "**[%s]** %s", f.Severity, f.Category)). While these fields are structurally constrained by the agent prompt and the realistic risk is low, the PR's stated goal is to sanitize all user-visible text fields before posting to the forge, making this an incomplete implementation.
    Remediation: Either apply pipeline.Scan() to r.Findings[i].Severity and r.Findings[i].Category inside the existing loop, or validate them against an allowlist of known values (e.g., critical/high/medium/low/info for severity) which would be stronger given the small valid set.

Low

  • [misleading-log-message] internal/cli/postreview.go:542 — The warning message says Redacted %d secret(s) in review body but len(result.Findings) counts all pipeline findings, including those from the UnicodeNormalizer (e.g., null bytes, zero-width chars). When the body contains only zero-width obfuscation and no actual secrets, the message will still claim secrets were redacted, which is misleading for operators reviewing logs.

Labels: PR fixes a security gap in the post-review CLI command

Previous run (3)

Review

Findings

Low

  • [logging-pattern] internal/cli/postreview.go:546 — The warning log for body sanitization emits a summary count but does not log per-finding details (scanner name and detail). The established pattern in scan.go:194-196 and run.go:1831 logs each finding individually after the summary line, which helps operators diagnose what was sanitized.

  • [repetitive-pattern] internal/cli/postreview.go:549 — The scan-check-assign pattern is repeated identically for each of the five finding fields (Severity, Category, Description, Remediation, and Body). Extracting a sanitizeField(pipeline, field) string helper would reduce repetition and make it easier to add fields in the future.

Info

  • [prior-finding-resolved] internal/cli/postreview.go — The prior review’s medium finding ([incomplete-sanitization]: Severity and Category fields not sanitized) has been fully resolved. Both fields now have pipeline.Scan() applied, with a dedicated test (TestSanitizeReviewResult_RedactsSecretsInSeverityAndCategory).

Security coverage verification: All five free-text ReviewFinding fields that reach the forge via formatFindingComment (Severity, Category, Description, Remediation) and sticky.Post (Body) are now sanitized through security.OutputPipeline(). The remaining unsanitized fields are protected by other mechanisms: File is used only as a GitHub API Path parameter (validated by GitHub against the diff), Action passes through a switch/case allowlist, HeadSHA is validated by hex regex, and Reason is validated by an alphanumeric regex.

Previous run (4)

Review

Findings

Medium

  • [incomplete-sanitization] internal/cli/postreview.go:549sanitizeReviewResult does not sanitize the Severity and Category fields of ReviewFinding. These fields are interpolated into Markdown posted to the forge by formatFindingComment (line 427: fmt.Fprintf(&b, "**[%s]** %s", f.Severity, f.Category)). While these fields are structurally constrained by the agent prompt and the realistic risk is low, the PR's stated goal is to sanitize all user-visible text fields before posting to the forge, making this an incomplete implementation.
    Remediation: Either apply pipeline.Scan() to r.Findings[i].Severity and r.Findings[i].Category inside the existing loop, or validate them against an allowlist of known values (e.g., critical/high/medium/low/info for severity) which would be stronger given the small valid set.

Low

  • [misleading-log-message] internal/cli/postreview.go:542 — The warning message says Redacted %d secret(s) in review body but len(result.Findings) counts all pipeline findings, including those from the UnicodeNormalizer (e.g., null bytes, zero-width chars). When the body contains only zero-width obfuscation and no actual secrets, the message will still claim secrets were redacted, which is misleading for operators reviewing logs.

Labels: PR fixes a security gap in the post-review CLI command

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Comment thread internal/cli/postreview.go Outdated
Comment thread internal/cli/postreview.go
@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment security Security threat model and related concerns type/bug Confirmed defect in existing behavior labels Jun 18, 2026
@ben-alkov ben-alkov self-assigned this Jun 22, 2026
@ben-alkov ben-alkov added the ok-to-test Allow e2e CI to run after maintainer review (must be re-applied after each push) label Jun 22, 2026
@ben-alkov

Copy link
Copy Markdown
Member

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ❌ Failure · Started 7:55 PM UTC · Completed 8:02 PM UTC
Commit: 4e21a60 · View workflow run →

@ben-alkov
ben-alkov force-pushed the agent/1230-sanitize-post-review branch from 6deedbe to fe4e8ea Compare June 22, 2026 20:05
@github-actions github-actions Bot removed the ok-to-test Allow e2e CI to run after maintainer review (must be re-applied after each push) label Jun 22, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jun 22, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 8:09 PM UTC · Ended 8:17 PM UTC
Commit: 4e21a60 · View workflow run →

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 8:10 PM UTC · Completed 8:17 PM UTC
Commit: 4e21a60 · View workflow run →

fullsend-ai-coder Bot added a commit that referenced this pull request Jun 22, 2026
…message

Address two review findings on PR #2444:

1. [incomplete-sanitization] Apply pipeline.Scan() to Severity and Category
   fields in ReviewFinding, which are interpolated into Markdown posted to
   the forge by formatFindingComment.

2. [misleading-log-message] Change warning from "Redacted N secret(s)" to
   "Sanitized review body (N finding(s))" since pipeline findings include
   unicode normalization, not just secrets.

Addresses review feedback on #2444
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (human-triggered)

Addressed both review findings: (1) added Severity and Category field sanitization in sanitizeReviewResult to close the incomplete-sanitization gap, and (2) fixed the misleading log message to not claim secret redaction when findings may be unicode normalization.

Fixed (2):

  1. incomplete-sanitization: Severity and Category fields not sanitized (internal/cli/postreview.go): Added pipeline.Scan() calls for Severity and Category fields in the ReviewFinding loop, matching the existing pattern for Description and Remediation. Added test TestSanitizeReviewResult_RedactsSecretsInSeverityAndCategory and extended TestSanitizeReviewResult_NoSecretsPassesThrough to assert clean severity/category pass through unchanged.
  2. misleading-log-message: warning claims secrets but counts all findings (internal/cli/postreview.go): Changed log message from 'Redacted %d secret(s) in review body' to 'Sanitized review body (%d finding(s))' to accurately reflect that pipeline findings include unicode normalization, not just secret redaction.

Tests: passed

Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 22, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 8:21 PM UTC · Completed 8:32 PM UTC
Commit: 54b77f0 · View workflow run →


// Sanitize the main body.
if r.Body != "" {
result := pipeline.Scan(r.Body)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] logging-pattern

The warning log for body sanitization emits a summary count but does not log per-finding details (scanner name and detail). The established pattern in scan.go:194-196 and run.go:1831 logs each finding individually after the summary line, which helps operators diagnose what was sanitized.

result := pipeline.Scan(r.Body)
if result.Sanitized != "" {
r.Body = result.Sanitized
printer.StepWarn(fmt.Sprintf("Sanitized review body (%d finding(s))", len(result.Findings)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] repetitive-pattern

The scan-check-assign pattern is repeated identically for each of the five finding fields (Severity, Category, Description, Remediation, and Body). Extracting a sanitizeField(pipeline, field) helper would reduce repetition and make it easier to add fields in the future.

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed requires-manual-review Review requires human judgment labels Jun 22, 2026
@ben-alkov

Copy link
Copy Markdown
Member

/fs-fix Fix finding "[logging-pattern] internal/cli/postreview.go:546"

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 8:57 PM UTC · Completed 9:04 PM UTC
Commit: 4e21a60 · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 22, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 9:15 PM UTC · Completed 9:26 PM UTC
Commit: 4cb8250 · View workflow run →

printer.StepWarn(fmt.Sprintf(" %s: %s", f.Name, f.Detail))
}
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] logging-gap

When a secret is redacted from a finding field (Severity, Category, Description, or Remediation), the sanitization is applied silently with no log output. The body sanitization path logs via printer.StepWarn but the per-finding-field loop has no logging. Logging condition should check len(result.Findings) > 0 rather than result.Sanitized != empty.

printer.StepWarn(fmt.Sprintf(" %s: %s", f.Name, f.Detail))
}
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] repetitive-pattern

The scan-check-assign pattern is repeated identically for four finding fields. Extracting a sanitizeField helper would reduce duplication and make it harder to forget logging when new fields are added.

// sanitizeReviewResult runs the security output pipeline over all
// user-visible text fields in a ReviewResult. This catches leaked
// secrets and zero-width–obfuscated tokens before they reach the
// forge API.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] data-exposure

sanitizeReviewResult does not sanitize the File field of ReviewFinding. While File is used as a structured GitHub API path parameter and the API would reject invalid paths, sanitizing it would complete the defense-in-depth boundary.

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jun 22, 2026
fullsend-ai-coder Bot and others added 3 commits June 23, 2026 15:06
The post-review command posted review content directly to the GitHub
API without running it through the security output pipeline. When
invoked standalone (outside fullsend run), secrets and zero-width-
obfuscated tokens in agent output could reach the forge unredacted.

Call security.OutputPipeline().Scan() on the review body and finding
text fields (description, remediation) before any forge API call.
This matches the pattern used by fullsend scan output and the sandbox
post-tool hooks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…message

Address two review findings on PR #2444:

1. [incomplete-sanitization] Apply pipeline.Scan() to Severity and Category
   fields in ReviewFinding, which are interpolated into Markdown posted to
   the forge by formatFindingComment.

2. [misleading-log-message] Change warning from "Redacted N secret(s)" to
   "Sanitized review body (N finding(s))" since pipeline findings include
   unicode normalization, not just secrets.

Addresses review feedback on #2444
Add individual finding logging (scanner name + detail) after the summary
count line in sanitizeReviewResult, matching the established pattern in
scan.go:194-196 and run.go:1831.

Addresses review feedback on #2444
@ben-alkov
ben-alkov force-pushed the agent/1230-sanitize-post-review branch from 4cb8250 to 5dc2d8f Compare June 23, 2026 19:06
@fullsend-ai-review

fullsend-ai-review Bot commented Jun 23, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:09 PM UTC · Completed 7:17 PM UTC
Commit: 5dc2d8f · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jun 23, 2026

@ralphbean ralphbean left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ben-alkov
ben-alkov added this pull request to the merge queue Jun 23, 2026
Merged via the queue into main with commit 76d7cd6 Jun 23, 2026
20 checks passed
@ben-alkov
ben-alkov deleted the agent/1230-sanitize-post-review branch June 23, 2026 19:59
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jun 23, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 8:04 PM UTC · Completed 8:17 PM UTC
Commit: 5dc2d8f · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #2444fix(#1230): run OutputPipeline on post-review before posting to forge

Timeline

  1. 2026-05-20 — Issue #1230 filed (high-severity security bug: post-review bypasses OutputPipeline).
  2. 2026-06-18 — Code agent creates PR #2444 with initial fix (+59 code, +111 tests). Sanitizes Body, Description, and Remediation fields but misses Severity, Category, and File.
  3. 2026-06-18 — Review agent (iteration 1) finds medium [incomplete-sanitization] (Severity/Category not sanitized) and low [misleading-log-message].
  4. 2026-06-22 — Human triggers /fs-fix. Fix agent addresses both findings (commit 2). Review agent (iteration 2) confirms medium resolved but surfaces 2 new low findings ([logging-pattern], [repetitive-pattern]) that existed in the original code.
  5. 2026-06-22 — Human triggers targeted /fs-fix for [logging-pattern]. Fix agent adds per-finding logging (commit 3). Review agent (iteration 3) surfaces yet another new low finding ([data-exposure] on File field) and morphs [logging-pattern] into [logging-gap].
  6. 2026-06-23 — Review agent (iteration 4) approves. Human approves ("LGTM"). PR merged.

Total: 3 commits, 4 review iterations, 2 fix agent runs, 7 inline review comments. Only the first review's medium finding was genuinely important. The remaining iterations addressed low-severity nits that were present from the start.

Workflow quality assessment

Dimension Rating Notes
Review quality Mixed Correctly caught the medium incomplete-sanitization bug, but exhibited a "finding treadmill" — surfacing new low-severity findings in each subsequent iteration that existed in the original code.
Rework rate High 2 fix cycles for findings the first review should have caught.
Token cost Elevated 4 review runs + 2 fix runs + associated CI for what was a straightforward 59-line security patch.
Time to resolution 5 days Mostly human latency (PR created June 18, first /fs-fix on June 22). Agent turnaround was fast.

Key observations

  1. Finding treadmill: The review agent found [logging-pattern] and [repetitive-pattern] in iteration 2, and [data-exposure] in iteration 3 — all low-severity findings that existed in the code from commit 1. The review agent inspected the same function in iteration 1 (it found the related Severity/Category gap) but failed to surface these findings until later passes.

  2. Marginal findings: The [data-exposure] finding on the File field was flagged despite the review agent's own acknowledgment that "File is a structured GitHub API path parameter and the API would reject invalid paths." The [repetitive-pattern] finding (extract a helper) is a style suggestion, not a correctness issue.

  3. Triage guidance was incomplete: The triage agent recommended sanitizing "the review body and inline comment bodies" but didn't enumerate all fields in the ReviewFinding struct. The code agent followed this guidance faithfully but missed fields the triage didn't mention.

Proposals

No new proposals filed. All improvement areas identified in this retro are well-covered by existing open issues:

  • Finding treadmill / first-pass completeness: #1367 (surfaces new low findings across passes), #1582 (catch all findings in first pass)
  • Low-severity findings driving rework: #2029 (use COMMENT verdict for low-only re-reviews), #1368 (approval with residual findings drives rework)
  • Suppressing non-actionable findings: #1881 (suppress findings the agent deems non-actionable)
  • Code agent field coverage: #1214 (analyze all usage sites), #2008 (validate against triage acceptance criteria)
  • Review iteration deduplication: #1013, #1212 (both recently closed — fixes may be in flight)

This PR is a good test case for validating whether progress on these issues reduces churn. The next time a similar sanitization PR goes through the pipeline, we should see fewer review iterations if #1367 and #1582 are addressed.

ifireball pushed a commit to ifireball/fullsend that referenced this pull request Jun 24, 2026
…eading log message

Address two review findings on PR fullsend-ai#2444:

1. [incomplete-sanitization] Apply pipeline.Scan() to Severity and Category
   fields in ReviewFinding, which are interpolated into Markdown posted to
   the forge by formatFindingComment.

2. [misleading-log-message] Change warning from "Redacted N secret(s)" to
   "Sanitized review body (N finding(s))" since pipeline findings include
   unicode normalization, not just secrets.

Addresses review feedback on fullsend-ai#2444
ifireball pushed a commit to ifireball/fullsend that referenced this pull request Jun 24, 2026
Add individual finding logging (scanner name + detail) after the summary
count line in sanitizeReviewResult, matching the established pattern in
scan.go:194-196 and run.go:1831.

Addresses review feedback on fullsend-ai#2444
ifireball pushed a commit to ifireball/fullsend that referenced this pull request Jun 24, 2026
…ize-post-review

fix(fullsend-ai#1230): run OutputPipeline on post-review before posting to forge
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge security Security threat model and related concerns type/bug Confirmed defect in existing behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

security: run OutputPipeline on post-review before forge API post

2 participants