Skip to content

E2E: Mattermost breadcrumbs for main-push + stop silent webhook drops - #96

Closed
yasserfaraazkhan wants to merge 2 commits into
masterfrom
e2e/webhook-ingest-and-main-watchdog
Closed

E2E: Mattermost breadcrumbs for main-push + stop silent webhook drops#96
yasserfaraazkhan wants to merge 2 commits into
masterfrom
e2e/webhook-ingest-and-main-watchdog

Conversation

@yasserfaraazkhan

@yasserfaraazkhan yasserfaraazkhan commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Minimal observability for “what happens in Matterwick after a mobile/desktop main merge?”

  • Stop silently dropping GitHub webhooks when remaining API tokens are under GitHubTokenReserve. That abort returned without processing the push, so main E2E never started and nothing was logged/alerted.
  • Mattermost breadcrumbs on the main/master push E2E path:
    • received push — provisioning…
    • provisioned N servers — dispatching…
    • workflow dispatched successfully
    • (existing) failure / skip reasons
  • Cap webhook body size (MaxBytesReader) and Mattermost webhook HTTP timeout (CodeRabbit).

No watchdog, self-heal, goSafe, quorum changes, or product-repo contract changes.

Test plan

  • Merge or simulate a mattermost-mobile main push and confirm Mattermost shows the receive → provision → dispatch sequence
  • Confirm provision failure still posts the existing “did not run” alert
  • Confirm webhooks still process when GitHub remaining tokens are near the reserve
  • Oversized webhook body returns 413
NONE

@mm-cloud-bot

Copy link
Copy Markdown

@yasserfaraazkhan: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

I understand the commands that are listed here

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 79ed6ee3-fea7-41d0-97ce-0d4cdfea1c68

📥 Commits

Reviewing files that changed from the base of the PR and between 89bdb3f and da75b77.

📒 Files selected for processing (2)
  • server/server.go
  • server/webhook.go

📝 Walkthrough

Walkthrough

Webhook ingestion no longer uses GitHub API rate limits to reject deliveries. It now limits request bodies to 10 MiB and handles read errors. Push E2E handling logs commit SHAs and sends Mattermost updates for skips, provisioning, and workflow dispatch. Webhook requests now use a timeout and close response bodies.

Changes

Webhook ingestion and E2E reporting

Layer / File(s) Summary
Webhook ingestion and request handling
server/server.go, server/limit_rate_gh.go
githubEvent no longer applies the rate-limit gate. It limits request bodies to 10 MiB, handles oversized and other read failures, and retains signature validation after successful reads.
Push E2E status reporting
server/push_events.go, server/utils.go
Push handling logs commit SHAs and reports skipped repositories, missing SHAs, provisioning progress, successful provisioning, and workflow dispatch through Mattermost. Notification handling is delegated to notifyMattermost.
Webhook HTTP client controls
server/webhook.go
sendToWebhook uses a 10-second HTTP client timeout and closes the response body after a successful request.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested labels: kind/bug

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the Mattermost breadcrumbs and webhook-drop prevention changes.
Description check ✅ Passed The description accurately explains the observability, webhook handling, timeout, and body-size changes.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch e2e/webhook-ingest-and-main-watchdog

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Nitpick comments (1)
server/e2e_watchdog_test.go (1)

91-91: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use context-aware test requests.

The httptest.NewRequest calls lack context. Pass context.Background() to provide context to the request, following Go testing conventions. This is a best practice for consistency with context-aware HTTP operations.

Proposed fix
-	req := httptest.NewRequest(http.MethodPost, "/github_event", bytes.NewReader(pingBody))
+	req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/github_event", bytes.NewReader(pingBody))
-	req := httptest.NewRequest(http.MethodPost, "/github_event", bytes.NewReader([]byte(`{}`)))
+	req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/github_event", bytes.NewReader([]byte(`{}`)))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/e2e_watchdog_test.go` at line 91, Update the httptest.NewRequest call
in the watchdog test to use the context-aware request constructor, passing
context.Background() while preserving the existing POST method, endpoint, and
pingBody.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@server/e2e_watchdog.go`:
- Around line 150-156: Remove the unconditional len(instances) fallback in the
watchdog logic so an active E2E run only suppresses the alert when
strings.HasSuffix(key, suffix) matches the current branch SHA. Add a test
covering an active earlier SHA alongside an uncovered current SHA, asserting
that the watchdog still alerts.
- Around line 202-207: Update the commit-time resolution near commitTime so it
no longer defaults to time.Now when both committer and author dates are absent.
Return or propagate an error from the surrounding reconciliation flow when
neither date is available, while preserving the committer-date preference and
author-date fallback when present.

In `@server/provision_errors.go`:
- Around line 31-39: The isCancellationError function incorrectly relies on
substring matching for "cancelled" and "canceled" to classify errors, which
causes wrapped provider errors containing those strings to be misclassified as
cancellation errors. Remove the substring matching logic that uses
strings.ToLower and strings.Contains in the isCancellationError function,
keeping only the identity-based check using errors.Is(err, context.Canceled).
Also add a regression test that verifies a provider error wrapped with format
string error wrapping containing the word "cancelled" is not misclassified as a
cancellation error.

In `@server/push_events.go`:
- Line 46: The worker goroutines created by
createMultipleE2EInstancesForPushEvent lack panic recovery, allowing panics to
leave results[idx] as a false success. Add deferred recovery in each worker that
records the recovered panic as results[idx].err and invokes cancel(), while
preserving wg.Done() execution.

In `@server/server.go`:
- Around line 203-208: Update the POST /github_event handler around io.ReadAll
to wrap r.Body with http.MaxBytesReader before reading, using the existing
request context and an appropriate body-size limit. Handle *http.MaxBytesError
with http.StatusRequestEntityTooLarge, while retaining http.StatusBadRequest for
all other read errors and preserving the current error logging and return flow.

---

Nitpick comments:
In `@server/e2e_watchdog_test.go`:
- Line 91: Update the httptest.NewRequest call in the watchdog test to use the
context-aware request constructor, passing context.Background() while preserving
the existing POST method, endpoint, and pingBody.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c7e789cd-047d-41e9-ac88-7853ab3d32bc

📥 Commits

Reviewing files that changed from the base of the PR and between 799a23b and b269fba.

📒 Files selected for processing (9)
  • server/e2e_tests.go
  • server/e2e_watchdog.go
  • server/e2e_watchdog_test.go
  • server/limit_rate_gh.go
  • server/provision_errors.go
  • server/pull_request.go
  • server/push_events.go
  • server/server.go
  • server/workflow_run.go

Comment thread server/e2e_watchdog.go Outdated
Comment thread server/e2e_watchdog.go Outdated
Comment thread server/provision_errors.go Outdated
Comment thread server/push_events.go Outdated
Comment thread server/server.go
@mm-cloud-bot mm-cloud-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed labels Aug 4, 2026
Stop silently dropping GitHub webhooks when API rate reserve is low, and
post Mattermost breadcrumbs for main/master push E2E (received →
provisioned → dispatched, or why it stopped).

Co-authored-by: Cursor <cursoragent@cursor.com>
@yasserfaraazkhan
yasserfaraazkhan force-pushed the e2e/webhook-ingest-and-main-watchdog branch from b269fba to 89bdb3f Compare August 4, 2026 16:24
@yasserfaraazkhan yasserfaraazkhan changed the title E2E: never drop webhooks; alert when main lacks e2e-test coverage E2E: Mattermost breadcrumbs for main-push + stop silent webhook drops Aug 4, 2026
@mm-cloud-bot

Copy link
Copy Markdown

@yasserfaraazkhan: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

I understand the commands that are listed here

@mm-cloud-bot mm-cloud-bot added do-not-merge/release-note-label-needed and removed release-note-none Denotes a PR that doesn't merit a release note. labels Aug 4, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@server/server.go`:
- Around line 180-182: Limit the request body in the GitHub webhook handler
before its io.ReadAll call by wrapping r.Body with http.MaxBytesReader. Handle
read failures by returning HTTP 413 for *http.MaxBytesError and HTTP 400 for all
other errors, while preserving the existing signature-validation flow.
- Around line 180-182: Add the repository-compliant release-note block for the
webhook ingestion and E2E status reporting change, placing it with the project’s
existing release-note metadata before merge. Keep the note focused on preventing
webhook deliveries from being dropped by GitHub API rate-reserve gating and
preserving E2E status reporting.

In `@server/utils.go`:
- Around line 16-17: Update sendToWebhook to use a finite request timeout so
slow Mattermost calls cannot block handlePushEventE2E during any lifecycle
stage. Apply the timeout to the HTTP client or request used by notifyMattermost,
preserving the existing webhook behavior while ensuring the call fails promptly.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9f578fba-3ddc-4567-80d4-30020d3445ef

📥 Commits

Reviewing files that changed from the base of the PR and between b269fba and 89bdb3f.

📒 Files selected for processing (4)
  • server/limit_rate_gh.go
  • server/push_events.go
  • server/server.go
  • server/utils.go
💤 Files with no reviewable changes (1)
  • server/limit_rate_gh.go

Comment thread server/server.go
Comment thread server/utils.go
Cap GitHub webhook body reads with MaxBytesReader and give Mattermost
webhook posts a finite HTTP timeout so notifications cannot hang E2E.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mm-cloud-bot mm-cloud-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed labels Aug 4, 2026
@yasserfaraazkhan

Copy link
Copy Markdown
Contributor Author

closing for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-note-none Denotes a PR that doesn't merit a release note.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants