Improve CMT reliability with partial provisioning fallback and smarter mobile version selection

[yasserfaraazkhan]

Summary

This change makes CMT more resilient to provisioning failures while reducing the provisioning load for mobile runs.

• Fixed CMT on release branches: the mobile CMT matrix again populates matrix.server.url, which every live build-release-* branch reads — without it those branches test against an empty server URL.

• Mobile CMT now selects by category — latest ESR, latest production, latest RC — cutting provisioning from 25 to 15 installations per run. Empty categories (no RC in flight, no ESR flagged) backfill with the next stable line.

• Desktop CMT is unchanged at up to five server versions and its existing selection logic.

• A server version that fails to provision is dropped instead of aborting the whole matrix, so CMT runs with the versions that came up. Reduced coverage is announced via the Mattermost webhook, naming kept and dropped versions. If every version fails, CMT still can't run — but it now says so instead of failing silently.

• Each installation gets one retry with a fresh DNS name, so a single transient provisioner failure no longer costs an entire version.

• Push-triggered E2E failures (provisioning, dispatch, zero instances) now post to the Mattermost webhook instead of only logging.

• E2EPRInstanceMaxAge reduced 24h → 8h to return capacity sooner, now that each mobile PR holds five servers instead of three

NONE

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6a5f1e20-aa45-453a-8aad-9962497544c9

📥 Commits

Reviewing files that changed from the base of the PR and between 90bb9fb and 5d137e8.

📒 Files selected for processing (3)

• server/e2e_dryrun_test.go
• server/e2e_tests.go
• server/e2e_tests_test.go

🚧 Files skipped from review as they are similar to previous changes (2)

• server/e2e_tests.go
• server/e2e_tests_test.go

---

📝 Walkthrough

Walkthrough

Changes

CMT provisioning and version selection

Layer / File(s)	Summary
Instance-type-aware CMT version resolution server/e2e_tests.go, server/e2e_dryrun_test.go	Release data is classified for desktop and mobile selection. Mobile selects ESR, stable, and RC versions with backfilling and fallback handling.
Version caps and range spanning server/workflow_run.go, server/e2e_tests_test.go, server/workflow_run_test.go	Desktop keeps the newest versions. Mobile preserves version-range endpoints within a three-version cap.
Retry-aware and best-effort provisioning server/e2e_tests.go, server/workflow_run.go, server/push_events.go, server/e2e_tests_test.go	Installations retry with fresh DNS names. Partial installations receive retrying cleanup. Failed or incomplete CMT versions are dropped. Provisioning failures generate Mattermost notifications.
Matrix compatibility and retention coverage server/workflow_run.go, server/e2e_dryrun_test.go, config/config-matterwick.default.json	Mobile matrix entries populate the compatibility url field. The default stale-instance age is eight hours. Tests cover matrix shape, URL alignment, platform parsing, and desktop compatibility.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• mattermost/matterwick#90: Modifies related CMT provisioning and workflow dispatch logic.
• mattermost/matterwick#92: Modifies related mobile CMT topology and version-capping behaviour.

Suggested labels: kind/bug

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main CMT reliability and mobile version-selection changes.
Description check	✅ Passed	The description directly covers the CMT provisioning, mobile selection, retry, notification, cleanup, and age-limit changes.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch reduce_overload_matterwick

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@config/config-matterwick.default.json`:
- Line 85: Update e2ePRInstanceMaxAge() to use an 8-hour fallback when
E2EPRInstanceMaxAge is unset, and revise its test to expect 8 hours instead of
24. Keep the configured-value behavior unchanged.

In `@server/push_events.go`:
- Around line 198-205: Update cleanupCreatedInstallation in the
createCloudInstallationWithRetry flow to handle DeleteInstallation failures
before another attempt: retry deletion or preserve the created installation ID
for later cleanup instead of discarding it. Add coverage for poll failure,
cancellation, and deletion failure after creation, while preserving fresh DNS
names per retry.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1ed9e773-98f2-4b2f-b65f-3aad6d7aebea

📥 Commits

Reviewing files that changed from the base of the PR and between e21db79 and 90bb9fb.

📒 Files selected for processing (7)

• config/config-matterwick.default.json
• server/e2e_dryrun_test.go
• server/e2e_tests.go
• server/e2e_tests_test.go
• server/push_events.go
• server/workflow_run.go
• server/workflow_run_test.go

[NARSimoes]

Thanks @yasserfaraazkhan looks good. @saturninoabril could you please also take look since you are familiar?

[saturninoabril]

A server version that fails to provision is dropped instead of aborting the whole matrix

Agree with this tradeoff so that failed installation won't stop CMT from dispatching at all. However, what's the retry path? Will there be a workflow that could re-dispatch the dropped versions so it can complete the CMT requirements?

[saturninoabril]

Thanks @yasserfaraazkhan!

Non-blocking, but worth checking the boundary between matterwick and the CI workflows, ex:

• matterwick: Actions trigger/listener, cloud test server management
• CI workflow - server versions

Mainly a separation of concerns: matterwirck owns infra lifecycle, the workflow owns the test policy. The workflow determines the versions and hand them over, while matterwick provisions what it's given and fails if it can't.

That keeps CMT versions declarative and maintainable in CI, where they can evolve with requirements.

[yasserfaraazkhan]

Non-blocking, but worth checking the boundary between matterwick and the CI workflows, ex:

• matterwick: Actions trigger/listener, cloud test server management
• CI workflow - server versions

@saturninoabril On the CI - matterwick : CI owning test policy and matterwick owning infra is the cleaner split. We actually tried that. The workflow declaring server_versions and handing them to matterwick. In practice workflow_run doesn’t expose those inputs to the listener, so we ended up with side channels (HTTP dispatch endpoints / artifact upload+download) that was not working and hard code all the logical sceanrios to handle.

That’s why #90 moved version resolution into matterwick and reduced the trigger workflow to a signal. Until GitHub gives us a reliable input path on workflow_run.

[yasserfaraazkhan]

what's the retry path? Will there be a workflow that could re-dispatch the dropped versions so it can complete the CMT requirements?

Dropped versions now get a targeted follow-up in matterwick. After the primary matrix runs, we wait for provisioner capacity, re-provision only the dropped versions, and if any recover we dispatch a second compatibility-matrix-testing.yml. This should happen automatically.