Feature: Allow PR status checks to be managed external to safe settings

[jitran]

Background

There have been a number of requests from users who have implemented branch protection rules and rulesets across their organisations, but would like their PR status checks to be defined outside of safe settings.

When these rules are defined at the repository or suborg level, it's not so bad, as the safe settings PR status checks only affect a subset of repositories. Users have a bit more control over their safe settings configuration.

When the rules are defined at the org level, it's a bit more problematic, as the safe settings PR status checks affect all repositories in the org.

Changes

• Added the ability for branch protection rules and rulesets' status checks to be managed outside of safe settings.
• When users specify {{EXTERNALLY_DEFINED}} in the status checks for their branch protection rules or rulesets, safe settings will not revert status checks in GitHub.
• Created an Overrides module to handle externally defined settings.
• Updated usage documentation in the README.
• Added unit tests for the new functionality.

Tests

Ensured all unit tests pass.

Manual testing with rulesets (repo, suborg, and org levels)

The following was tested in sequential order:

1. Started with no rulesets in GitHub for a repository
2. Created a new safe setting for a ruleset that had {{EXTERNALLY_DEFINED}}defined in the status checks
   • Ruleset was deployed with an empty list of status checks
3. Added custom status checks to the new ruleset via GitHub UI
   • The ruleset's custom status checks in GitHub were preserved
4. Updated the ruleset's status checks via the GitHub UI
   • The ruleset's custom status checks in GitHub were preserved
5. Added a new status check to the existing ruleset in safe settings and left {{EXTERNALLY_DEFINED}} in the list
   • The ruleset's custom status checks in GitHub were preserved
6. Removed all status checks from the ruleset via the GitHub UI
   • The ruleset's empty status checks in GitHub was preserved
7. Removed {{EXTERNALLY_DEFINED}} from the existing ruleset's status checks in safe settings
   • The ruleset's status checks in GitHub was reverted to the status checks from safe settings
8. Removed all status checks from the existing ruleset in safe settings
   • The ruleset's status checks in GitHub was reverted to the status checks from safe settings
9. Removed the ruleset from GitHub
   • Safe settings restored the ruleset
10. Added custom status checks to the ruleset via GitHub UI
    • The ruleset's status checks was reverted to the status checks from safe settings
11. Added a completely new ruleset (branch|tag|push) via GitHub UI
    • Safe settings removed all new rulesets

Manual testing with branch protection rules (repo, suborg, and org levels)

The following was tested in sequential order:

1. Deleted all branch protection rules from a repository
2. Created two branch protection rules in safe settings; the first rule had {{EXTERNALLY_DEFINED}} in the status checks, and the second rule had two status checks defined
   • Two branch protection rules were deployed into GitHub, where the first had no status checks, and the second had two status checks defined.
3. Updated the status checks for the first rule via GitHub UI
   • The custom status checks for the first rule in GitHub were preserved
4. Updated the status checks for the second rule via GitHub UI
   • The custom status checks for the second rule in GitHub were reverted back to the safe settings

[Copilot]

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

lib/plugins/branches.js:53

• Redundant check for 'protection.allow_deletions'. It should be simplified to 'protection.allow_deletions = protection.allow_deletions && protection.allow_deletions.enabled'.

protection.allow_deletions = protection.allow_deletions && protection.allow_deletions && protection.allow_deletions.enabled

Tip: Leave feedback on Copilot's review comments with the 👎 and 👍 buttons to help improve review quality. Learn more

[igorcosta]

@jitran , Thanks for the detailed work on this feature! I’ve reviewed the changes, and overall, this is a solid implementation that brings much-needed flexibility to managing PR status checks. That said, I have a few suggestions to improve error handling, documentation clarity, and test coverage:

• In branches.js and rulesets.js, there are scenarios where API failures (e.g., 500-series errors) are logged but not surfaced to the user. Consider adding retry logic for transient errors and ensuring failures are gracefully handled with actionable error messages.

• Include a warning about potential desync when rulesets are modified directly via the GitHub UI.

• In overrides.js, it would be good to add safeguards in removeOverrides to handle malformed configurations or unexpected data structures more robustly.

• Add an example showing how {{EXTERNALLY_DEFINED}} interacts with hierarchical rulesets (org, repo, and sub-org levels).

Let me know your thoughts on these suggestions!

Nice job!

[jitran]

Hi @igorcosta and @PendaGTP, I have rebased my branch with the latest default branch and validated that all the existing tests are still successful.

As for #741 (review):

• The documentation has been updated with a warning about potential de-syncs when rulesets are modified directly via the GitHub UI.
• Safe guards have been added to overrides.js to handle configurations and data structures more robustly.
• I have added examples of how {{EXTERNALLY_DEFINED}} interacts across Org, Sub-org, and Repo levels.
• I think a separate pull request should be raised to address the transient errors to keep this PR clean though :)

Finally, I had to rework the solution to handle the recent ruleset changes where it no longer allows the Require status checks to pass checkbox to be enabled with no status checks defined. It has added further complexity to the solution though.

[igorcosta]

LGTM thanks for revisiting and improving @jitran