Feature: Allow PR status checks to be managed external to safe settings

README.md

@@ -123,6 +123,46 @@ overridevalidators:
 
 A sample of `deployment-settings` file is found [here](docs/sample-settings/sample-deployment-settings.yml).
 
+### Custom Status Checks
+For branch protection rules and rulesets, you can allow for status checks to be defined outside of safe-settings together with your usual safe settings.
+
+This can be defined at the org, sub-org, and repo level.
+
+To configure this for branch protection rules, specify `{{EXTERNALLY_DEFINED}}` under the `contexts` keyword:
+```yaml
+branches:
+  - name: main
+    protection:
+      ...
+      required_status_checks:
+        contexts:
+          - "{{EXTERNALLY_DEFINED}}"
+```
+
+For rulesets, specify `{{EXTERNALLY_DEFINED}}` under the `required_status_checks` keyword:
+```yaml
+rulesets:
+  - name: Status Checks
+    ...
+    rules:
+      - type: required_status_checks
+        parameters:
+          required_status_checks:
+            - context: "{{EXTERNALLY_DEFINED}}"
+```
+
+Notes:
+  - For the same branch that is covered by multi-level branch protection rules, contexts defined at the org level are merged into the sub-org and repo level contexts, while contexts defined at the sub-org level are merged into the repo level contexts.
+  - Rules from the sub-org level are merged into the repo level when their ruleset share the same name. Becareful not to define the same rule type in both levels as it will be rejected by GitHub.
+  - When `{{EXTERNALLY_DEFINED}}` is defined for a new branch protection rule or ruleset configuration, they will be deployed with no status checks.
+  - When an existing branch protection rule or ruleset configuration is amended with `{{EXTERNALLY_DEFINED}}`, the status checks in the existing rules in GitHub will remain as is.
+
+> ⚠️ **Warning:**
+When `{{EXTERNALLY_DEFINED}}` is removed from an existing branch protection rule or ruleset configuration, the status checks in the existing rules in GitHub will revert to the checks that are defined in safe-settings. From this point onwards, all status checks configured through the GitHub UI will be reverted back to the safe-settings configuration.
+
+#### Status checks inheritance across scopes
+Refer to [Status checks](docs/status-checks.md).
+
 ### Performance
 When there are 1000s of repos to be managed -- and there is a global settings change -- safe-settings will have to work efficiently and only make the necessary API calls.
 

docs/status-checks.md

@@ -0,0 +1,172 @@
+## Status checks inheritance across scopes
+
+### Rulesets
+
+The status checks between organisation and repository rulesets are independent of each together.
+
+In the following examples, a common ruleset name is used at all levels. Repo1 and Repo2 are managed at the Sub-org level.
+
+#### No custom checks
+```
+Org checks:
+  Org Check
+Sub-org checks:
+  Sub-org Check
+Repo checks for Repo2:
+  Repo Check
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Org: Org Check
+    - Repo1: Sub-org Check
+    - Repo2: _Failed to deploy as required_status_checks can't be defined twice in both sub-org and repo level_
+  - Updating status checks via GitHub UI:
+    - Org: Status checks reverted back to safe settings
+    - Repo1: Status checks reverted back to safe settings
+    - Repo2: NA
+
+#### No custom checks 2
+```
+Org checks:
+  Org Check
+Sub-org checks:
+  Sub-org Check
+Repo checks for Repo2:
+  _NONE_
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Org: Org Check
+    - Repo1: Sub-org Check
+    - Repo2: _NONE_
+  - Updating status checks via GitHub UI:
+    - Org: Status checks reverted back to safe settings
+    - Repo1: Status checks reverted back to safe settings
+    - Repo2: Custom status checks are retained
+
+**The remaining tests will leave Repo2 out of the Sub-org.**
+
+#### Custom checks enabled at the Org and Sub-org level
+```
+Org:
+  Org Check
+  {{EXTERNALLY_DEFINED}}
+Sub-org checks:
+  Sub-org Check
+  {{EXTERNALLY_DEFINED}}
+Repo checks for Repo2:
+  Repo Check
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Org: []
+    - Repo1: []
+    - Repo2: Repo Check
+  - Updating status checks via GitHub UI:
+    - Org: Custom status checks are retained
+    - Repo1: Custom status checks are retained
+    - Repo2: Status checks reverted back to safe settings
+
+#### Custom checks enabled at the Repo level
+```
+Org:
+  Org Check
+Sub-org checks:
+  Sub-org Check
+Repo checks for Repo2:
+  Repo Check
+  {{EXTERNALLY_DEFINED}}
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Org: Org Check
+    - Repo1: Sub-org Check
+    - Repo2: []
+  - Updating status checks via GitHub UI:
+    - Org: Status checks reverted back to safe settings
+    - Repo1: Status checks reverted back to safe settings
+    - Repo2: Custom status checks are retained
+
+
+### Branch protection rules
+
+In the following examples the `main` branch is protected at all levels. Repo1 and Repo2 are managed at the Sub-org level.
+
+#### No custom checks
+```
+Org checks:
+  Org Check
+Sub-org checks:
+  Sub-org Check
+Repo checks for Repo2:
+  Repo Check
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Repo1: Org Check, Sub-org Check
+    - Repo2: Org Check, Sub-org Check, Repo Check
+  - Updating status checks via GitHub UI:
+    - Repo1: Status checks reverted back to safe settings
+    - Repo2: Status checks reverted back to safe settings
+
+#### Custom checks enabled at the Org level
+```
+Org:
+  Org Check
+  {{EXTERNALLY_DEFINED}}
+Sub-org checks:
+  Sub-org Check
+Repo checks for Repo2:
+  Repo Check
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Repo1: []
+    - Repo2: []
+  - Updating status checks via GitHub UI:
+    - Repo1: Custom status checks are retained
+    - Repo2: Custom status checks are retained
+
+#### Custom checks enabled at the Sub-org level
+```
+Org:
+  Org Check
+Sub-org checks:
+  Sub-org Check
+  {{EXTERNALLY_DEFINED}}
+Repo checks for Repo2:
+  Repo Check
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Repo1: []
+    - Repo2: []
+  - Updating status checks via GitHub UI:
+    - Repo1: Custom status checks are retained
+    - Repo2: Custom status checks are retained
+
+#### Custom checks enabled at the Repo level
+```
+Org:
+  Org Check
+Sub-org checks:
+  Sub-org Check
+Repo checks for Repo2:
+  Repo Check
+  {{EXTERNALLY_DEFINED}}
+```
+
+Status checks:
+  - Newly deployed rules:
+    - Repo1: Org Check, Sub-org Check
+    - Repo2: []
+  - Updating status checks via GitHub UI:
+    - Repo1: Status checks reverted back to safe settings
+    - Repo2: Custom status checks are retained

lib/plugins/branches.js

@@ -1,15 +1,22 @@
 const ErrorStash = require('./errorStash')
 const NopCommand = require('../nopcommand')
 const MergeDeep = require('../mergeDeep')
+const Overrides = require('./overrides')
 const ignorableFields = []
 const previewHeaders = { accept: 'application/vnd.github.hellcat-preview+json,application/vnd.github.luke-cage-preview+json,application/vnd.github.zzzax-preview+json' }
+const overrides = {
+  'contexts': {
+    'action': 'reset',
+    'type': 'array'
+  },
+}
 
 module.exports = class Branches extends ErrorStash {
   constructor (nop, github, repo, settings, log, errors) {
     super(errors)
     this.github = github
     this.repo = repo
-    this.branches = settings
+    this.branches = structuredClone(settings)
     this.log = log
     this.nop = nop
   }
@@ -49,7 +56,7 @@ module.exports = class Branches extends ErrorStash {
               const params = Object.assign({}, p)
               return this.github.repos.getBranchProtection(params).then((result) => {
                 const mergeDeep = new MergeDeep(this.log, this.github, ignorableFields)
-                const changes = mergeDeep.compareDeep({ branch: { protection: this.reformatAndReturnBranchProtection(result.data) } }, { branch: { protection: branch.protection } })
+                const changes = mergeDeep.compareDeep({ branch: { protection: this.reformatAndReturnBranchProtection(result.data) } }, { branch: { protection: Overrides.removeOverrides(overrides, branch.protection, result.data) } })
                 const results = { msg: `Followings changes will be applied to the branch protection for ${params.branch.name} branch`, additions: changes.additions, modifications: changes.modifications, deletions: changes.deletions }
                 this.log.debug(`Result of compareDeep = ${results}`)
 
@@ -76,7 +83,7 @@ module.exports = class Branches extends ErrorStash {
                 return this.github.repos.updateBranchProtection(params).then(res => this.log(`Branch protection applied successfully ${JSON.stringify(res.url)}`)).catch(e => { this.logError(`Error applying branch protection ${JSON.stringify(e)}`); return [] })
               }).catch((e) => {
                 if (e.status === 404) {
-                  Object.assign(params, branch.protection, { headers: previewHeaders })
+                  Object.assign(params, Overrides.removeOverrides(overrides, branch.protection, {}), { headers: previewHeaders })
                   if (this.nop) {
                     resArray.push(new NopCommand(this.constructor.name, this.repo, this.github.repos.updateBranchProtection.endpoint(params), 'Add Branch Protection'))
                     return Promise.resolve(resArray)

lib/plugins/overrides.js

@@ -0,0 +1,100 @@
+const ErrorStash = require('./errorStash')
+
+module.exports = class Overrides extends ErrorStash {
+  // Find all object references for a given key from the source.
+  static getObjectRef (source, dataKey) {
+    const results = []
+    const traverse = (obj) => {
+      for (const key in obj) {
+        if (key === dataKey) {
+          results.push(obj)
+        } else if (Array.isArray(obj[key])) {
+          obj[key].forEach(element => traverse(element))
+        } else if (typeof obj[key] === 'object' && obj[key]) {
+          traverse(obj[key])
+        }
+      }
+    }
+    traverse(source)
+    return results
+  }
+
+  // Find the parent object reference for a given child object and
+  // allow the option to remove the parent object from the source.
+  static getParentObjectRef (source, child, remove = false) {
+    let parent = null
+    const traverse = (obj, parentObj = null, parentKey = '') => {
+      for (const key in obj) {
+        if (obj[key] === child) {
+          parent = obj
+          if (remove && parentObj && parentKey) {
+            delete parentObj[parentKey]
+          }
+        } else if (Array.isArray(obj[key])) {
+          obj[key].forEach((element, index) => {
+            if (element === child) {
+              parent = obj[key]
+              if (remove) {
+                obj[key].splice(index, 1)
+              }
+              return
+            }
+            traverse(element)
+          })
+        } else if (typeof obj[key] === 'object' && obj[key]) {
+          traverse(obj[key], obj, key)
+        }
+      }
+    }
+    traverse(source)
+    return parent
+  }
+
+  // Traverse the source and remove the top level parent object
+  static removeTopLevelParent (source, child, levels) {
+    let parent = child
+    for (let i = 0; i < levels; i++) {
+      if (i + 1 === levels) {
+        parent = Overrides.getParentObjectRef(source, parent, true)
+      } else {
+        parent = Overrides.getParentObjectRef(source, parent, false)
+      }
+    }
+  }
+
+  // When {{EXTERNALLY_DEFINED}} is found in the override value, retain the
+  // existing value from GitHub. If GitHub does not have a value, then
+  //   - A/ If the action is delete, then remove the top level parent object
+  //        and the override value from the source.
+  //   - B/ Otherwise, initialise the value to an appropriate value.
+  // Note:
+  //   - The admin settings could define multiple status check rules for a
+  //     ruleset, but the GitHub API retains one only, i.e. the last one.
+  //   - The PUT method for rulesets (update) allows for multiple overrides.
+  //   - The POST method for rulesets (create) allows for one override only.
+  static removeOverrides (overrides, source, existing) {
+    Object.entries(overrides).forEach(([override, props]) => {
+      let sourceRefs = Overrides.getObjectRef(source, override)
+      let data = JSON.stringify(sourceRefs)
+
+      if (data.includes('{{EXTERNALLY_DEFINED}}')) {
+        let existingRefs = Overrides.getObjectRef(existing, override)
+        sourceRefs.forEach(sourceRef => {
+          if (existingRefs[0]) {
+            sourceRef[override] = existingRefs[0][override]
+          } else if (props['action'] === 'delete') {
+            Overrides.removeTopLevelParent(source, sourceRef[override], props['parents'])
+            delete sourceRef[override]
+          } else if (props['type'] === 'array') {
+            sourceRef[override] = []
+          } else if (props['type'] === 'dict') {
+            sourceRef[override] = {}
+          } else {
+            throw new Error(`Unknown type ${props['type']} for ${override}`)
+          }
+        })
+      }
+    })
+    return source
+  }
+}