Skip to content

Commit

Permalink
Use new client created for PnP ops in purge script (#5442)
Browse files Browse the repository at this point in the history
PowerShell switched to requiring certificate credentials so the existing cleanup jobs have been failing since the switch

---

#### Does this PR need a docs update or release note?

- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
  • Loading branch information
ashmrtn authored Sep 17, 2024
1 parent d9bf48b commit b086f8c
Show file tree
Hide file tree
Showing 17 changed files with 177 additions and 92 deletions.
1 change: 1 addition & 0 deletions .github/actions/backup-restore-test/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Backup Restore Test
description: Run various backup/restore/export tests for a service.

inputs:
service:
Expand Down
1 change: 1 addition & 0 deletions .github/actions/go-setup-cache/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Setup and Cache Golang
description: Build golang binaries for later use in CI.

# clone of: https://github.com/magnetikonline/action-golang-cache/blob/main/action.yaml
#
Expand Down
1 change: 1 addition & 0 deletions .github/actions/publish-binary/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Publish Binary
description: Publish binary artifacts.

inputs:
version:
Expand Down
1 change: 1 addition & 0 deletions .github/actions/publish-website/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Publish Website
description: Publish website artifacts.

inputs:
aws-iam-role:
Expand Down
20 changes: 15 additions & 5 deletions .github/actions/purge-m365-data/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Purge M365 User Data
description: Deletes M365 data generated during CI tests.

# Hard deletion of an m365 user's data. Our CI processes create a lot
# of data churn (creation and immediate deletion) of files, the likes
Expand Down Expand Up @@ -30,12 +31,19 @@ inputs:
description: Secret value of for AZURE_CLIENT_ID
azure-client-secret:
description: Secret value of for AZURE_CLIENT_SECRET
azure-pnp-client-id:
description: Secret value of AZURE_PNP_CLIENT_ID
azure-pnp-client-cert:
description: Base64 encoded private certificate for the azure-pnp-client-id (Secret value of AZURE_PNP_CLIENT_CERT)
azure-tenant-id:
description: Secret value of for AZURE_TENANT_ID
description: Secret value of AZURE_TENANT_ID
m365-admin-user:
description: Secret value of for M365_TENANT_ADMIN_USER
m365-admin-password:
description: Secret value of for M365_TENANT_ADMIN_PASSWORD
tenant-domain:
description: The domain of the tenant (ex. 10rqc2.onmicrosft.com)
required: true

runs:
using: composite
Expand Down Expand Up @@ -80,8 +88,9 @@ runs:
shell: pwsh
working-directory: ./src/cmd/purge/scripts
env:
M365_TENANT_ADMIN_USER: ${{ inputs.m365-admin-user }}
M365_TENANT_ADMIN_PASSWORD: ${{ inputs.m365-admin-password }}
AZURE_CLIENT_ID: ${{ inputs.azure-pnp-client-id }}
AZURE_APP_CERT: ${{ inputs.azure-pnp-client-cert }}
TENANT_DOMAIN: ${{ inputs.tenant-domain }}
run: |
for ($ATTEMPT_NUM = 1; $ATTEMPT_NUM -le 3; $ATTEMPT_NUM++)
{
Expand All @@ -99,8 +108,9 @@ runs:
shell: pwsh
working-directory: ./src/cmd/purge/scripts
env:
M365_TENANT_ADMIN_USER: ${{ inputs.m365-admin-user }}
M365_TENANT_ADMIN_PASSWORD: ${{ inputs.m365-admin-password }}
AZURE_CLIENT_ID: ${{ inputs.azure-pnp-client-id }}
AZURE_APP_CERT: ${{ inputs.azure-pnp-client-cert }}
TENANT_DOMAIN: ${{ inputs.tenant-domain }}
run: |
for ($ATTEMPT_NUM = 1; $ATTEMPT_NUM -le 3; $ATTEMPT_NUM++)
{
Expand Down
1 change: 1 addition & 0 deletions .github/actions/teams-message/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Send a message to Teams
description: Send messages to communication apps.

inputs:
msg:
Expand Down
1 change: 1 addition & 0 deletions .github/actions/website-linting/action.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Lint Website
description: Lint website content.

inputs:
version:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/binary-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,5 @@ jobs:
if: failure()
uses: ./.github/actions/teams-message
with:
msg: "[FAILED] Publishing Binary"
msg: "[CORSO FAILED] Publishing Binary"
teams_url: ${{ secrets.TEAMS_CORSO_CI_WEBHOOK_URL }}
14 changes: 10 additions & 4 deletions .github/workflows/ci_test_cleanup.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
continue-on-error: true
strategy:
matrix:
user: [ CORSO_M365_TEST_USER_ID, CORSO_SECONDARY_M365_TEST_USER_ID, '' ]
user: [CORSO_M365_TEST_USER_ID, CORSO_SECONDARY_M365_TEST_USER_ID, ""]

steps:
- uses: actions/checkout@v4
Expand All @@ -33,12 +33,15 @@ jobs:
azure-tenant-id: ${{ secrets.TENANT_ID }}
m365-admin-user: ${{ secrets.M365_TENANT_ADMIN_USER }}
m365-admin-password: ${{ secrets.M365_TENANT_ADMIN_PASSWORD }}
azure-pnp-client-id: ${{ secrets.AZURE_PNP_CLIENT_ID }}
azure-pnp-client-cert: ${{ secrets.AZURE_PNP_CLIENT_CERT }}
tenant-domain: ${{ vars.TENANT_DOMAIN }}

- name: Notify failure in teams
if: failure()
uses: ./.github/actions/teams-message
with:
msg: "[FAILED] ${{ vars[matrix.user] }} CI Cleanup"
msg: "[CORSO FAILED] ${{ vars[matrix.user] }} CI Cleanup"
teams_url: ${{ secrets.TEAMS_CORSO_CI_WEBHOOK_URL }}

Test-Site-Data-Cleanup:
Expand All @@ -47,7 +50,7 @@ jobs:
continue-on-error: true
strategy:
matrix:
site: [ CORSO_M365_TEST_SITE_URL, CORSO_M365_TEST_GROUPS_SITE_URL ]
site: [CORSO_M365_TEST_SITE_URL, CORSO_M365_TEST_GROUPS_SITE_URL]

steps:
- uses: actions/checkout@v4
Expand All @@ -70,10 +73,13 @@ jobs:
azure-tenant-id: ${{ secrets.TENANT_ID }}
m365-admin-user: ${{ secrets.M365_TENANT_ADMIN_USER }}
m365-admin-password: ${{ secrets.M365_TENANT_ADMIN_PASSWORD }}
azure-pnp-client-id: ${{ secrets.AZURE_PNP_CLIENT_ID }}
azure-pnp-client-cert: ${{ secrets.AZURE_PNP_CLIENT_CERT }}
tenant-domain: ${{ vars.TENANT_DOMAIN }}

- name: Notify failure in teams
if: failure()
uses: ./.github/actions/teams-message
with:
msg: "[FAILED] ${{ vars[matrix.site] }} CI Cleanup"
msg: "[CORSO FAILED] ${{ vars[matrix.site] }} CI Cleanup"
teams_url: ${{ secrets.TEAMS_CORSO_CI_WEBHOOK_URL }}
3 changes: 3 additions & 0 deletions .github/workflows/load_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,6 @@ jobs:
azure-tenant-id: ${{ secrets.TENANT_ID }}
m365-admin-user: ${{ secrets.M365_TENANT_ADMIN_USER }}
m365-admin-password: ${{ secrets.M365_TENANT_ADMIN_PASSWORD }}
azure-pnp-client-id: ${{ secrets.AZURE_PNP_CLIENT_ID }}
azure-pnp-client-cert: ${{ secrets.AZURE_PNP_CLIENT_CERT }}
tenant-domain: ${{ vars.TENANT_DOMAIN }}
16 changes: 8 additions & 8 deletions .github/workflows/longevity_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
workflow_dispatch:
inputs:
user:
description: 'User to run longevity test on'
description: "User to run longevity test on"

permissions:
# required to retrieve AWS credentials
Expand All @@ -23,7 +23,7 @@ jobs:
uses: alcionai/corso/.github/workflows/accSelector.yaml@main

Longevity-Tests:
needs: [ SetM365App ]
needs: [SetM365App]
environment: Testing
runs-on: ubuntu-latest
env:
Expand All @@ -37,7 +37,7 @@ jobs:
CORSO_LOG_FILE: ${{ github.workspace }}/src/testlog/run-longevity.log
RESTORE_DEST_PFX: Corso_Test_Longevity_
TEST_USER: ${{ github.event.inputs.user != '' && github.event.inputs.user || vars.CORSO_M365_TEST_USER_ID }}
PREFIX: 'longevity'
PREFIX: "longevity"

# Options for retention.
RETENTION_MODE: GOVERNANCE
Expand All @@ -46,7 +46,7 @@ jobs:
defaults:
run:
working-directory: src

############################################################################
# setup
steps:
Expand Down Expand Up @@ -78,7 +78,7 @@ jobs:
- run: go build -o corso
timeout-minutes: 10

- run: mkdir ${CORSO_LOG_DIR}

# Use shorter-lived credentials obtained from assume-role since these
Expand Down Expand Up @@ -163,7 +163,7 @@ jobs:
data=$( echo $resultjson | jq -r '.[0] | .id' )
echo result=$data >> $GITHUB_OUTPUT
##########################################################################
# Onedrive

Expand Down Expand Up @@ -328,7 +328,7 @@ jobs:
--hide-progress \
--force \
--json \
2>&1 | tee ${{ env.CORSO_LOG_DIR }}/maintenance_metadata.txt
2>&1 | tee ${{ env.CORSO_LOG_DIR }}/maintenance_metadata.txt
- name: Maintenance test Weekly
id: maintenance-test-weekly
Expand Down Expand Up @@ -392,5 +392,5 @@ jobs:
if: failure()
uses: ./.github/actions/teams-message
with:
msg: "[FAILED] Longevity Test"
msg: "[CORSO FAILED] Longevity Test"
teams_url: ${{ secrets.TEAMS_CORSO_CI_WEBHOOK_URL }}
8 changes: 4 additions & 4 deletions .github/workflows/nightly_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
# ----------------------------------------------------------------------------------------------------

Test-Suite-Trusted:
needs: [ Checkout, SetM365App]
needs: [Checkout, SetM365App]
environment: Testing
runs-on: ubuntu-latest
defaults:
Expand Down Expand Up @@ -100,9 +100,9 @@ jobs:
-timeout 2h \
./... 2>&1 | tee ./testlog/gotest-nightly.log | gotestfmt -hide successful-tests
##########################################################################################################################################
##########################################################################################################################################

# Logging & Notifications
# Logging & Notifications

# Upload the original go test output as an artifact for later review.
- name: Upload test log
Expand All @@ -118,5 +118,5 @@ jobs:
if: failure()
uses: ./.github/actions/teams-message
with:
msg: "[FAILED] Nightly Checks"
msg: "[COROS FAILED] Nightly Checks"
teams_url: ${{ secrets.TEAMS_CORSO_CI_WEBHOOK_URL }}
Loading

0 comments on commit b086f8c

Please sign in to comment.