[POC] CI automation scripts #133

manika137 · 2024-07-26T07:50:19Z

Description of PR

Automating the daily script runs and saving the results on storage account.

Saves the branch name and commit number in the test-results file
Results saved as- container-name/YY-MM-DD/branch-name/result_files

Screenshot of the results stored.

anmolanmol1234 · 2024-07-26T08:37:26Z

Looks good, can you also share how the results are stored in the storage account ? Like a snapshot of the results stored.

manika137 · 2024-07-30T07:00:00Z

Looks good, can you also share how the results are stored in the storage account ? Like a snapshot of the results stored.

Added

anujmodi2021

Added a few suggestions, rest looks good.

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

…into automateScript

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

anmolanmol1234 · 2024-12-17T10:06:19Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh


-runHNSOAuthDFSTest()
-{
+runHNSOAuthDFSTest() {


these diffs are not needed

anmolanmol1234 · 2024-12-17T10:10:01Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh

-         exit 0
-         ;;
-      *) logOutput "ERROR: Invalid selection"
+  echo ' '


Same dont see any change between new code and old one for this selection. minimizing the diffs would be better for backporting

anmolanmol1234 · 2024-12-17T10:12:26Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

+  azureConfigFilePath="${accountSettingsDir}runresult${accountConfigFileSuffix}"
+  testResultsAccountName=$(xmlstarlet sel -t -v '//property[name = "fs.azure.test.results.account.name"]/value' -n $azureConfigFilePath)
+  testResultsAccountKey=$(xmlstarlet sel -t -v '//property[name = "fs.azure.test.results.account.key"]/value' -n $azureConfigFilePath)
+  branchName="${branchName,,}"


Are the two commas needed here ?

They are used to convert into lowercase letters in Bash

anmolanmol1234 · 2024-12-17T10:14:32Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

+if ! command -v az &> /dev/null
+then
+  echo "Azure CLI (az) could not be found. Installing Azure CLI..."
+sudo apt update


This should be under a wait process ? What happens if the installation fails

Right. Added a check for it

anujmodi2021

Added a few more comments.

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/cronjob.sh

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

anujmodi2021 · 2024-12-27T12:03:18Z

hadoop-tools/hadoop-azure/.gitignore

 dev-support/testlogs
 src/test/resources/accountSettings/*
 !src/test/resources/accountSettings/accountName_settings.xml.template
+dev-support/testrun-scripts/cronjob.sh


EOF Error:
Every file should end with a new line.

anujmodi2021

+1

anmolanmol1234 · 2025-01-15T12:28:07Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh

+      if [[ ! -f "$accountSettingsFile" ]];
+      then
+        logOutput "No settings present. Creating new settings file ($accountSettingsFile) from template"
+        cp src/test/resources/azure-auth-keys.xml.template $accountSettingsFile


Path should be taken in variable

anmolanmol1234 · 2025-01-15T12:28:46Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh

+fi

-if [ $runTest == true ]
+if [[ $runTest && $"$IS_CRON_JOB" != "true" ]]


The variable name format should be camelCase

anmolanmol1234 · 2025-01-15T12:30:06Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

+  AggregatedTestFolder="$testOutputLogFolder"
+
+  checkCronjobDependencies
+  az storage container create --name $containerName --account-name $testResultsAccountName --account-key "$testResultsAccountKey"


what happens if az copy command execution fails ?

…Script

anmolanmol1234 · 2025-02-24T06:08:07Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

    summarycontent
  } >> "$aggregatedTestResult"
-  printf "\n :::: AGGREGATED TEST RESULT :::: \n" >> "$aggregatedTestResult"
+  printf "\n----- Test results -----\n"


do we not need to redirect the results now ?

anujmodi2021 · 2025-02-25T10:25:31Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

+  testResultsAccountName=$(xmlstarlet sel -t -v '//property[name = "fs.azure.test.results.account.name"]/value' -n $azureConfigFilePath)
+  testResultsAccountKey=$(xmlstarlet sel -t -v '//property[name = "fs.azure.test.results.account.key"]/value' -n $azureConfigFilePath)
+  branchName="${branchName,,}"
+  containerName="$(xmlstarlet sel -t -v '//property[name = "fs.azure.container.name"]/value' -n $azureConfigFilePath)"


Nit: this can also be fs.azure.test.results.container.name?

anujmodi2021 · 2025-02-25T10:26:11Z

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh

+
+uploadToAzure() {
+  azureConfigFilePath="${accountSettingsDir}runresult${accountConfigFileSuffix}"
+  testResultsAccountName=$(xmlstarlet sel -t -v '//property[name = "fs.azure.test.results.account.name"]/value' -n $azureConfigFilePath)


What happens if cron job is running and these test result account details are not added?

CI automation scipts

bd91890

github-actions bot added trunk TOOLS ABFS labels Jul 26, 2024

anujmodi2021 requested changes Jul 30, 2024

View reviewed changes

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh Outdated Show resolved Hide resolved

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh Outdated Show resolved Hide resolved

manika137 added 2 commits August 5, 2024 01:49

Merge branch 'wasbDepCodeReview' of github.com:ABFSDriver/AbfsHadoop …

556ca8c

…into automateScript

Refactor results saving structure

c3f5935

anujmodi2021 changed the title ~~CI automation scipts~~ [POC] CI automation scipts Aug 28, 2024

manika137 added 5 commits November 21, 2024 01:36

Merge branch 'wasbDepCodeReview' of github.com:ABFSDriver/AbfsHadoop …

1d61fb6

…into automateScript

Merge branch 'wasbDepCodeReview' of github.com:ABFSDriver/AbfsHadoop …

131e66a

…into automateScript

changes

2752ca1

Merge branch 'wasbDepCodeReview' of github.com:ABFSDriver/AbfsHadoop …

37ba557

…into automateScript

Adding branch name, commit ID to the results file

2aae4ef

anujmodi2021 reviewed Dec 3, 2024

View reviewed changes

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh Outdated Show resolved Hide resolved

anujmodi2021 reviewed Dec 3, 2024

View reviewed changes

hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh Outdated Show resolved Hide resolved

discussed changes

4678058

anmolanmol1234 reviewed Dec 17, 2024

View reviewed changes

manika137 force-pushed the automateScript branch 2 times, most recently from 74a8d40 to d5e93eb Compare December 23, 2024 03:16

remove diffs

bfe34e1

manika137 force-pushed the automateScript branch from d5e93eb to bfe34e1 Compare December 23, 2024 03:17

manika137 changed the title ~~[POC] CI automation scipts~~ [POC] CI automation scripts Dec 23, 2024

anujmodi2021 reviewed Dec 23, 2024

View reviewed changes

manika137 force-pushed the automateScript branch from 397b378 to 00f5e74 Compare December 27, 2024 10:03

resolving comments

548b4f8

manika137 force-pushed the automateScript branch from 00f5e74 to 548b4f8 Compare December 27, 2024 10:06

anujmodi2021 reviewed Dec 27, 2024

View reviewed changes

anujmodi2021 approved these changes Dec 27, 2024

View reviewed changes

eof change

c92c764

anmolanmol1234 reviewed Jan 15, 2025

View reviewed changes

manika137 added 3 commits January 19, 2025 21:16

address comments

1f76c2e

Merge remote-tracking branch 'origin/wasbDepCodeReview' into automate…

6bc44ad

…Script

Merge branch 'wasbDepCodeReview' into automateScript

af28a74

anmolanmol1234 reviewed Feb 24, 2025

View reviewed changes

anmolanmol1234 approved these changes Feb 25, 2025

View reviewed changes

anujmodi2021 reviewed Feb 25, 2025

View reviewed changes

[POC] CI automation scripts #133

Are you sure you want to change the base?

[POC] CI automation scripts #133

Uh oh!

Conversation

manika137 commented Jul 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Uh oh!

anmolanmol1234 commented Jul 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manika137 commented Jul 30, 2024

Uh oh!

anujmodi2021 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anujmodi2021 Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

manika137 commented Jul 26, 2024 •

edited

Loading

anmolanmol1234 commented Jul 26, 2024 •

edited

Loading

anujmodi2021 Feb 25, 2025 •

edited

Loading