Skip to content

HDDS-8152. Reduce S3 acceptance test setup time #4393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 1 commit into from
Mar 22, 2023
Merged

HDDS-8152. Reduce S3 acceptance test setup time #4393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions hadoop-ozone/dist/src/main/smoketest/s3/bucketlist.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ Get bucket info with Ozone Shell to check the owner field
# Also see "Setup dummy credentials for S3" in commonawslib.robot

List buckets with empty access id
Execute aws configure set aws_access_key_id ''
[setup] Save AWS access key
Execute aws configure set aws_access_key_id ''
${result} = Execute AWSS3APICli and checkrc list-buckets 255
Should contain ${result} The authorization header you provided is invalid
Should contain ${result} The authorization header you provided is invalid
[teardown] Restore AWS access key
27 changes: 20 additions & 7 deletions hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ ${OZONE_S3_HEADER_VERSION} v4
${OZONE_S3_SET_CREDENTIALS} true
${BUCKET} generated
${KEY_NAME} key1
${OZONE_S3_TESTS_SET_UP} ${FALSE}
${OZONE_AWS_ACCESS_KEY_ID} ${EMPTY}

*** Keywords ***
Execute AWSS3APICli
Expand Down Expand Up @@ -88,6 +90,13 @@ Setup dummy credentials for S3
Execute aws configure set aws_secret_access_key dlfknslnfslf
Execute aws configure set region us-west-1

Save AWS access key
${OZONE_AWS_ACCESS_KEY_ID} = Execute aws configure get aws_access_key_id
Set Test Variable ${OZONE_AWS_ACCESS_KEY_ID}

Restore AWS access key
Execute aws configure set aws_access_key_id ${OZONE_AWS_ACCESS_KEY_ID}

Generate Ozone String
${randStr} = Generate Random String 10 [NUMBERS]
[Return] ozone-test-${randStr}
Expand All @@ -105,15 +114,15 @@ Create bucket with name
Should contain ${result} ${bucket}

Setup s3 tests
Return From Keyword if ${OZONE_S3_TESTS_SET_UP}
Run Keyword Generate random prefix
Run Keyword Install aws cli
Run Keyword if '${OZONE_S3_SET_CREDENTIALS}' == 'true' Setup v4 headers
${BUCKET} = Run Keyword if '${BUCKET}' == 'generated' Create bucket
... ELSE Set Variable ${BUCKET}
Set Suite Variable ${BUCKET}
Run Keyword if '${BUCKET}' == 'link' Setup links for S3 tests
Run Keyword if '${BUCKET}' == 'encrypted' Create encrypted bucket
Run Keyword if '${BUCKET}' == 'erasure' Create EC bucket
Run Keyword if '${BUCKET}' == 'generated' Create generated bucket
Run Keyword if '${BUCKET}' == 'link' Setup links for S3 tests
Run Keyword if '${BUCKET}' == 'encrypted' Create encrypted bucket
Run Keyword if '${BUCKET}' == 'erasure' Create EC bucket
Set Global Variable ${OZONE_S3_TESTS_SET_UP} ${TRUE}

Setup links for S3 tests
${exists} = Bucket Exists o3://${OM_SERVICE_ID}/s3v/link
Expand All @@ -122,6 +131,10 @@ Setup links for S3 tests
Execute ozone sh bucket create o3://${OM_SERVICE_ID}/legacy/source-bucket
Create link link

Create generated bucket
${BUCKET} = Create bucket
Set Global Variable ${BUCKET}

Create encrypted bucket
Return From Keyword if '${SECURITY_ENABLED}' == 'false'
${exists} = Bucket Exists o3://${OM_SERVICE_ID}/s3v/encrypted
Expand All @@ -140,7 +153,7 @@ Create EC bucket

Generate random prefix
${random} = Generate Ozone String
Set Suite Variable ${PREFIX} ${random}
Set Global Variable ${PREFIX} ${random}

Perform Multipart Upload
[arguments] ${bucket} ${key} @{files}
Expand Down