Skip to content

HDDS-4194. Create a script to check AWS S3 compatibility #1383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 4 commits into from
Sep 24, 2020
Merged

HDDS-4194. Create a script to check AWS S3 compatibility #1383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
240f7f0
HDDS-4194. Create a script to check AWS S3 compatiblity
elek Sep 2, 2020
5a6d263
remove generated test results
elek Sep 3, 2020
e8e86a2
rename test method to avoid conflict with the built-in
elek Sep 23, 2020
a8a83be
Merge remote-tracking branch 'origin/master' into HDDS-4194
elek Sep 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 65 additions & 65 deletions hadoop-ozone/dist/src/main/smoketest/s3/MultipartUpload.robot
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions hadoop-ozone/dist/src/main/smoketest/s3/commonawslib.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ Create bucket with name
Should contain ${result} ${ENDPOINT_URL}/${bucket}

Setup s3 tests
Run Keyword Generate random prefix
Run Keyword Install aws cli
Run Keyword if '${OZONE_S3_SET_CREDENTIALS}' == 'true' Setup v4 headers
${BUCKET} = Run Keyword if '${BUCKET}' == 'generated' Create bucket
Expand All @@ -109,3 +110,7 @@ Create link
[arguments] ${bucket}
Execute ozone sh bucket link o3://${OM_SERVICE_ID}/legacy/source-bucket o3://${OM_SERVICE_ID}/s3v/${bucket}
[return] ${bucket}

Generate random prefix
${random} = Generate Random String 5 [NUMBERS]
Set Suite Variable ${PREFIX} ${random}
23 changes: 10 additions & 13 deletions hadoop-ozone/dist/src/main/smoketest/s3/objectcopy.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,41 +27,38 @@ ${ENDPOINT_URL} http://s3g:9878
${BUCKET} generated
${DESTBUCKET} generated1


*** Keywords ***
Create Dest Bucket

${postfix} = Generate Random String 5 [NUMBERS]
Set Suite Variable ${DESTBUCKET} destbucket-${postfix}
Execute AWSS3APICli create-bucket --bucket ${DESTBUCKET}


*** Test Cases ***
Copy Object Happy Scenario
Run Keyword if '${DESTBUCKET}' == 'generated1' Create Dest Bucket
Execute date > /tmp/copyfile
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key copyobject/f1 --body /tmp/copyfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix copyobject/
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/copyobject/f1 --body /tmp/copyfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/copyobject/
Should contain ${result} f1

${result} = Execute AWSS3ApiCli copy-object --bucket ${DESTBUCKET} --key copyobject/f1 --copy-source ${BUCKET}/copyobject/f1
${result} = Execute AWSS3ApiCli list-objects --bucket ${DESTBUCKET} --prefix copyobject/
${result} = Execute AWSS3ApiCli copy-object --bucket ${DESTBUCKET} --key ${PREFIX}/copyobject/f1 --copy-source ${BUCKET}/${PREFIX}/copyobject/f1
${result} = Execute AWSS3ApiCli list-objects --bucket ${DESTBUCKET} --prefix ${PREFIX}/copyobject/
Should contain ${result} f1
#copying again will not throw error
${result} = Execute AWSS3ApiCli copy-object --bucket ${DESTBUCKET} --key copyobject/f1 --copy-source ${BUCKET}/copyobject/f1
${result} = Execute AWSS3ApiCli list-objects --bucket ${DESTBUCKET} --prefix copyobject/
${result} = Execute AWSS3ApiCli copy-object --bucket ${DESTBUCKET} --key ${PREFIX}/copyobject/f1 --copy-source ${BUCKET}/${PREFIX}/copyobject/f1
${result} = Execute AWSS3ApiCli list-objects --bucket ${DESTBUCKET} --prefix ${PREFIX}/copyobject/
Should contain ${result} f1

Copy Object Where Bucket is not available
${result} = Execute AWSS3APICli and checkrc copy-object --bucket dfdfdfdfdfnonexistent --key copyobject/f1 --copy-source ${BUCKET}/copyobject/f1 255
${result} = Execute AWSS3APICli and checkrc copy-object --bucket dfdfdfdfdfnonexistent --key ${PREFIX}/copyobject/f1 --copy-source ${BUCKET}/${PREFIX}/copyobject/f1 255
Should contain ${result} NoSuchBucket
${result} = Execute AWSS3APICli and checkrc copy-object --bucket ${DESTBUCKET} --key copyobject/f1 --copy-source dfdfdfdfdfnonexistent/copyobject/f1 255
${result} = Execute AWSS3APICli and checkrc copy-object --bucket ${DESTBUCKET} --key ${PREFIX}/copyobject/f1 --copy-source dfdfdfdfdfnonexistent/${PREFIX}/copyobject/f1 255
Should contain ${result} NoSuchBucket

Copy Object Where both source and dest are same with change to storageclass
${result} = Execute AWSS3APICli copy-object --storage-class REDUCED_REDUNDANCY --bucket ${DESTBUCKET} --key copyobject/f1 --copy-source ${DESTBUCKET}/copyobject/f1
${result} = Execute AWSS3APICli copy-object --storage-class REDUCED_REDUNDANCY --bucket ${DESTBUCKET} --key ${PREFIX}/copyobject/f1 --copy-source ${DESTBUCKET}/${PREFIX}/copyobject/f1
Should contain ${result} ETag

Copy Object Where Key not available
${result} = Execute AWSS3APICli and checkrc copy-object --bucket ${DESTBUCKET} --key copyobject/f1 --copy-source ${BUCKET}/nonnonexistentkey 255
${result} = Execute AWSS3APICli and checkrc copy-object --bucket ${DESTBUCKET} --key ${PREFIX}/copyobject/f1 --copy-source ${BUCKET}/nonnonexistentkey 255
Should contain ${result} NoSuchKey
28 changes: 14 additions & 14 deletions hadoop-ozone/dist/src/main/smoketest/s3/objectdelete.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,11 @@ ${BUCKET} generated
*** Test Cases ***
Delete file with s3api
Execute date > /tmp/testfile
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key deletetestapi/f1 --body /tmp/testfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix deletetestapi/
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/deletetestapi/f1 --body /tmp/testfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/deletetestapi/
Should contain ${result} f1
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key deletetestapi/f1
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix deletetestapi/
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key ${PREFIX}/deletetestapi/f1
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/deletetestapi/
Should not contain ${result} f1
#In case of HTTP 500, the error code is printed out to the console.
Should not contain ${result} 500
Expand All @@ -47,24 +47,24 @@ Delete file with s3api, file doesn't exist

Delete dir with s3api
Execute date > /tmp/testfile
${result} = Execute AWSS3Cli cp /tmp/testfile s3://${BUCKET}/deletetestapidir/f1
${result} = Execute AWSS3Cli ls s3://${BUCKET}/deletetestapidir/
${result} = Execute AWSS3Cli cp /tmp/testfile s3://${BUCKET}/${PREFIX}/deletetestapidir/f1
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/deletetestapidir/
Should contain ${result} f1
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key deletetestapidir/
${result} = Execute AWSS3Cli ls s3://${BUCKET}/deletetestapidir/
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key ${PREFIX}/deletetestapidir/
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/deletetestapidir/
Should contain ${result} f1
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key deletetestapidir/f1
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key ${PREFIX}/deletetestapidir/f1


Delete file with s3api, file doesn't exist, prefix of a real file
Execute date > /tmp/testfile
${result} = Execute AWSS3Cli cp /tmp/testfile s3://${BUCKET}/deletetestapiprefix/filefile
${result} = Execute AWSS3Cli ls s3://${BUCKET}/deletetestapiprefix/
${result} = Execute AWSS3Cli cp /tmp/testfile s3://${BUCKET}/${PREFIX}/deletetestapiprefix/filefile
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/deletetestapiprefix/
Should contain ${result} filefile
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key deletetestapiprefix/file
${result} = Execute AWSS3Cli ls s3://${BUCKET}/deletetestapiprefix/
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key ${PREFIX}/deletetestapiprefix/file
${result} = Execute AWSS3Cli ls s3://${BUCKET}/${PREFIX}/deletetestapiprefix/
Should contain ${result} filefile
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key deletetestapiprefix/filefile
${result} = Execute AWSS3APICli delete-object --bucket ${BUCKET} --key ${PREFIX}/deletetestapiprefix/filefile



Expand Down
24 changes: 12 additions & 12 deletions hadoop-ozone/dist/src/main/smoketest/s3/objectmultidelete.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,20 +30,20 @@ ${BUCKET} generated

Delete file with multi delete
Execute date > /tmp/testfile
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key multidelete/f1 --body /tmp/testfile
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key multidelete/f2 --body /tmp/testfile
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key multidelete/f3 --body /tmp/testfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix multidelete/
Should contain ${result} multidelete/f1
Should contain ${result} multidelete/f2
Should contain ${result} multidelete/f3
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/multidelete/f1 --body /tmp/testfile
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/multidelete/f2 --body /tmp/testfile
${result} = Execute AWSS3ApiCli put-object --bucket ${BUCKET} --key ${PREFIX}/multidelete/f3 --body /tmp/testfile
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/multidelete/
Should contain ${result} ${PREFIX}/multidelete/f1
Should contain ${result} ${PREFIX}/multidelete/f2
Should contain ${result} ${PREFIX}/multidelete/f3
Should contain ${result} STANDARD
Should not contain ${result} REDUCED_REDUNDANCY
${result} = Execute AWSS3APICli delete-objects --bucket ${BUCKET} --delete 'Objects=[{Key=multidelete/f1},{Key=multidelete/f2},{Key=multidelete/f4}]'
${result} = Execute AWSS3APICli delete-objects --bucket ${BUCKET} --delete 'Objects=[{Key=${PREFIX}/multidelete/f1},{Key=${PREFIX}/multidelete/f2},{Key=${PREFIX}/multidelete/f4}]'
Should not contain ${result} Error
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix multidelete/
Should not contain ${result} multidelete/f1
Should not contain ${result} multidelete/f2
Should contain ${result} multidelete/f3
${result} = Execute AWSS3ApiCli list-objects --bucket ${BUCKET} --prefix ${PREFIX}/multidelete/
Should not contain ${result} ${PREFIX}/multidelete/f1
Should not contain ${result} ${PREFIX}/multidelete/f2
Should contain ${result} ${PREFIX}/multidelete/f3
Should contain ${result} STANDARD
Should not contain ${result} REDUCED_REDUNDANCY
Loading