Cleanup created accounts and buckets after tests #2152

williamlardier · 2024-09-20T14:11:29Z

Without cleanup, the GetRolesForWebIdentity API ends up being very slow, eventually causing Vault to be unresponsive and thus causing timeouts, and tests failures. This API is not supposed to work with more than 100 accounts (as per requirements), and here, we end up with >1000, meaning at least this number of internal (and sequential) DB calls per "GetRolesForWebIdentity" API call.

The logic detects that we created an account (during cross-account tests), and delete all resources, before deleting the account.

Issue: ZENKO-4898

bert-e · 2024-09-20T14:11:33Z

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2024-09-20T14:19:14Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

bert-e · 2024-09-23T07:28:56Z

Incorrect fix version

The Fix Version/s in issue ZENKO-4898 contains:

2.10.0
2.6.68
2.7.64
2.8.44
2.9.20

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

2.10.0
2.6.68
2.7.64
2.8.44
2.9.21

Please check the Fix Version/s of ZENKO-4898, or the target
branch of this pull request.

bert-e · 2024-09-23T08:47:52Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

williamlardier · 2024-09-24T08:54:49Z

/create_integration_branches

bert-e · 2024-09-24T08:57:12Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/ZENKO-4898 into
development/2.6
w/2.7/bugfix/ZENKO-4898 will be merged into development/2.7
w/2.8/bugfix/ZENKO-4898 will be merged into development/2.8
w/2.9/bugfix/ZENKO-4898 will be merged into development/2.9
w/2.10/bugfix/ZENKO-4898 will be merged into development/2.10

The following branches will NOT be impacted:

development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-09-24T08:57:14Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

williamlardier · 2024-09-24T16:10:05Z

.github/actions/archive-artifacts/action.yaml

+
+        MONGODB_COLLECTIONS=$(kubectl exec -n ${NAMESPACE} data-db-mongodb-sharded-mongos-0 -- mongo ${ZENKO_MONGODB_DATABASE} -u ${MONGODB_ROOT_USERNAME} -p ${MONGODB_ROOT_PASSWORD} --authenticationDatabase admin --quiet --eval "db.getCollectionNames().join(' ')")
+
+        for collection in ${MONGODB_COLLECTIONS}; do


do we want to keep only the special collections?
For now, I see ~400 collections kept (due to object lock or test failing), they all have less than 10 objects anyway

not sure what we want to do...

it takes ~1min20, which is significant (maybe not after 2h of test, but we need to reduce this 2h anyway...)

we should try to reduce the time. Probably a large time is related to kubectl kube setup & I/O transfers out of the container... May be much faster using mongo dump ; and possibly "copying" the file from the container after the tool has finished, or creating a new container with a host-volume to directly "save" the file to the host...

if we can't make it significantly faster, we may need some filtering indeed AND/OR do this only if the previous job has failed...

tests/ctst/common/hooks.ts

tests/ctst/world/Zenko.ts

francoisferrand · 2024-09-25T07:09:16Z

tests/ctst/steps/utils/utils.ts

    await uploadSetup(world, 'PutObject');
    world.addCommandParameter({ key: world.getSaved<string>('objectName') });
    world.addCommandParameter({ bucket: world.getSaved<string>('bucketName') });
    const result = await S3.putObject(world.getCommandParameters());
+    const versionId = extractPropertyFromResults<string>(result, 'VersionId');
+    world.saveCreatedObject(world.getSaved<string>('objectName'), versionId || '');
+
    world.addToSaved('versionId', extractPropertyFromResults(


saving versionId seems redundant with world.saveCreatedObject ; even if we keep it to avoid too wide-ranging changes to the tests, should it not be done inside saveCreatedObject ?

I removed the lines, and updated the function: it was storing always the last version id, but we may need to store all the known version IDs... We can later add a function in the world to delete the object if removed from the bucket, but I would like to do it in a dedicated ticket, where we centralize all the duplicated logic (so far I see that tests usually require some common functions to manage objects, that should be part of the World and not re-implemented in different places).

-> ZENKO-4904

tests/ctst/steps/utils/utils.ts

tests/ctst/steps/pra.ts

francoisferrand · 2024-09-25T07:31:00Z

.github/actions/archive-artifacts/action.yaml

+        MONGODB_COLLECTIONS=$(kubectl exec -n ${NAMESPACE} data-db-mongodb-sharded-mongos-0 -- mongo ${ZENKO_MONGODB_DATABASE} -u ${MONGODB_ROOT_USERNAME} -p ${MONGODB_ROOT_PASSWORD} --authenticationDatabase admin --quiet --eval "db.getCollectionNames().join(' ')")
+
+        for collection in ${MONGODB_COLLECTIONS}; do
+            kubectl exec -n ${NAMESPACE} data-db-mongodb-sharded-mongos-0 -- mongo ${ZENKO_MONGODB_DATABASE} -u ${MONGODB_ROOT_USERNAME} -p ${MONGODB_ROOT_PASSWORD} --authenticationDatabase admin --quiet --eval "db.getCollection((\"${collection}\")).find().toArray()" > /tmp/artifacts/data/${STAGE}/mongodb-${collection}.log


can't we use mongodump instead?
it allows dumping the whole DB or specific collection, may be faster. However, it exports a BSON: more compact, but maybe less practical to use (for analysis, may need to pass it through bsondump or restore it into another mongo)

otherwise, we may use mongoexport to dump a collection to JSON, one at a time...

Updated with mongodump & bsondump, let's see if my approach is faster...

New logic takes 10s to complete, I think we are good...

bert-e · 2024-09-25T09:17:08Z

Conflict

A conflict has been raised during the update of
integration branch w/2.7/bugfix/ZENKO-4898 with contents from bugfix/ZENKO-4898
and development/2.7.

Please resolve the conflict on the integration branch (w/2.7/bugfix/ZENKO-4898).

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout w/2.7/bugfix/ZENKO-4898
 $ git pull  # or "git reset --hard origin/w/2.7/bugfix/ZENKO-4898"
 $ git merge origin/development/2.7
 $ # <intense conflict resolution>
 $ git commit
 $ git merge origin/bugfix/ZENKO-4898
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.7/bugfix/ZENKO-4898

The following options are set: create_integration_branches

Issue: ZENKO-4898

bert-e · 2024-09-25T13:01:40Z

History mismatch

Merge commit #c2e65127c6f4162bc8e06aa73caeb351601707fc on the integration branch
w/2.7/bugfix/ZENKO-4898 is merging a branch which is neither the current
branch bugfix/ZENKO-4898 nor the development branch
development/2.7.

It is likely due to a rebase of the branch bugfix/ZENKO-4898 and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

williamlardier · 2024-09-25T13:05:18Z

/reset

bert-e · 2024-09-25T13:05:28Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

- Remove duplicated logic - Use a world-managed function to keep track of created objects - Clean the buckets at the end of any test, unless it failed, to ease debugging. Issue: ZENKO-4898

- Assume role tests create additional accounts that we must clean at the end of any (successful) scenario. - This ensures that the GetRolesForWebIdentity calls are not impacted by the high number of accounts. Issue: ZENKO-4898

- Quotas are not needed, plus, they cause errors when using the env variable in a kubectl command. Issue: ZENKO-4898

- Tests should cleanup resources, unless a scenario failed - Use of mongodump and bsondump for fast speed Issue: ZENKO-4898

bert-e · 2024-09-26T10:17:40Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/ZENKO-4898 into
development/2.6
w/2.7/bugfix/ZENKO-4898 will be merged into development/2.7
w/2.8/bugfix/ZENKO-4898 will be merged into development/2.8
w/2.9/bugfix/ZENKO-4898 will be merged into development/2.9
w/2.10/bugfix/ZENKO-4898 will be merged into development/2.10

The following branches will NOT be impacted:

development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-09-26T10:17:41Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

williamlardier force-pushed the bugfix/ZENKO-4898 branch from 6162a8c to 8f4cd25 Compare September 20, 2024 14:16

scality deleted a comment from bert-e Sep 20, 2024

williamlardier force-pushed the bugfix/ZENKO-4898 branch 5 times, most recently from 241c291 to eea904c Compare September 24, 2024 08:54

scality deleted a comment from bert-e Sep 24, 2024

williamlardier commented Sep 24, 2024

View reviewed changes

francoisferrand reviewed Sep 25, 2024

View reviewed changes

tests/ctst/common/hooks.ts Show resolved Hide resolved

francoisferrand reviewed Sep 25, 2024

View reviewed changes

tests/ctst/common/hooks.ts Outdated Show resolved Hide resolved

francoisferrand reviewed Sep 25, 2024

View reviewed changes

tests/ctst/world/Zenko.ts Show resolved Hide resolved

francoisferrand reviewed Sep 25, 2024

View reviewed changes

tests/ctst/steps/utils/utils.ts Outdated Show resolved Hide resolved

francoisferrand reviewed Sep 25, 2024

View reviewed changes

tests/ctst/steps/pra.ts Outdated Show resolved Hide resolved

francoisferrand reviewed Sep 25, 2024

View reviewed changes

williamlardier force-pushed the bugfix/ZENKO-4898 branch 5 times, most recently from febf8e3 to 7eb9161 Compare September 25, 2024 11:41

williamlardier force-pushed the bugfix/ZENKO-4898 branch from 7eb9161 to da7994a Compare September 25, 2024 12:50

Code formatting

b6314d8

Issue: ZENKO-4898

williamlardier force-pushed the bugfix/ZENKO-4898 branch from da7994a to f9d31df Compare September 25, 2024 13:01

williamlardier marked this pull request as ready for review September 25, 2024 13:05

williamlardier added 4 commits September 26, 2024 10:36

Unify bucket creation and cleanup

a464062

- Remove duplicated logic - Use a world-managed function to keep track of created objects - Clean the buckets at the end of any test, unless it failed, to ease debugging. Issue: ZENKO-4898

Cleanup created accounts

a781df4

- Assume role tests create additional accounts that we must clean at the end of any (successful) scenario. - This ensures that the GetRolesForWebIdentity calls are not impacted by the high number of accounts. Issue: ZENKO-4898

Simplify and unify pradb name

9bbca11

- Quotas are not needed, plus, they cause errors when using the env variable in a kubectl command. Issue: ZENKO-4898

Dump mongodb at the end of the tests

7ab8dce

- Tests should cleanup resources, unless a scenario failed - Use of mongodump and bsondump for fast speed Issue: ZENKO-4898

williamlardier force-pushed the bugfix/ZENKO-4898 branch from 77045d6 to 7ab8dce Compare September 26, 2024 08:38

williamlardier changed the title ~~Cleanup created accounts after tests~~ Cleanup created accounts and buckets after tests Sep 26, 2024

scality deleted a comment from bert-e Sep 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup created accounts and buckets after tests #2152

Cleanup created accounts and buckets after tests #2152

williamlardier commented Sep 20, 2024

bert-e commented Sep 20, 2024

bert-e commented Sep 20, 2024

bert-e commented Sep 23, 2024

bert-e commented Sep 23, 2024

williamlardier commented Sep 24, 2024

bert-e commented Sep 24, 2024

bert-e commented Sep 24, 2024

williamlardier Sep 24, 2024

francoisferrand Sep 25, 2024

francoisferrand Sep 25, 2024

williamlardier Sep 25, 2024 •

edited

Loading

williamlardier Sep 25, 2024

francoisferrand Sep 25, 2024

williamlardier Sep 25, 2024

williamlardier Sep 26, 2024

bert-e commented Sep 25, 2024

bert-e commented Sep 25, 2024

williamlardier commented Sep 25, 2024

bert-e commented Sep 25, 2024

bert-e commented Sep 26, 2024

bert-e commented Sep 26, 2024


		MONGODB_COLLECTIONS=$(kubectl exec -n ${NAMESPACE} data-db-mongodb-sharded-mongos-0 -- mongo ${ZENKO_MONGODB_DATABASE} -u ${MONGODB_ROOT_USERNAME} -p ${MONGODB_ROOT_PASSWORD} --authenticationDatabase admin --quiet --eval "db.getCollectionNames().join(' ')")

		for collection in ${MONGODB_COLLECTIONS}; do

Cleanup created accounts and buckets after tests #2152

Are you sure you want to change the base?

Cleanup created accounts and buckets after tests #2152

Conversation

williamlardier commented Sep 20, 2024

bert-e commented Sep 20, 2024

Hello williamlardier,

bert-e commented Sep 20, 2024

Request integration branches

bert-e commented Sep 23, 2024

Incorrect fix version

bert-e commented Sep 23, 2024

Request integration branches

williamlardier commented Sep 24, 2024

bert-e commented Sep 24, 2024

Integration data created

bert-e commented Sep 24, 2024

Waiting for approval

williamlardier Sep 24, 2024

Choose a reason for hiding this comment

francoisferrand Sep 25, 2024

Choose a reason for hiding this comment

francoisferrand Sep 25, 2024

Choose a reason for hiding this comment

williamlardier Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

williamlardier Sep 25, 2024

Choose a reason for hiding this comment

francoisferrand Sep 25, 2024

Choose a reason for hiding this comment

williamlardier Sep 25, 2024

Choose a reason for hiding this comment

williamlardier Sep 26, 2024

Choose a reason for hiding this comment

bert-e commented Sep 25, 2024

Conflict

bert-e commented Sep 25, 2024

History mismatch

williamlardier commented Sep 25, 2024

bert-e commented Sep 25, 2024

Reset complete

bert-e commented Sep 26, 2024

Integration data created

bert-e commented Sep 26, 2024

Waiting for approval

williamlardier Sep 25, 2024 •

edited

Loading