Skip to content

HDDS-9031. Upgrade acceptance tests to Docker Compose v2 #6667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
adoroszlai merged 4 commits into from
May 16, 2024
Merged

HDDS-9031. Upgrade acceptance tests to Docker Compose v2 #6667

adoroszlai merged 4 commits into from
May 16, 2024

Conversation

@adoroszlai
Copy link
Contributor

@adoroszlai adoroszlai commented May 11, 2024 •
edited
Loading

What changes were proposed in this pull request?

Change acceptance tests to use Docker Compose v2, if available.

ozonescripts environment has some non-test scripts (start.sh, etc.), these require Docker Compose v2 after the change.

env-compose.robot is unused, can be removed.

https://issues.apache.org/jira/browse/HDDS-9031

How was this patch tested?

Acceptance tests with Docker Compose v1 uninstalled:
https://github.com/adoroszlai/ozone/actions/runs/9041151300

Regular CI:
https://github.com/adoroszlai/ozone/actions/runs/9041151733

@adoroszlai adoroszlai self-assigned this May 11, 2024
@adoroszlai adoroszlai requested review from errose28 and smengcl May 13, 2024 07:27
@kerneltime
Copy link
Contributor

kerneltime commented May 14, 2024 •
edited by smengcl
Loading

ozonescripts environment has some non-test scripts (start.sh, etc.), these require Docker Compose v2 after the change.

Can you elaborate on why they require docker compose v2 vs an optional use like the hadoop-ozone/dist/src/main/compose/testlib.sh script?

@smengcl
Copy link
Contributor

smengcl commented May 15, 2024 •
edited
Loading

ozonescripts environment has some non-test scripts (start.sh, etc.), these require Docker Compose v2 after the change.

Can you elaborate on why they require docker compose v2 vs an optional use like the hadoop-ozone/dist/src/main/compose/testlib.sh script?

Actually testlib.sh is already using v2 if it is available. It is a clever trick to override docker-compose to use v2 if that is available:

https://github.com/apache/ozone/pull/6667/files#diff-f26853740acf7f82a794bddb61e32ae145d12fdb0e08cd4a4c18a991e93742d0R34-R36

Back to your point I think it is possible to use the same trick as well.

smengcl
smengcl approved these changes May 15, 2024
View reviewed changes
Copy link
Contributor

@smengcl smengcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai !

@adoroszlai
Copy link
Contributor Author

adoroszlai commented May 15, 2024 •
edited
Loading

Can you elaborate on why they require docker compose v2

For these scripts I wanted to avoid using testlib.sh or duplicating the code.

I think it is possible to use the same trick as well.

Done by extracting the docker-compose function definition to its own script. Updated ozonescripts to use it. Also in ozone/run.sh, which I completely missed earlier.

v2-only run with updated patch:
https://github.com/adoroszlai/ozone/actions/runs/9090438317

@adoroszlai adoroszlai merged commit d80c45b into apache:master May 16, 2024
@adoroszlai adoroszlai deleted the HDDS-9031 branch May 16, 2024 07:27
@adoroszlai
Copy link
Contributor Author

adoroszlai commented May 16, 2024

Thanks @kerneltime, @smengcl for the review.

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request May 23, 2024
@adoroszlai @jojochuang
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
de22899
errose28 added a commit to errose28/ozone that referenced this pull request May 28, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10372-re…
97a76d7 
…concile-cli

* HDDS-10239-container-reconciliation: (296 commits)
  HDDS-10897. Refactor OzoneQuota (apache#6714)
  HDDS-10422. Fix some warnings about exposing internal representation in hdds-common (apache#6351)
  HDDS-10899. Refactor Lease callbacks (apache#6715)
  HDDS-10890. Increase default value for hdds.container.ratis.log.appender.queue.num-elements (apache#6711)
  HDDS-10832. Client should switch to streaming based on OpenKeySession replication (apache#6683)
  HDDS-10435. Support S3 object tags for existing requests (apache#6607)
  HDDS-10883. Improve logging in Recon for finalising DN logic. (apache#6704)
  HDDS-8752. Enable TestOzoneRpcClientAbstract#testOverWriteKeyWithAndWithOutVersioning (apache#6702)
  HDDS-10875. XceiverRatisServer#getRaftPeersInPipeline should be called before XceiverRatisServer#removeGroup (apache#6696)
  HDDS-10514. Recon - Provide DN decommissioning detailed status and info inline with current CLI command output. (apache#6376)
  HDDS-10878. Bump zstd-jni to 1.5.6-3 (apache#6701)
  HDDS-10877. Bump Dropwizard metrics to 3.2.6 (apache#6699)
  HDDS-10876. Bump jackson to 2.16.2 (apache#6697)
  HDDS-6116. Remove flaky tag from TestSCMInstallSnapshot (apache#6695)
  HDDS-2643. TestOzoneDelegationTokenSecretManager#testRenewTokenFailureRenewalTime fails intermittently.
  HDDS-10699. Refactor ContainerBalancerTask and TestContainerBalancerTask (apache#6537)
  HDDS-10861. Ozone cli supports default ozone.om.service.id (apache#6680)
  HDDS-10859. Improve error messages when decommission and maintenance fail-early (apache#6678)
  HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
  HDDS-10559. Add a warning or a check to run repair tool as System user (apache#6574)
  ...

Conflicts:
    hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
adoroszlai added a commit to adoroszlai/ozone that referenced this pull request Jul 12, 2024
@adoroszlai
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
6f7c25e 
(cherry picked from commit d80c45b)
adoroszlai added a commit to adoroszlai/ozone that referenced this pull request Jul 12, 2024
@adoroszlai
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
6d803c4 
(cherry picked from commit d80c45b)
devabhishekpal pushed a commit to devabhishekpal/ozone that referenced this pull request Jul 12, 2024
@adoroszlai @devabhishekpal
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
4fe44f4
@devabhishekpal devabhishekpal mentioned this pull request Jul 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smengcl smengcl smengcl approved these changes

@errose28 errose28 Awaiting requested review from errose28

Assignees

@adoroszlai adoroszlai

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adoroszlai @kerneltime @smengcl