Skip to content

Conversation

@ghuangups
Copy link
Contributor

HDDS-4864. Add acceptance tests to certify Ozone with boto3 python client.

What changes were proposed in this pull request?

Add an acceptance tests to certify Ozone with boto3 python client.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4864

How was this patch tested?

  • Did a full build and tested it locally.
  • adoroszlai executed my patch locally in: ozone compose, ozonesecure and ozone-ha, all passed.

@adoroszlai
Copy link
Contributor

@ghuangups can you please add hadoop-ozone/dist/pom.xml, too? We need the new ozone-runner image for running boto3 tests.

diff --git hadoop-ozone/dist/pom.xml hadoop-ozone/dist/pom.xml
index c1da927cd..5454799d1 100644
--- hadoop-ozone/dist/pom.xml
+++ hadoop-ozone/dist/pom.xml
@@ -28,7 +28,7 @@
   <properties>
     <file.encoding>UTF-8</file.encoding>
     <downloadSources>true</downloadSources>
-    <docker.ozone-runner.version>20200625-1</docker.ozone-runner.version>
+    <docker.ozone-runner.version>20210226-1</docker.ozone-runner.version>
   </properties>

   <build>

Jun Huang added 2 commits March 1, 2021 14:44
This reverts commit e758a4c.
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ghuangups for updating the patch. Now boto3 test is executed in multiple environments:


*** Test Cases ***

Bobo3 Client Test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Bobo3 Client Test
Boto3 Client Test

@swagle
Copy link
Contributor

swagle commented Mar 2, 2021

Thanks @adoroszlai for review, merging this!

@swagle swagle merged commit b07162b into apache:master Mar 2, 2021
errose28 added a commit to errose28/ozone that referenced this pull request Mar 2, 2021
…ing-upgrade

* upstream/master: (29 commits)
  HDDS-4741. Modularize upgrade test (apache#1928)
  HDDS-4864. Add acceptance tests to certify Ozone with boto3 python client. (apache#1976)
  HDDS-4791. StateContext.getReports may return list with size larger t… (apache#1892)
  HDDS-4867. Ozone admin datanode list should report dead and stale nodes (apache#1966)
  HDDS-4858. Useless Maven cache cleanup (apache#1956)
  HDDS-4769. Simplify insert operation of ContainerAttribute (apache#1865)
  HDDS-4847. Fix typo in name of IdentityService (apache#1941)
  HDDS-4869. Bump jackson version number (apache#1963)
  HDDS-4871. Fix intellij runConfigurations for datanode (apache#1968)
  HDDS-4870. Bump jetty version (apache#1964)
  HDDS-4722. Creating RDBStore fails due to RDBMetrics instance race (apache#1820)
  HDDS-4138. Improve crc efficiency by using Java.util.zip.CRC when available (apache#1950)
  HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. (apache#1919)
  HDDS-4754. Make scm heartbeat rpc retry interval configurable (apache#1942)
  HDDS-4832. Show Datanode OperationalState in Recon (apache#1937)
  HDDS-4653. Support TDE for MPU Keys on Encrypted Buckets (apache#1766)
  HDDS-4853. libexec/entrypoint.sh might copy from wrong path (apache#1951)
  HDDS-4857. Format ReplicationType.java which indentation are confusion (apache#1952)
  HDDS-4850. Intermittent failure in ozonesecure due to unable to allocate block (apache#1948)
  HDDS-4808. Add Genesis benchmark for various CRC implementations (apache#1910)
  ...

Conflicts:
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
	hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java
	hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto
	hadoop-hdds/interface-client/src/main/proto/hdds.proto
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
	hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
errose28 added a commit to errose28/ozone that referenced this pull request Mar 11, 2021
…ing-upgrade-merge-candidate

* upstream/master: (29 commits)
  HDDS-4741. Modularize upgrade test (apache#1928)
  HDDS-4864. Add acceptance tests to certify Ozone with boto3 python client. (apache#1976)
  HDDS-4791. StateContext.getReports may return list with size larger t… (apache#1892)
  HDDS-4867. Ozone admin datanode list should report dead and stale nodes (apache#1966)
  HDDS-4858. Useless Maven cache cleanup (apache#1956)
  HDDS-4769. Simplify insert operation of ContainerAttribute (apache#1865)
  HDDS-4847. Fix typo in name of IdentityService (apache#1941)
  HDDS-4869. Bump jackson version number (apache#1963)
  HDDS-4871. Fix intellij runConfigurations for datanode (apache#1968)
  HDDS-4870. Bump jetty version (apache#1964)
  HDDS-4722. Creating RDBStore fails due to RDBMetrics instance race (apache#1820)
  HDDS-4138. Improve crc efficiency by using Java.util.zip.CRC when available (apache#1950)
  HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. (apache#1919)
  HDDS-4754. Make scm heartbeat rpc retry interval configurable (apache#1942)
  HDDS-4832. Show Datanode OperationalState in Recon (apache#1937)
  HDDS-4653. Support TDE for MPU Keys on Encrypted Buckets (apache#1766)
  HDDS-4853. libexec/entrypoint.sh might copy from wrong path (apache#1951)
  HDDS-4857. Format ReplicationType.java which indentation are confusion (apache#1952)
  HDDS-4850. Intermittent failure in ozonesecure due to unable to allocate block (apache#1948)
  HDDS-4808. Add Genesis benchmark for various CRC implementations (apache#1910)
  ...

Conflicts:
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
	hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java
	hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto
	hadoop-hdds/interface-client/src/main/proto/hdds.proto
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
	hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
	hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconNodeManager.java
@mukul1987
Copy link
Contributor

@ghuangups, I feel that we should not have aws_secret_access_key in a commit, even as an example.

errose28 added a commit to errose28/ozone that referenced this pull request Mar 16, 2021
* HDDS-3698-nonrolling-upgrade: (29 commits)
  HDDS-4741. Modularize upgrade test (apache#1928)
  HDDS-4864. Add acceptance tests to certify Ozone with boto3 python client. (apache#1976)
  HDDS-4791. StateContext.getReports may return list with size larger t… (apache#1892)
  HDDS-4867. Ozone admin datanode list should report dead and stale nodes (apache#1966)
  HDDS-4858. Useless Maven cache cleanup (apache#1956)
  HDDS-4769. Simplify insert operation of ContainerAttribute (apache#1865)
  HDDS-4847. Fix typo in name of IdentityService (apache#1941)
  HDDS-4869. Bump jackson version number (apache#1963)
  HDDS-4871. Fix intellij runConfigurations for datanode (apache#1968)
  HDDS-4870. Bump jetty version (apache#1964)
  HDDS-4722. Creating RDBStore fails due to RDBMetrics instance race (apache#1820)
  HDDS-4138. Improve crc efficiency by using Java.util.zip.CRC when available (apache#1950)
  HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. (apache#1919)
  HDDS-4754. Make scm heartbeat rpc retry interval configurable (apache#1942)
  HDDS-4832. Show Datanode OperationalState in Recon (apache#1937)
  HDDS-4653. Support TDE for MPU Keys on Encrypted Buckets (apache#1766)
  HDDS-4853. libexec/entrypoint.sh might copy from wrong path (apache#1951)
  HDDS-4857. Format ReplicationType.java which indentation are confusion (apache#1952)
  HDDS-4850. Intermittent failure in ozonesecure due to unable to allocate block (apache#1948)
  HDDS-4808. Add Genesis benchmark for various CRC implementations (apache#1910)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants