HDDS-6633. Refactor upgrade acceptance tests to reduce code duplication. #4456

errose28 · 2023-03-23T06:18:54Z

What changes were proposed in this pull request?

Refactor of upgrade/downgrade acceptance tests for easier maintenance. Changes include:

Reduced code duplication among upgrades/downgrades between different versions.
- A new common set of callbacks can be used for any version being tested.
Improved directory structure.
Decoupling of docker compose cluster from versions being tested.
Ability to specify tests that are only run when upgrading to a certain version.
- Example: Check that EC is blocked in 1.3.0 pre-finalized.
Simple changes for release managers to add the new version to the test matrix and test the whole matrix before releasing.
Full HA cluster available for testing.
Replaced direct MLV checks in VERSION files with finalization status checks so the tests do not need to be updated for each layout feature.
Testing 1.30 -> master (1.4.0) upgrade and downgrade in regular CI runs.

To help with reviewing, please see the updated README.md. It may also be useful to pull the changes locally to look at the file and directory structure.

What is the link to the Apache JIRA

HDDS-6633

How was this patch tested?

The full matrix of upgrade/downgrade acceptance tests passed on my fork.

This reverts commit 0fffde2.

* master: (262 commits) HDDS-8153. Integrate ContainerBalancer with MoveManager (apache#4391) HDDS-8090. When getBlock from a datanode fails, retry other datanodes. (apache#4357) HDDS-8163 Use try-with-resources to ensure close rockdb connection in SstFilteringService (apache#4402) HDDS-8065. Provide GNU long options (apache#4394) HDDS-7930. [addendum] input stream does not refresh expired block token. HDDS-7930. input stream does not refresh expired block token. (apache#4378) HDDS-7740. [Snapshot] Implement SnapshotDeletingService (apache#4244) HDDS-8076. Use container cache in Key listing API. (apache#4346) HDDS-8091. [addendum] Generate list of config tags from ConfigTag enum - Hadoop 3.1 compatibility fix (apache#4374) HDDS-8144. TestDefaultCertificateClient#testTimeBeforeExpiryGracePeriod fails as we approach DST. (apache#4382) HDDS-8151. Support fine grained lifetime for root CA certificate (apache#4386) HDDS-8150. RpcClientTest and ConfigurationSourceTest not run due to naming convention (apache#4388) HDDS-8131. Add Configuration for OM Ratis Log Purge Tuning Parameters. (apache#4371) HDDS-8133. Create ozone sh key checksum command (apache#4375) HDDS-8142. Check if no entries in Block DB for a container on container delete (apache#4379) HDDS-8118. Fail container delete on non empty chunks dir (apache#4367) HDDS-8028. JNI for RocksDB SST Dump tool (apache#4315) HDDS-8129. ContainerStateMachine allows two different tasks with the same container id running in parallel. (apache#4370) HDDS-8119. Remove loosely related AutoCloseable from SendContainerOutputStream (apache#4368) close db connection (apache#4366) ...

* master: (43 commits) HDDS-8148. Improve log for Pipeline creation failure (apache#4385) HDDS-7853. Add support for RemoveSCM in SCMRatisServer. (apache#4358) HDDS-8042. Display certificate issuer in cert list command. (apache#4429) HDDS-8189. [Snapshot] renamedKeyTable should only track keys in buckets that has at least one active snapshot. (apache#4436) HDDS-8154. Perf: Reuse Mac instances in S3 token validation (apache#4433) HDDS-8245. Info log for keyDeletingService when nonzero number of keys are deleted. (apache#4451) HDDS-8233. ReplicationManager: Throttle delete container commands from over-replication handlers (apache#4447) HDDS-8220. [Ozone-Streaming] Trigger volume check on IOException in StreamDataChannelBase (apache#4428) HDDS-8173. Fix to remove enrties from RocksDB after container gets deleted. (apache#4445) HDDS-7975. Rebalance acceptance tests (apache#4437) HDDS-8152. Reduce S3 acceptance test setup time (apache#4393) HDDS-8172. ECUnderReplicationHandler should consider commands already sent when processing the container (apache#4435) HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService (apache#4407) HDDS-8168. Make deadlines inside MoveManager for move commands configurable (apache#4415) HDDS-7918. EC: ECBlockReconstructedStripeInputStream should check for spare replicas before failing an index (apache#4441) HDDS-8222. EndpointBase#getBucket should handle BUCKET_NOT_FOUND (apache#4431) HDDS-8068. Fix Exception: JMXJsonServlet, getting attribute RatisRoles of Hadoop:service=OzoneManager. (apache#4352) HDDS-8139. Datanodes should not drop block delete transactions based on transaction ID (apache#4384) HDDS-8216. EC: OzoneClientConfig is overwritten in ECKeyOutputStream (apache#4425) HDDS-8054. Fix NPE in metrics for failed volume (apache#4340) ...

fapifta

Hi @errose28 thank you for your great work on the upgrade tests, this is a well thought out change.
I noticed one thing: we might not need the load.sh, please see the proposal in one of the comments and let me know what do you think about it.

Other than that I have found small things in the documentation that sounded strange to my non-native ear, and some versions I think does not fit into the context but I am unsure, so I reserve the right to be completely wrong with those suggestions, but please take a look ;)

fapifta · 2023-03-24T01:33:29Z

hadoop-ozone/dist/src/main/compose/upgrade/README.md

+- These tests will not catch backwards incompatible changes against commits in between releases.
+    - Example:
+        1. After 1.2.0, a change *c1* is made that is backwards compatible with 1.2.0.
+        2. After *c1*, a new change *c2* is made that is also backwards compatible with 1.0.0 but backwards *incompatible* with *c1*.


Suggested change

2. After *c1*, a new change *c2* is made that is also backwards compatible with 1.0.0 but backwards *incompatible* with *c1*.

2. After *c1*, a new change *c2* is made that is also backwards compatible with 1.2.0 but backwards *incompatible* with *c1*.

hadoop-ozone/dist/src/main/compose/upgrade/README.md

fapifta · 2023-03-24T01:35:41Z

hadoop-ozone/dist/src/main/compose/upgrade/README.md

-            - This test suite will not raise an error for *c2*, because it only tests against the last release
-            (1.0.0), and not the last commit (*c1*).
+1. Before the release, test the whole matrix of upgrades from the previous version to the version you are releasing.
+    - This is important manual verification that the release does not break backwards compatibility, and the results can be included in the release vote mailing thread.


Suggested change

- This is important manual verification that the release does not break backwards compatibility, and the results can be included in the release vote mailing thread.

- This is an important manual verification that the release does not break backwards compatibility, and the results can be included in the release vote mailing thread.

hadoop-ozone/dist/src/main/compose/upgrade/README.md

hadoop-ozone/dist/src/main/compose/upgrade/compose/ha/load.sh

adoroszlai

Thanks @errose28 for the big batch of improvements.

kerneltime · 2023-03-27T16:25:49Z

cc @xBis7 @SaketaChalamchala @DaveTeng0 @tanvipenumudy can you take a look?

fapifta · 2023-03-28T22:37:04Z

Thank you @adoroszlai to check on this one, I accept your comments, I have created HDDS-8305 to track the additional suggestion on load.sh.

I am ok committing this one, and let the change as it is as Ethan will not be able to look into this further earlier than next week, and I think it is better to unblock the DataNodeDetails/Port/Upgrade related changes that are dependent on this work.

errose28 · 2023-04-04T17:49:21Z

Thanks for reviewing and merging this large PR @fapifta and @adoroszlai. I will look at the follow-up jira to remove load.sh, and filed HDDS-8375 (#4535) to fix the typos in the readme.

errose28 added 30 commits January 24, 2023 15:30

Add common set of callbacks to shell scripts

b56ee79

Remove MLV VERSION file checks

317a9f7

Add finalization status check

cf9dd45

Base callbacks on starting version only, and better support test matrix

e3ab1f9

Fix bugs discovered when running

b4964db

Make CI only run upgrade tests

2b517ec

Bash quoting fix in for robot args in testlib

0fffde2

Revert "Bash quoting fix in for robot args in testlib"

446d9b7

This reverts commit 0fffde2.

Fixes around finalization status robot checks

67338db

Add scm ha cluster

6abf01f

Rename callbacks and fix finalization checks

ef982af

Add data dur creation for full HA cluster

ebafcca

Update upgrade acc test README with new instructions

e5795f0

Fix scripting issues when loading compose files

67cdd19

Move and rename callbacks

db1e1f7

Improve warning about finding callbacks

e65b2ec

Handle finalization/prepare checks based on version

8e24f84

Fix parameter order

52bcb3e

Handle current version in runner image

278de5b

Update README

b829b56

Update unused manual upgrade callbacks

33d6979

Test om-ha

0896abe

Reverse test order

b7852ec

Fix typo using wrong cluster in om-ha

2e60fb9

Propogate upgrade version env vars into containers

139d500

Use scm to run commands whether using HA or not

9606427

Fix typo in robot

bf124a7

Fix grep

fddd42f

Restore original test order and clusters used

284d37c

errose28 added 7 commits March 21, 2023 16:04

Lots of bug fixes

a74bfb1

Remove failing callback check

357d1a3

Fix disk state being deleted on restart

19ac3fd

Update directory structure and README

b5c06ff

Revert Github Actions changes for testing

c296d21

Only test upgrade/downgrade with latest version

fd2164c

errose28 requested review from adoroszlai and fapifta March 23, 2023 06:18

errose28 mentioned this pull request Mar 23, 2023

HDDS-7817. Add HTTP and HTTPS ports to DatanodeDetails #4369

Merged

fapifta reviewed Mar 24, 2023

View reviewed changes

adoroszlai approved these changes Mar 24, 2023

View reviewed changes

adoroszlai requested a review from fapifta March 28, 2023 08:16

fapifta approved these changes Mar 28, 2023

View reviewed changes

fapifta merged commit 650e306 into apache:master Mar 28, 2023

errose28 mentioned this pull request Apr 4, 2023

HDDS-8375. Fix typos in upgrade acceptance test README. #4535

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-6633. Refactor upgrade acceptance tests to reduce code duplication. #4456

HDDS-6633. Refactor upgrade acceptance tests to reduce code duplication. #4456

Uh oh!

errose28 commented Mar 23, 2023 •

edited

Loading

Uh oh!

fapifta left a comment

Uh oh!

fapifta Mar 24, 2023

Uh oh!

Uh oh!

fapifta Mar 24, 2023

Uh oh!

Uh oh!

Uh oh!

adoroszlai left a comment

Uh oh!

kerneltime commented Mar 27, 2023

Uh oh!

fapifta commented Mar 28, 2023

Uh oh!

errose28 commented Apr 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	2. After c1, a new change c2 is made that is also backwards compatible with 1.0.0 but backwards incompatible with c1.
	2. After c1, a new change c2 is made that is also backwards compatible with 1.2.0 but backwards incompatible with c1.

	- This is important manual verification that the release does not break backwards compatibility, and the results can be included in the release vote mailing thread.
	- This is an important manual verification that the release does not break backwards compatibility, and the results can be included in the release vote mailing thread.

HDDS-6633. Refactor upgrade acceptance tests to reduce code duplication. #4456

HDDS-6633. Refactor upgrade acceptance tests to reduce code duplication. #4456

Uh oh!

Conversation

errose28 commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

fapifta left a comment

Choose a reason for hiding this comment

Uh oh!

fapifta Mar 24, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fapifta Mar 24, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

kerneltime commented Mar 27, 2023

Uh oh!

fapifta commented Mar 28, 2023

Uh oh!

errose28 commented Apr 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

errose28 commented Mar 23, 2023 •

edited

Loading