HDDS-6169. Selective checks: skip junit tests on ozone-runner image update #2974

adoroszlai · 2022-01-10T19:25:18Z

What changes were proposed in this pull request?

ozone-runner image version is defined in hadoop-ozone/dist/pom.xml. Changing it triggers a complete CI run, although it does not affect unit or integration tests in any way. Actually, hadoop-ozone/dist does not have any Java code, as it assembles other modules after build. So we can skip all Java-specific checks.

https://issues.apache.org/jira/browse/HDDS-6169

How was this patch tested?

Added test case in selective_ci_checks.bats.

$ bats dev-support/ci/selective_ci_checks.bats
...
 ✓ runner image update
...
17 tests, 0 failures

The script includes instructions on how to run:

ozone/dev-support/ci/selective_ci_checks.bats

Lines 17 to 31 in a31b79a

    
           # This script confirms that selective_ci_checks.sh works correctly 
        
           # against some known commits. 
        
           # 
        
           # Prerequisites: 
        
           # 
        
           # 1. Install bats-core, see: 
        
           #    https://bats-core.readthedocs.io/en/stable/installation.html 
        
           # 
        
           # 2. Clone libraries into dev-support/ci: 
        
           #    cd dev-support/ci 
        
           #    git clone https://github.com/bats-core/bats-assert 
        
           #    git clone https://github.com/bats-core/bats-support 
        
           # 
        
           # Usage: 
        
           #    bats dev-support/ci/selective_ci_checks.bats

…pdate

adoroszlai · 2022-01-24T17:15:30Z

@GeorgeJahad could you please also review if you have some time?

GeorgeJahad · 2022-01-24T18:19:12Z

@adoroszlai will do, hopefully in the next day or two

GeorgeJahad · 2022-01-25T17:51:28Z

dev-support/ci/selective_ci_checks.bats

+@test "runner image update" {
+  run dev-support/ci/selective_ci_checks.sh b95eeba82a
+
+  assert_output -p 'basic-checks=["rat"]'


Is it correct to say that without these changes the basic-checks array would have looked like this?:

basic-checks=["rat","checkstyle","findbugs","unit"]

Correct. The removed ones are all java-specific.

GeorgeJahad · 2022-01-25T17:57:40Z

+1

The changes in this PR look good to me.

One other thing I noted is that the instructions for running the test didn't work for me. To get them to work I had to make this change:

-load bats-support/load.bash
-load bats-assert/load.bash
+load bats-support/load
+load bats-assert/load

otherwise i got this error:

~/incoming/build-master/dev-support/ci ->     bats dev-support/ci/selective_ci_checks.bats 
bats: /home/gbj/incoming/build-master/dev-support/ci/bats-support/load.bash.bash does not exist

GeorgeJahad · 2022-01-25T18:08:03Z

This comment confused me for a while, because it emphasizes the ozone-runner image:

ozone-runner image version is defined in hadoop-ozone/dist/pom.xml. Changing it triggers a complete CI run, although it does not affect unit or integration tests in any way. Actually, hadoop-ozone/dist does not have any Java code, as it assembles other modules after build. So we can skip all Java-specific checks.

I would have left the ozone-runner out of it. To me it is clearer to just say:

hadoop-ozone/dist does not have any Java code, as it assembles other modules after build. So we can skip all Java-specific checks, for any changes in that directory.

adoroszlai · 2022-01-25T19:19:20Z

Thanks @GeorgeJahad for the review.

~/incoming/build-master/dev-support/ci ->     bats dev-support/ci/selective_ci_checks.bats 
bats: /home/gbj/incoming/build-master/dev-support/ci/bats-support/load.bash.bash does not exist

I think that behavior is specific to old versions of Bats (pre-1.2.1).

This comment confused me for a while, because it emphasizes the ozone-runner image:

ozone-runner image version is defined in hadoop-ozone/dist/pom.xml. Changing it triggers a complete CI run, although it does not affect unit or integration tests in any way. Actually, hadoop-ozone/dist does not have any Java code, as it assembles other modules after build. So we can skip all Java-specific checks.

I would have left the ozone-runner out of it.

I made the comment that way because it was a recent runner image version number bump that had started me on this change. Initially I wanted to extract the version number to a properties file. Only later did I realize that we can skip java checks for the entire submodule. Sorry for the confusion.

GeorgeJahad · 2022-01-25T19:58:15Z

I think that behavior is specific to old versions of Bats (pre-1.2.1).

You are right. I had an old version in my path.

smengcl

Followed the prerequisite guide. Passed locally on my Linux box (bash-bats 1.5.0-1).

Thanks @adoroszlai . I'm +1

adoroszlai · 2022-02-08T11:57:01Z

Thanks @GeorgeJahad and @smengcl for the review.

@REPLicated

…ansport support (#3074) * HDDS-6149. Remove unused keytabs (#2960) * HDDS-6094. Some unit tests are skipped due to using JUnit4 runner (#2909) * HDDS-6075. OzoneConfiguration constructor overrides input configuration keys. (#2921) * HDDS-4177. SCM Container DB bootstrap on Recon startup (#2942) * HDDS-6086. Compute MD5MD5CRC file checksum using chunk checksums from DataNodes (#2919) * HDDS-6148. Validate ContainerBalancerConfiguration before start ContainerBalancer (#2957) * HDDS-6161. SCM StateMachine failing to reinitialize doesn't terminate the process. (#2971) * HDDS-6134. Move replication-specific config to ReplicationServer (#2943) * HDDS-4010. S3G startup fails when multiple service ids are configured. (#2976) * HDDS-6170. Add metrics to replication manager to track container health states (#2975) * HDDS-3231. Cleanup KeyManagerImpl (#2961) * HDDS-5927. Improve defaults in ContainerBalancerConfiguration (#2892) * HDDS-6157. More consistent synchronization in InputStreams (#2965) * HDDS-4743. [FSO] Add FSO variant of ITestOzoneContractDistcp. (#2980) * HDDS-6114. Intermittent error due to Failed to init RocksDB (#2947) * HDDS-6175. Use s3Auth during proxy during decrypt in RpcClient. (#2981) * HDDS-6175. Use s3Auth during proxy during decrypt in RpcClient. * HDDS-5740. Enable ratis by default for SCM. (#2637) * HDDS-6183. Intermittent failure in TestKeyDeletingService.checkIfDeleteServiceWithFailingSCM. (#2991) * HDDS-4190. Intermittent failure in TestOMVolumeSetOwnerRequest and TestOMVolumeSetQuotaRequest. (#2982) * HDDS-6120. Compute block checksum using chunk checksums (#2930) * HDDS-6147. Add ability in OM to get limited delta updates (#2956) * HDDS-6195. Remove unused jmh-core dependency. (#2997) * HDDS-6167. Update ozone-runner version to 20211202-1 (#2969) * HDDS-6171. Create an API to change Bucket Owner (#2988) * HDDS-6163. Fix PATH in docker image (#2967) * HDDS-6202. Avoid using jmh-generator-annprocess since it is GPL2.0. (#2998) * HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. (#2972) * HDDS-6109. Preserve the underlying exception raised in client lib. (#2989) * HDDS-3408. Rename ChunkLayOutVersion to ContainerLayoutVersion. (#2983) * HDDS-6203. CleanUp incomplete gz files during Container move (#3000) * HDDS-6216. Move OMOpenKeysDeleteRequest to package om.request.key (#3011) * HDDS-6191. Intermittent failure in TestDeleteWithSlowFollower (#3015) * HDDS-6128. CLI tool that downloads all the block replicas and creates a manifest file (#2987) * HDDS-6177. Extend container info command to include replica details (#2995) * HDDS-6211. [Docs] Image styling on deployed site does not replicate local builds. (#3007) * HDDS-6219. Switch to RATIS ReplicationType from STAND_ALONE in our tests. (#3014) * HDDS-6192. feature/Observability.md translated to Chinese (#2994) * HDDS-6205. Add CLI command to display the latest Replication Manager report (#3013) * HDDS-6227. Test helpers should observe naming conditions (#3020) * HDDS-6239. ozonesecure-mr failing with No URLs in mirrorlist (#3029) * HDDS-6201. Fix NPE for DataScanner with scanned container deleted by others. (#3005) * HDDS-5529. For any IOexception from @REPLicated method we should throw it (#2788) * HDDS-6181. Change SCMHAInvocationHandler#invokeRatis() logging to TRACE (#2992) * HDDS-6206. Application errors must not flood system log (#3001) * HDDS-6245. Add BucketLayout logging to Audit Logs (#3040) * HDDS-6238 Reduce memory requirements for list keys. (#3032) * HDDS-2919. Intermittent failure in TestRDBStore (#3028) * HDDS-6253. Unnecessary duplicate smoketest after defaulting to FSO (#3036) * HDDS-6204. Cleanup handling malformed authorization header (#2999) * HDDS-6169. Selective checks: skip junit tests on ozone-runner image update (#2974) * HDDS-6270. Use a dedicated file instead of /etc/passwd for xcompat acceptance test (#3050) * HDDS-6273. Amend doc SecuringTDE.md (#3047) * HDDS-6140. Selective checks: skip unit check for integration-test changes (#2948) * HDDS-6215. Recon get limited delta updates from OM (#3009) * HDDS-6125. Recon get limited delta updates from OM * HDDS-6215. Fix unit test * trigger new CI check * HDDS-6215. Fix typo * trigger new CI check Co-authored-by: Symious <[email protected]> * HDDS-6226. Run tests for selective CI checks in CI (#3019) * HDDS-6247. Avoid logging stack trace for user input problems (#3039) * HDDS-6208. New checkstyle: WhitespaceAround (#3003) * HDDS-6289. Upgrade acceptance test log flooded with parse error (#3063) Co-authored-by: Siyao Meng <[email protected]> * Trigger Build * Fix integration test for added configuation field for selecting OmTransport for s3 gateway - TestOzoneConfigurationFields (added config key not in xml). Co-authored-by: Doroszlai, Attila <[email protected]> Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Aswin Shakil Balasubramanian <[email protected]> Co-authored-by: Wei-Chiu Chuang <[email protected]> Co-authored-by: Symious <[email protected]> Co-authored-by: Bharat Viswanadham <[email protected]> Co-authored-by: Stephen O'Donnell <[email protected]> Co-authored-by: GeorgeJahad <[email protected]> Co-authored-by: Siddhant Sangwan <[email protected]> Co-authored-by: Jyotinder Singh <[email protected]> Co-authored-by: Shashikant Banerjee <[email protected]> Co-authored-by: Tsz-Wo Nicholas Sze <[email protected]> Co-authored-by: Kiyoshi Mizumaru <[email protected]> Co-authored-by: Ritesh H Shukla <[email protected]> Co-authored-by: Nibiru <[email protected]> Co-authored-by: Kaijie Chen <[email protected]> Co-authored-by: Zita Dombi <[email protected]> Co-authored-by: Istvan Fajth <[email protected]> Co-authored-by: steinsgateted <[email protected]> Co-authored-by: Gui Hecheng <[email protected]> Co-authored-by: Jackson Yao <[email protected]> Co-authored-by: Keyi Song <[email protected]> Co-authored-by: UENISHI Kota <[email protected]> Co-authored-by: Siyao Meng <[email protected]> Co-authored-by: Symious <[email protected]>

HDDS-6169. Selective checks: skip junit tests on ozone-runner image u…

3f57aab

…pdate

adoroszlai self-assigned this Jan 10, 2022

adoroszlai added build Pull request that modifies the build process enhancement New feature or request labels Jan 10, 2022

adoroszlai requested a review from fapifta January 11, 2022 14:06

GeorgeJahad reviewed Jan 25, 2022

View reviewed changes

swagle requested a review from smengcl January 27, 2022 20:46

smengcl approved these changes Feb 8, 2022

View reviewed changes

adoroszlai merged commit b32e135 into apache:master Feb 8, 2022

adoroszlai deleted the HDDS-6169 branch February 8, 2022 11:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-6169. Selective checks: skip junit tests on ozone-runner image update #2974

HDDS-6169. Selective checks: skip junit tests on ozone-runner image update #2974

Uh oh!

adoroszlai commented Jan 10, 2022

Uh oh!

adoroszlai commented Jan 24, 2022

Uh oh!

GeorgeJahad commented Jan 24, 2022

Uh oh!

GeorgeJahad Jan 25, 2022

Uh oh!

adoroszlai Jan 25, 2022

Uh oh!

GeorgeJahad commented Jan 25, 2022

Uh oh!

GeorgeJahad commented Jan 25, 2022

Uh oh!

adoroszlai commented Jan 25, 2022

Uh oh!

GeorgeJahad commented Jan 25, 2022

Uh oh!

smengcl left a comment

Uh oh!

adoroszlai commented Feb 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	# This script confirms that selective_ci_checks.sh works correctly
	# against some known commits.
	#
	# Prerequisites:
	#
	# 1. Install bats-core, see:
	# https://bats-core.readthedocs.io/en/stable/installation.html
	#
	# 2. Clone libraries into dev-support/ci:
	# cd dev-support/ci
	# git clone https://github.com/bats-core/bats-assert
	# git clone https://github.com/bats-core/bats-support
	#
	# Usage:
	# bats dev-support/ci/selective_ci_checks.bats

HDDS-6169. Selective checks: skip junit tests on ozone-runner image update #2974

HDDS-6169. Selective checks: skip junit tests on ozone-runner image update #2974

Uh oh!

Conversation

adoroszlai commented Jan 10, 2022

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

adoroszlai commented Jan 24, 2022

Uh oh!

GeorgeJahad commented Jan 24, 2022

Uh oh!

GeorgeJahad Jan 25, 2022

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jan 25, 2022

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad commented Jan 25, 2022

Uh oh!

GeorgeJahad commented Jan 25, 2022

Uh oh!

adoroszlai commented Jan 25, 2022

Uh oh!

GeorgeJahad commented Jan 25, 2022

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Feb 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants