HBASE-24710 Incorrect checksum calculation in saveVersion.sh #2056

petersomogyi · 2020-07-13T12:35:56Z

No description provided.

ndimiduk

Thanks for the general cleanup.

hbase-common/src/saveVersion.sh

ndimiduk · 2020-07-15T16:28:38Z

hbase-common/src/saveVersion.sh

  if ! [ -x "$(command -v gpg)" ]; then
    srcChecksum="Unknown"
  else
-    srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | cut -d ' ' -f 1`


Previous use of cut here at the end is surprising. It took only the first block...

Based on history it was used for md5 and md5sum commands but it wasn't removed when SHA512 was introduced in saveVersion.sh.

root@8224f4c69e6f:/# echo "foo" | md5sum d3b07384d113edec49eaa6238ad5ff00 -

Does tr -d '[:space:]' work? It's simpler and shorter than tr '\n' ' ' | sed 's/[[:space:]]*//g'

I get a different output from your command, an extra %.

find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | tr -d '[:space:]' 3BF4CCC275112769C7AB0EDC302793FB81005BDC9681E1604BD965918363FC93311FC60B24000563DD4D3EFDA10685F549721A28428F23487813A7397FBC9BD0% ind hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | tr '\n' ' ' | sed 's/[[:space:]]*//g' 3BF4CCC275112769C7AB0EDC302793FB81005BDC9681E1604BD965918363FC93311FC60B24000563DD4D3EFDA10685F549721A28428F23487813A7397FBC9BD0

That's weird. I did not see in my macOS and Ubuntu Linux. But since this is not reliable, I guess we can stay with your script tr and sed - which works for me on both macOS and Ubuntu.

ndimiduk · 2020-07-15T16:32:01Z

hbase-common/src/saveVersion.sh

  fi
 else
-  srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs openssl dgst -sha512 | openssl dgst -sha512 | cut -d ' ' -f 1`
+  srcChecksum=$(find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort xargs openssl dgst -sha512 | openssl dgst -sha512 | sed 's/^.* //')


what is the purpose of the final sed? I see no difference.

$ echo "foo" | openssl dgst -sha512 | sed 's/^.* //' 0cf9180a764aba863a67b6d72f0918bc131c6772642cb2dce5a34f0a702f9470ddc2bf125c12198b1995c233c34b4afd346c54a2334c350a948a51b6e8b4e6b6 $ echo "foo" | openssl dgst -sha512 0cf9180a764aba863a67b6d72f0918bc131c6772642cb2dce5a34f0a702f9470ddc2bf125c12198b1995c233c34b4afd346c54a2334c350a948a51b6e8b4e6b6

Oh, this is the example from your comment on JIRA, a difference between the mac and linux versions of openssl dgst. Disregard.

Apache-HBase · 2020-07-15T17:16:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 30s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2056
Optional Tests
uname	Linux d8d04fa9a000 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2505c77`
Max. process+thread count	47 (vs. ulimit of 12500)
modules	C: hbase-common U: hbase-common
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-07-15T17:16:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
		_ Patch Compile Tests _
		_ Other Tests _
		1m 28s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2056
Optional Tests
uname	Linux a40603fd2e90 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2505c77`
Max. process+thread count	43 (vs. ulimit of 12500)
modules	C: hbase-common U: hbase-common
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-07-15T17:17:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+0 🆗	shelldocs	0m 0s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
		_ Patch Compile Tests _
+1 💚	shellcheck	0m 0s	The patch generated 0 new + 0 unchanged - 10 fixed = 0 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		1m 46s

Subsystem	Report/Notes
Docker	Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2056
Optional Tests	dupname asflicense shellcheck shelldocs
uname	Linux 2b002e7bec9d 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2505c77`
Max. process+thread count	42 (vs. ulimit of 12500)
modules	C: hbase-common U: hbase-common
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2056/2/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

liuml07 · 2020-07-17T07:48:37Z

LGTM

Signed-off-by: Nick Dimiduk <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

…2056) Signed-off-by: Nick Dimiduk <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

HBASE-24710 Incorrect checksum calculation in saveVersion.sh

686e041

petersomogyi requested a review from ndimiduk July 13, 2020 12:35

This comment has been minimized.

Sign in to view

petersomogyi requested a review from busbey July 14, 2020 09:30

ndimiduk requested changes Jul 15, 2020

View reviewed changes

Add missing pipe

73f809a

ndimiduk approved these changes Jul 20, 2020

View reviewed changes

petersomogyi merged commit ea96743 into apache:master Jul 21, 2020

petersomogyi added a commit that referenced this pull request Jul 21, 2020

HBASE-24710 Incorrect checksum calculation in saveVersion.sh (#2056)

e0c917b

Signed-off-by: Nick Dimiduk <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

petersomogyi added a commit that referenced this pull request Jul 21, 2020

HBASE-24710 Incorrect checksum calculation in saveVersion.sh (#2056)

0c4170b

Signed-off-by: Nick Dimiduk <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

clarax pushed a commit to clarax/hbase that referenced this pull request Nov 15, 2020

HBASE-24710 Incorrect checksum calculation in saveVersion.sh (apache#…

857f622

…2056) Signed-off-by: Nick Dimiduk <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

HBASE-24710 Incorrect checksum calculation in saveVersion.sh #2056

HBASE-24710 Incorrect checksum calculation in saveVersion.sh #2056

Uh oh!

Conversation

petersomogyi commented Jul 13, 2020

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ndimiduk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ndimiduk Jul 15, 2020

Choose a reason for hiding this comment

Uh oh!

petersomogyi Jul 15, 2020

Choose a reason for hiding this comment

Uh oh!

liuml07 Jul 17, 2020

Choose a reason for hiding this comment

Uh oh!

petersomogyi Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

liuml07 Jul 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ndimiduk Jul 15, 2020

Choose a reason for hiding this comment

Uh oh!

ndimiduk Jul 15, 2020

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Jul 15, 2020

Uh oh!

Apache-HBase commented Jul 15, 2020

Uh oh!

Apache-HBase commented Jul 15, 2020

Uh oh!

liuml07 commented Jul 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liuml07 Jul 20, 2020 •

edited

Loading