Skip to content

HBASE-24710 Incorrect checksum calculation in saveVersion.sh #2056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
petersomogyi merged 2 commits into from
Jul 21, 2020
Merged

HBASE-24710 Incorrect checksum calculation in saveVersion.sh #2056

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
686e041
HBASE-24710 Incorrect checksum calculation in saveVersion.sh
petersomogyi Jul 13, 2020
73f809a
Add missing pipe
petersomogyi Jul 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions hbase-common/src/saveVersion.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,34 +30,34 @@ nativeOutputDirectory="$2/native/utils/"
pushd .
cd ..

user=`whoami | sed -n -e 's/\\\/\\\\\\\\/p'`
user=$(whoami | sed -n -e "s/\\\/\\\\\\\\/p")
if [ "$user" == "" ]
then
user=`whoami`
user=$(whoami)
fi
date=`date`
cwd=`pwd`
date=$(date)
cwd=$(pwd)
if [ -d .svn ]; then
revision=`(svn info | sed -n -e 's/Last Changed Rev: \(.*\)/\1/p') || true`
url=`(svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true`
revision=$( (svn info | sed -n -e "s/Last Changed Rev: \(.*\)/\1/p") || true)
url=$( (svn info | sed -n -e 's/^URL: \(.*\)/\1/p') || true)
elif [ -d .git ]; then
revision=`git log -1 --no-show-signature --pretty=format:"%H" || true`
hostname=`hostname`
revision=$(git log -1 --no-show-signature --pretty=format:"%H" || true)
hostname=$(hostname)
url="git://${hostname}${cwd}"
fi
if [ -z "${revision}" ]; then
echo "[WARN] revision info is empty! either we're not in VCS or VCS commands failed." >&2
revision="Unknown"
url="file://$cwd"
fi
if ! [ -x "$(command -v openssl)" ]; then
if ! [ -x "$(command -v openssl)" ]; then
if ! [ -x "$(command -v gpg)" ]; then
srcChecksum="Unknown"
else
srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | cut -d ' ' -f 1`
Copy link
Member

@ndimiduk ndimiduk Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous use of cut here at the end is surprising. It took only the first block...

Copy link
Contributor Author

@petersomogyi petersomogyi Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on history it was used for md5 and md5sum commands but it wasn't removed when SHA512 was introduced in saveVersion.sh.

root@8224f4c69e6f:/# echo "foo" | md5sum
d3b07384d113edec49eaa6238ad5ff00  -

Copy link
Member

@liuml07 liuml07 Jul 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does tr -d '[:space:]' work? It's simpler and shorter than tr '\n' ' ' | sed 's/[[:space:]]*//g'

Copy link
Contributor Author

@petersomogyi petersomogyi Jul 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get a different output from your command, an extra %.

find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | tr -d '[:space:]'
3BF4CCC275112769C7AB0EDC302793FB81005BDC9681E1604BD965918363FC93311FC60B24000563DD4D3EFDA10685F549721A28428F23487813A7397FBC9BD0%
ind hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs gpg --print-md sha512 | gpg --print-md sha512 | tr '\n' ' ' | sed 's/[[:space:]]*//g'
3BF4CCC275112769C7AB0EDC302793FB81005BDC9681E1604BD965918363FC93311FC60B24000563DD4D3EFDA10685F549721A28428F23487813A7397FBC9BD0

Copy link
Member

@liuml07 liuml07 Jul 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's weird. I did not see in my macOS and Ubuntu Linux. But since this is not reliable, I guess we can stay with your script tr and sed - which works for me on both macOS and Ubuntu.

srcChecksum=$(find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort xargs gpg --print-md sha512 | gpg --print-md sha512 | tr '\n' ' ' | sed 's/[[:space:]]*//g')
fi
else
srcChecksum=`find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort | xargs openssl dgst -sha512 | openssl dgst -sha512 | cut -d ' ' -f 1`
srcChecksum=$(find hbase-*/src/main/ | grep -e "\.java" -e "\.proto" | LC_ALL=C sort xargs openssl dgst -sha512 | openssl dgst -sha512 | sed 's/^.* //')
Copy link
Member

@ndimiduk ndimiduk Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the purpose of the final sed? I see no difference.

$ echo "foo" | openssl dgst -sha512 | sed 's/^.* //'
0cf9180a764aba863a67b6d72f0918bc131c6772642cb2dce5a34f0a702f9470ddc2bf125c12198b1995c233c34b4afd346c54a2334c350a948a51b6e8b4e6b6
$ echo "foo" | openssl dgst -sha512
0cf9180a764aba863a67b6d72f0918bc131c6772642cb2dce5a34f0a702f9470ddc2bf125c12198b1995c233c34b4afd346c54a2334c350a948a51b6e8b4e6b6

Copy link
Member

@ndimiduk ndimiduk Jul 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this is the example from your comment on JIRA, a difference between the mac and linux versions of openssl dgst. Disregard.

fi
popd

Expand Down