module: update cespare/xxhash to v2

[zchee]

Update cespare/xxhash package to github.com/cespare/xxhash/v2. Also, update related modules to the current latest.

Here is the benchmark result using GCP Compute instance:

• machine_type
  • n1-standard-16
• image-family
  • debian-9
• boot-disk-type
  • pd-ssd
• preemptible
  • true

cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU @ 2.20GHz
stepping        : 0
microcode       : 0x1
cpu MHz         : 2200.000
cache size      : 56320 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat arch_capabilities
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4400.00
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:
.
.
.

gce-benchmark.bash startup-script

#!/bin/bash
set -x

start_time="$(date -u +%s)"

echo 'Install google-fluentd ...'
curl -sSL https://dl.google.com/cloudagents/install-logging-agent.sh | sudo bash

echo 'Restart google-fluentd ...'
service google-fluentd restart

# ----------------------------------------------------------------------------
# Start tasks

sudo apt-get update
sudo apt-get install -yqq --no-install-recommends --no-install-suggests \
  ca-certificates \
  curl \
  netbase \
  wget \
  \
  bzr \
  git \
  mercurial \
  openssh-client \
  subversion \
  \
  procps \
  \
  g++ \
  gcc \
  libc6-dev \
  make \
  pkg-config

GITHUB_USER="$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/github_user)"
GIT_REMOTE="$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/git_remote)"
PACKAGE="$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/package)"
BRANCH="$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/branch)"
BENCH_FUNCS="$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/bench_funcs)"
STATE="$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/state)"

export GOLANG_VERSION=1.12.4

wget -qq -O ~/go.tgz "https://golang.org/dl/go${GOLANG_VERSION}.linux-amd64.tar.gz"
sudo tar -C /usr/local -xzf ~/go.tgz
rm ~/go.tgz

export GOPATH=~/go
export GOCACHE=~/.cache/go-build
export PATH="$GOPATH/bin:/usr/local/go/bin:$PATH"
mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
mkdir -p "$GOCACHE" && chmod -R 777 "$GOCACHE"

go version

go get -u -v "$PACKAGE"

export GO111MODULE=on

cd "$GOPATH/src/$PACKAGE" || return
if echo "$PACKAGE" | grep "$GITHUB_USER"; then
  git checkout "origin/$BRANCH"
else
  git remote add "$GITHUB_USER" "$GIT_REMOTE"
  git fetch "$GITHUB_USER"
  git checkout "$GITHUB_USER/$BRANCH"
fi

go mod download
go mod tidy -v
go mod vendor -v

REVISION="$(git rev-parse --short -q HEAD)"
BENCH_OUT="$PWD/$(echo "$PACKAGE" | tr '/' '-')@$REVISION.$STATE.txt"

GOMAXPROCS=$(nproc) go test -v -mod=vendor -run='^$' -bench="$BENCH_FUNCS" -benchmem -benchtime=10s "$PACKAGE" | tee -a "$BENCH_OUT" || true
# touch "$BENCH_OUT"
# for ncpu in 4 8 16; do
#   GOMAXPROCS=$ncpu go test -v -mod=vendor -run='^$' -bench="$BENCH_FUNCS" -benchmem -benchtime=10s "$PACKAGE" | tee -a "$BENCH_OUT" || true
# done

CPUINFO_OUT="$PWD/$(echo "$PACKAGE" | tr '/' '-')@$REVISION.$STATE.cpuinfo.txt"

cat /proc/cpuinfo > "$CPUINFO_OUT"

gsutil cp "$BENCH_OUT" "$CPUINFO_OUT" "$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/gsutil_benchstat_bucket_name)" || true

# End tasks
# ----------------------------------------------------------------------------

end_time="$(date -u +%s)"
elapsed="$(("$end_time-$start_time"))"
echo "Total of $elapsed seconds elapsed for tasks"

echo 'Delete own GCE instance ...'
yes | gcloud compute instances delete "$(hostname)" --zone "$(curl -s -H 'Metadata-Flavor:Google' http://metadata.google.internal/computeMetadata/v1/instance/zone | cut -d/ -f4)"

gcloud command

GOOGLE_CLOUD_PROJECT='fooproject' \
  MACHINE_TYPE='n1-standard-16' \
  GITHUB_USER='{VictoriaMetrics or zchee}' \
  GIT_REMOTE='https://github.com/VictoriaMetrics/fastcache' \
  PACKAGE='github.com/VictoriaMetrics/fastcache' \
  BRANCH='{master or xxhash/v2}' \
  BENCH_FUNCS='Set|Get' \
  STATE='{old or new}' \
  GSUTIL_BENCHSTAT_BUCKET_NAME='gs://foobucket/benchstat/' ;
  gcloud --project="$GOOGLE_CLOUD_PROJECT" alpha compute instances create --zone 'asia-northeast1-a' \
  --machine-type "$MACHINE_TYPE" --image-project='debian-cloud' --image-family='debian-9' \
  --boot-disk-type='pd-ssd' --preemptible \
  --scopes 'https://www.googleapis.com/auth/cloud-platform' \
  --metadata="github_user=${GITHUB_USER},git_remote=${GIT_REMOTE},package=${PACKAGE},branch=${BRANCH},bench_funcs=${BENCH_FUNCS},state=${STATE},gsutil_benchstat_bucket_name=${GSUTIL_BENCHSTAT_BUCKET_NAME}" \
  --metadata-from-file='startup-script=gce-benchmark.bash' \
  --verbosity='debug' \
  'benchstat'

benchstat result

name               old time/op    new time/op    delta
SetBig-16            18.6µs ± 0%    18.0µs ± 0%   ~     (p=1.000 n=1+1)
GetBig-16            3.91µs ± 0%    3.89µs ± 0%   ~     (p=1.000 n=1+1)
BigCacheSet-16       5.91ms ± 0%    5.92ms ± 0%   ~     (p=1.000 n=1+1)
BigCacheGet-16       1.53ms ± 0%    1.63ms ± 0%   ~     (p=1.000 n=1+1)
BigCacheSetGet-16    6.69ms ± 0%    7.34ms ± 0%   ~     (p=1.000 n=1+1)
CacheSet-16          1.89ms ± 0%    1.88ms ± 0%   ~     (p=1.000 n=1+1)
CacheGet-16          1.18ms ± 0%    1.01ms ± 0%   ~     (p=1.000 n=1+1)
CacheSetGet-16       6.12ms ± 0%    6.39ms ± 0%   ~     (p=1.000 n=1+1)
StdMapSet-16         19.0ms ± 0%    18.4ms ± 0%   ~     (p=1.000 n=1+1)
StdMapGet-16         3.00ms ± 0%    3.00ms ± 0%   ~     (p=1.000 n=1+1)
StdMapSetGet-16      78.7ms ± 0%    78.6ms ± 0%   ~     (p=1.000 n=1+1)
SyncMapSet-16        49.2ms ± 0%    47.5ms ± 0%   ~     (p=1.000 n=1+1)
SyncMapGet-16         609µs ± 0%     612µs ± 0%   ~     (p=1.000 n=1+1)
SyncMapSetGet-16     4.01ms ± 0%    4.01ms ± 0%   ~     (p=1.000 n=1+1)

name               old speed      new speed      delta
SetBig-16          14.1GB/s ± 0%  14.6GB/s ± 0%   ~     (p=1.000 n=1+1)
GetBig-16          69.4GB/s ± 0%  69.8GB/s ± 0%   ~     (p=1.000 n=1+1)
BigCacheSet-16     11.1MB/s ± 0%  11.1MB/s ± 0%   ~     (p=1.000 n=1+1)
BigCacheGet-16     42.9MB/s ± 0%  40.1MB/s ± 0%   ~     (p=1.000 n=1+1)
BigCacheSetGet-16  19.6MB/s ± 0%  17.9MB/s ± 0%   ~     (p=1.000 n=1+1)
CacheSet-16        34.6MB/s ± 0%  34.8MB/s ± 0%   ~     (p=1.000 n=1+1)
CacheGet-16        55.7MB/s ± 0%  65.0MB/s ± 0%   ~     (p=1.000 n=1+1)
CacheSetGet-16     21.4MB/s ± 0%  20.5MB/s ± 0%   ~     (p=1.000 n=1+1)
StdMapSet-16       3.45MB/s ± 0%  3.57MB/s ± 0%   ~     (p=1.000 n=1+1)
StdMapGet-16       21.9MB/s ± 0%  21.9MB/s ± 0%   ~     (p=1.000 n=1+1)
StdMapSetGet-16    1.67MB/s ± 0%  1.67MB/s ± 0%   ~     (all equal)
SyncMapSet-16      1.33MB/s ± 0%  1.38MB/s ± 0%   ~     (p=1.000 n=1+1)
SyncMapGet-16       108MB/s ± 0%   107MB/s ± 0%   ~     (p=1.000 n=1+1)
SyncMapSetGet-16   32.7MB/s ± 0%  32.7MB/s ± 0%   ~     (p=1.000 n=1+1)

name               old alloc/op   new alloc/op   delta
SetBig-16             1.00B ± 0%     1.00B ± 0%   ~     (all equal)
GetBig-16             1.00B ± 0%     1.00B ± 0%   ~     (all equal)
BigCacheSet-16       6.30MB ± 0%    6.30MB ± 0%   ~     (p=1.000 n=1+1)
BigCacheGet-16        556kB ± 0%     556kB ± 0%   ~     (p=1.000 n=1+1)
BigCacheSetGet-16    5.18MB ± 0%    5.18MB ± 0%   ~     (p=1.000 n=1+1)
CacheSet-16            571B ± 0%      572B ± 0%   ~     (p=1.000 n=1+1)
CacheGet-16            572B ± 0%      572B ± 0%   ~     (all equal)
CacheSetGet-16       1.15kB ± 0%    1.91kB ± 0%   ~     (p=1.000 n=1+1)
StdMapSet-16          275kB ± 0%     275kB ± 0%   ~     (p=1.000 n=1+1)
StdMapGet-16         2.57kB ± 0%    2.57kB ± 0%   ~     (p=1.000 n=1+1)
StdMapSetGet-16       325kB ± 0%     325kB ± 0%   ~     (p=1.000 n=1+1)
SyncMapSet-16        3.44MB ± 0%    3.44MB ± 0%   ~     (p=1.000 n=1+1)
SyncMapGet-16          424B ± 0%      423B ± 0%   ~     (p=1.000 n=1+1)
SyncMapSetGet-16     3.41MB ± 0%    3.41MB ± 0%   ~     (p=1.000 n=1+1)

name               old allocs/op  new allocs/op  delta
SetBig-16              0.00           0.00        ~     (all equal)
GetBig-16              0.00           0.00        ~     (all equal)
BigCacheSet-16         4.00 ± 0%      4.00 ± 0%   ~     (all equal)
BigCacheGet-16         131k ± 0%      131k ± 0%   ~     (all equal)
BigCacheSetGet-16      131k ± 0%      131k ± 0%   ~     (p=1.000 n=1+1)
CacheSet-16            1.00 ± 0%      1.00 ± 0%   ~     (all equal)
CacheGet-16            1.00 ± 0%      1.00 ± 0%   ~     (all equal)
CacheSetGet-16         2.00 ± 0%      3.00 ± 0%   ~     (p=1.000 n=1+1)
StdMapSet-16          65.5k ± 0%     65.5k ± 0%   ~     (all equal)
StdMapGet-16           13.0 ± 0%      13.0 ± 0%   ~     (all equal)
StdMapSetGet-16       65.5k ± 0%     65.5k ± 0%   ~     (all equal)
SyncMapSet-16          263k ± 0%      263k ± 0%   ~     (all equal)
SyncMapGet-16          13.0 ± 0%      13.0 ± 0%   ~     (all equal)
SyncMapSetGet-16       262k ± 0%      262k ± 0%   ~     (all equal)

benchcmp result

benchmark                      old ns/op     new ns/op     delta
BenchmarkSetBig-16             18645         18014         -3.38%
BenchmarkGetBig-16             3910          3886          -0.61%
BenchmarkBigCacheSet-16        5910041       5920862       +0.18%
BenchmarkBigCacheGet-16        1526565       1634421       +7.07%
BenchmarkBigCacheSetGet-16     6693952       7341088       +9.67%
BenchmarkCacheSet-16           1892147       1883619       -0.45%
BenchmarkCacheGet-16           1177109       1008929       -14.29%
BenchmarkCacheSetGet-16        6121587       6393339       +4.44%
BenchmarkStdMapSet-16          18968931      18382867      -3.09%
BenchmarkStdMapGet-16          2999080       2995737       -0.11%
BenchmarkStdMapSetGet-16       78671513      78627566      -0.06%
BenchmarkSyncMapSet-16         49229967      47471593      -3.57%
BenchmarkSyncMapGet-16         609224        612426        +0.53%
BenchmarkSyncMapSetGet-16      4013008       4010546       -0.06%

benchmark                      old MB/s     new MB/s     speedup
BenchmarkSetBig-16             14059.39     14551.92     1.04x
BenchmarkGetBig-16             69399.58     69813.42     1.01x
BenchmarkBigCacheSet-16        11.09        11.07        1.00x
BenchmarkBigCacheGet-16        42.93        40.10        0.93x
BenchmarkBigCacheSetGet-16     19.58        17.85        0.91x
BenchmarkCacheSet-16           34.64        34.79        1.00x
BenchmarkCacheGet-16           55.68        64.96        1.17x
BenchmarkCacheSetGet-16        21.41        20.50        0.96x
BenchmarkStdMapSet-16          3.45         3.57         1.03x
BenchmarkStdMapGet-16          21.85        21.88        1.00x
BenchmarkStdMapSetGet-16       1.67         1.67         1.00x
BenchmarkSyncMapSet-16         1.33         1.38         1.04x
BenchmarkSyncMapGet-16         107.57       107.01       0.99x
BenchmarkSyncMapSetGet-16      32.66        32.68        1.00x

benchmark                      old allocs     new allocs     delta
BenchmarkSetBig-16             0              0              +0.00%
BenchmarkGetBig-16             0              0              +0.00%
BenchmarkBigCacheSet-16        4              4              +0.00%
BenchmarkBigCacheGet-16        131072         131072         +0.00%
BenchmarkBigCacheSetGet-16     131078         131077         -0.00%
BenchmarkCacheSet-16           1              1              +0.00%
BenchmarkCacheGet-16           1              1              +0.00%
BenchmarkCacheSetGet-16        2              3              +50.00%
BenchmarkStdMapSet-16          65538          65538          +0.00%
BenchmarkStdMapGet-16          13             13             +0.00%
BenchmarkStdMapSetGet-16       65547          65547          +0.00%
BenchmarkSyncMapSet-16         262589         262589         +0.00%
BenchmarkSyncMapGet-16         13             13             +0.00%
BenchmarkSyncMapSetGet-16      262188         262188         +0.00%

benchmark                      old bytes     new bytes     delta
BenchmarkSetBig-16             1             1             +0.00%
BenchmarkGetBig-16             1             1             +0.00%
BenchmarkBigCacheSet-16        6302976       6302970       -0.00%
BenchmarkBigCacheGet-16        556265        556263        -0.00%
BenchmarkBigCacheSetGet-16     5184662       5184654       -0.00%
BenchmarkCacheSet-16           571           572           +0.18%
BenchmarkCacheGet-16           572           572           +0.00%
BenchmarkCacheSetGet-16        1145          1908          +66.64%
BenchmarkStdMapSet-16          274693        274704        +0.00%
BenchmarkStdMapGet-16          2566          2567          +0.04%
BenchmarkStdMapSetGet-16       324890        324849        -0.01%
BenchmarkSyncMapSet-16         3439036       3439029       -0.00%
BenchmarkSyncMapGet-16         424           423           -0.24%
BenchmarkSyncMapSetGet-16      3411022       3411018       -0.00%

---

The performance increases a bit. Basically, this pull request updates and use the latest xxhash package.

Whether the merge or not, up to you.

[codecov-io]

Codecov Report

Merging #7 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master       #7   +/-   ##
=======================================
  Coverage   75.99%   75.99%           
=======================================
  Files           4        4           
  Lines         504      504           
=======================================
  Hits          383      383           
  Misses         66       66           
  Partials       55       55

Impacted Files	Coverage Δ
fastcache.go	88.3% <ø> (ø)	⬆️
bigcache.go	77.14% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e17c1b...9338041. Read the comment docs.

[valyala]

Thanks, @zchee !

[zchee]

@valyala :)

[narqo]

Wasn't this a breaking change? For projects that doesn't use gomodules, e.g. dep, there is an error when one tries to use [email protected]

Solving failure: No versions of github.com/cespare/xxhash met constraints:
	v2.0.0: Could not introduce github.com/cespare/[email protected], as its subpackage github.com/cespare/xxhash/v2 is missing. (Package is required by github.com/VictoriaMetrics/[email protected].)

See golang/dep#2139

[valyala]

@narqo , yup, this is unfortunate. There are the following solutions:

• to stick to v1.4.6 when working with dep;
• to wait for the fix for dep ensure does not work with "go modules" import path suffixes (/vN) golang/dep#2139;
• to upgrade to go mod, since dep is deprecated.

I'd prefer the last one :)

[zchee]

@narqo @valyala
Ohh, sorry I forget this change is no backward compatibility.
But yes, I also suggest @valyala solution (last one)