…ookup"

This reverts commit 4f51cc1.

Revert etcd-io#13714

When clients have no permission to perform whatever operation, then
the applying may fail. We should also move consistent_index forward
in this case, otherwise the consitent_index may smaller than the
snapshot index.

…_353

[3.5] Update consitent_index when applying fails

Update crypto to address CVE-2022-27191.

The CVE fix is added in 0.0.0-20220315160706-3147a52a75dd but this
change updates to latest.

Update golang.org/x/crypto to latest

we found a lease leak issue:
if a new member(by member add) is recovered by snapshot, and then
become leader, the lease will never expire afterwards. leader will
log the revoke failure caused by "invalid auth token", since the
token provider is not functional, and drops all generated token
from upper layer, which in this case, is the lease revoking
routine.

[backport 3.5]: server/auth: enable tokenProvider if recoved store enables auth

[release-3.5] Test release scripts

This PR removes additional clone when building artifacts.

When releasing v3.5.4 this clone was main cause of issues and
confusion about what release script is doing.

release.sh script already clones repo in /tmp/ directory, so clonning
before build is not needed. As precautions for bug in script leaving
/tmp/ clone in bad state  I moved "Verify the latest commit has the
version tag" and added "Verify the clean working tree" to be always run
before build.

[release-3.5] scripts: Avoid additional repo clone

The first bug fix is to resolve the race condition between goroutine
and channel on the same leases to be revoked. It's a classic mistake
in using Golang channel + goroutine. Please refer to
https://go.dev/doc/effective_go#channels

The second bug fix is to resolve the issue that etcd lessor may
continue to schedule checkpoint after stepping down the leader role.

[3.5] Backport two lease related bug fixes to 3.5

The FileReader interface is the wrapper of io.Reader. It provides
the fs.FileInfo as well. The FileBufReader struct is the wrapper of
bufio.Reader, it also provides fs.FileInfo.

Signed-off-by: Benjamin Wang <[email protected]>

…file

Currently the max size of each WAL entry is hard coded as 10MB. If users
set a value > 10MB for the flag --max-request-bytes, then etcd may run
into a situation that it successfully processes a big request, but fails
to decode it when replaying the WAL file on startup.

On the other hand, we can't just remove the limitation, because if a
WAL entry is somehow corrupted, and its recByte is a huge value, then
etcd may run out of memory. So the solution is to restrict the max size
of each WAL entry as a dynamic value, which is the remaining size of
the WAL file.

Signed-off-by: Benjamin Wang <[email protected]>

[3.5] Restrict the max size of each WAL entry to the remaining size of the WAL file

Cherry pick the PR etcd-io#12992
to 3.5, so please refer to the original PR for more detailed info.

Signed-off-by: Benjamin Wang <[email protected]>

[3.5] client/v3: do not overwrite authTokenBundle on dial

Make sure that WithPrefix correctly set the flag, and add test.
Also, add test for WithFromKey.

fixes etcd-io#14056

Signed-off-by: Sahdev Zala <[email protected]>

…-#14182-upstream-release-3.5

Automated cherry pick of etcd-io#14182

The golang buildin package `flag` doesn't support `uint32` data
type, so we need to support it via the `flag.Var`.

Signed-off-by: Benjamin Wang <[email protected]>

…each client can open at a time

Also refer to etcd-io#14169 (comment)

Signed-off-by: Benjamin Wang <[email protected]>

Signed-off-by: Benjamin Wang <[email protected]>

[3.5] Support configuring `MaxConcurrentStreams` for http2

Signed-off-by: Jille Timmermans <[email protected]>

[3.5] clientv3: Fix parsing of ETCD_CLIENT_DEBUG

Signed-off-by: Hitoshi Mitake <[email protected]>

server/auth: protect rangePermCache with a RW lock

This changes the default parent-based trace sampling rate from
100% to 0%. Due to the high QPS etcd can handle, having 100% trace
sampling leads to very high resource usage. Defaulting to 0% means
that only already-sampled traces will be sampled in etcd.

Fixes etcd-io#14310

Signed-off-by: Mike Dame <[email protected]>

Change default sampling rate from 100% to 0%

Signed-off-by: Benjamin Wang <[email protected]>

Signed-off-by: Benjamin Wang <[email protected]>

Upgrade grpc to 1.41.0;
Run ./script/fix.sh to fix all related issue.

Signed-off-by: Benjamin Wang <[email protected]>

Refer to etcd-io@a0bdfc4

Signed-off-by: Benjamin Wang <[email protected]>

Refer to etcd-io#14318

Signed-off-by: Benjamin Wang <[email protected]>

[3.5] etcdserver: bump OpenTelemetry to 1.0.1 and gRPC to 1.41.0

Streams are now closed after being used in the lessor `keepAliveOnce` method.
This prevents the "failed to receive lease keepalive request from gRPC stream"
message from being logged by the server after the context is cancelled by the
client.

Signed-off-by: Justin Kolberg <[email protected]>

…stream

[3.5] clientv3: close streams after use in lessor keepAliveOnce method

Only `net.TCPConn` supports `SetKeepAlive` and `SetKeepAlivePeriod`
by default, so if you want to warp multiple layers of net.Listener,
the `keepaliveListener` should be the one which is closest to the
original `net.Listener` implementation, namely `TCPListener`.

Also refer to etcd-io#14356

Signed-off-by: Benjamin Wang <[email protected]>

[3.5] Refactor the keepAliveListener and keepAliveConn

Signed-off-by: Vitalii Levitskii <[email protected]>

Problem: We pass grpc context down to applier in readonly serializable txn.
This context can be cancelled for example due to timeout.
This will trigger panic inside applyTxn

Solution: Only panic for transactions with write operations

fixes etcd-io#14110
main PR etcd-io#14149

Signed-off-by: Bogdan Kanivets <[email protected]>

…oints-autosync

Backport of pull/14354 to release-3.5

For a cluster with only one member, the raft always send identical
unstable entries and committed entries to etcdserver, and etcd
responds to the client once it finishes (actually partially) the
applying workflow.

When the client receives the response, it doesn't mean etcd has already
successfully saved the data, including BoltDB and WAL, because:
   1. etcd commits the boltDB transaction periodically instead of on each request;
   2. etcd saves WAL entries in parallel with applying the committed entries.
Accordingly, it may run into a situation of data loss when the etcd crashes
immediately after responding to the client and before the boltDB and WAL
successfully save the data to disk.
Note that this issue can only happen for clusters with only one member.

For clusters with multiple members, it isn't an issue, because etcd will
not commit & apply the data before it being replicated to majority members.
When the client receives the response, it means the data must have been applied.
It further means the data must have been committed.
Note: for clusters with multiple members, the raft will never send identical
unstable entries and committed entries to etcdserver.

Signed-off-by: Benjamin Wang <[email protected]>

…raft_3_5

[release-3.5] fix the potential data loss for clusters with only one member

Cherry pick etcd-io#14419 to 3.5.

Signed-off-by: Benjamin Wang <[email protected]>

[3.5] Move consistent_index forward when executing alarmList operation

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Get 100% coverage on InitialCheck and PeriodicCheck functions to avoid
any mistakes.

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Makes it easier to test hash match between scheduleCompaction and
HashByRev.

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

… Defrag

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Signed-off-by: Marek Siarkowicz <[email protected]>

Fix corruption checks v3.5

This has been additionally verified by running the tests locally as a
basic smoke test. GitHub Actions doesn't provide MacOS M1 (arm64) yet,
so there's no good way to automate testing.

Ran `TMPDIR=/tmp make test` locally. The `TMPDIR` bit is needed so
there's no really long path used that breaks Unix socket setup in one of
the tests.

Signed-off-by: Marek Siarkowicz <[email protected]>

- permissions were incorrectly loaded on restarts.
- etcd-io#14355
- Backport of etcd-io#14358

Signed-off-by: vivekpatani <[email protected]>

[release-3.5] server,test: refresh cache on each NewAuthStore

[release-3.5] server/etcdmain: add build support for Apple M1

A WAL object was closed by defer, however the WAL was rewritten afterwards,
so defer closed already closed WAL but not the new one. It caused a data
race between writing file and cleaning up a temporary test directory,
which led to a non-deterministic bug.

Fixes etcd-io#14332

Signed-off-by: Vladimir Sokolov <[email protected]>

…-max-index-test-fix-3.5

[release-3.5] testing: fix TestOpenWithMaxIndex cleanup

Due to a duplicate call of clientConfigFromCmd, the move-leader command
would fail with "conflicting environment variable is shadowed by corresponding command-line flag".
Also in scenarios where no command-line flag was supplied.

Signed-off-by: Thomas Jungblut <[email protected]>

etcdctl: allow move-leader to connect to multiple endpoints

[3.5] server: don't panic in readonly serializable txn

Signed-off-by: Benjamin Wang <[email protected]>

…er_20220913_3.5

[release-3.5] fix the flaky test TestV3AuthRestartMember