new etcdv3 implementation

[hubo1016]

This should be a fix for #671

I have tested it with etcdv3 and it appears to be working, but still want some review.

Fixes #671
Fixes #685
Fixes #693

[hubo1016]

There is still a corner case when a connection failure happens between the watch response and the calling of GetValues. When this happens, the GetValues call fails and is not retried. If there are no further updates, the destination file is not updated even after the connection restore. It should be a rare case. We can add some retry logic in GetValues to prevent this from happening.

@okushchenko @trystanj What's your opinions?

[hubo1016]

@okushchenko I think this is ready to merge... In fact I've already using this implementation in the production environment of my company for a month

[hubo1016]

should also fix #693

[EyciaZhou]

doesn't call defer close(cancelRoutine) is ok? because seems the goroutine below will leak also.

go func() {
	select {
	case <-stopChan:
		cancel()
	case <-cancelRoutine:
		return
	}
}()

[hubo1016]

Quite right I think. Should I remove the channel and wait for ctx.Done() instead?

[hubo1016]

@EyciaZhou should be fixed

[hubo1016]

Also fix #685

[okushchenko]

@hubo1016 thanks for submitting a PR! Sorry for not getting back to you for so long. The code looks solid. The only thing that I'm getting a hard time to grasp is why have you decided to group the get requests using a transaction? Does it reduce the number of requests to Etcd? What was your reasoning behind that?

The other thing that bugs me is that your changes are actually bigger than the current implementation of etcdv3 backend all together, but I'm ok with that.

[hubo1016]

@okushchenko this helps on the consistency of the configuration, ensuring all values are retrieved in the same revision. With the old implementation, if two changes are made in revision 1 and 2, both affect two keys (e.g. delete with prefix, and then insert two keys with transaction), confd may get the first key with value in revision 1, but the second in revision 2. Though finally the configuration will be synced to the latest revision, reloading an inconsistent configuration may cause errors.

It also reduces the latency of updating when there are a lot of keys.

[tinder-andyphan]

@hubo1016 @okushchenko curious about this line

https://github.com/kelseyhightower/confd/pull/678/files#diff-8386a15dc01968620740d0c9083008c7R88

and why it's necessary? couldn't this revision possibly not exist due to the disconnect/error being, say, a new leader being reeelcted/repartitioned, not due to compaction?