Error while proposing node removal

[danielmai]

What version of Dgraph are you using?

v1.1.0

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, OS)?

Ubuntu Linux

Steps to reproduce the issue (command/config used to run Dgraph).

1. Run a Dgraph cluster with multiple Alpha replicas in a group.

# From dgraph-io/dgraph repo root directory
cd ./compose
./run.sh

2. Remove an Alpha.

curl localhost:6180/removeNode?id=1&group=1

3. The Alpha is removed successfully according to the logs and /state. But the new Alpha leader continues to print the following log every second:

E0925 00:17:12.688470       1 groups.go:322] Error while proposing node removal: Node 0x1 not part of group
github.com/dgraph-io/dgraph/conn.(*Node).ProposePeerRemoval
	/tmp/go/src/github.com/dgraph-io/dgraph/conn/node.go:594
github.com/dgraph-io/dgraph/worker.(*groupi).applyState.func1
	/tmp/go/src/github.com/dgraph-io/dgraph/worker/groups.go:320
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337

Expected behaviour and actual result.

A successful node removal should not make an error log show up repeatedly.

[raasss]

Me having the same problem. But I think I did alpha node rebuild the wrong way.

1. I stopped 1 of 3 alpha nodes.
2. Wiped node database data and started node again.
3. The node started to panic because I forgot to do rest api call to remove node from cluster.
4. Stopped alpha node again
5. Wiped node database data
6. I did api call and removed node with http://:6080/removeNode?id=3&group=1'
7. Started node and he joined the cluster but I started getting same error log

[martinmr]

There doesn't to be an actual issue but I agree that the logs should be cleaned up. Here's what's happening.

1. Node is removed via the zero endpoint.
2. A new leader is elected.
3. The leader receives an update with the other node marked as removed. Node is removed successfully.
4. Subsequent updates try to remove the node again. It's already removed and not a peer of the new leader so an error is returned.

The solution is to make the new leader know that it's already removed the node so that it doesn't try again.