Chain Halt: Invariant broken on GoS

[hendrikhofstadt]

What happened is:

A unbonding period was complete and in the EndBlocker of x/stake UnbondAllMatureValidatorQueue was called which completed the unbonding by removing the validator.

This triggered the hook RemoveValidator which is handled by x/distribution. The hook there will delete the validatorDistInfo. There is a sanity check that DelAccum is 0 but the validator still had commission in his distInfo that was unclaimed. When then the DistInfo was deleted the invariant broke because all of the tokens in the distInfo were wiped/burnt but still in the LooseTokens.

Expected behaviour: Either withdraw commission when transitioning bonded->unbonding or withdraw the commission in the ValidatorRemoved hook. Tokens should be withdrawn and not burnt (and if burnt, they should be removed from the LooseTokens).

I think the actual problem already occurred when the unbonding was issued.
We should do the following:

In onDelegationRemoved if the delAddr = valAddr, we have to withdraw the commission as this means that the validator will be jailed/unbonded. Also a sanity check in RemoveValidatorDistInfo would make sense.

That sounds like the best possible patch to me. However that also means we won't be able to live-patch GoS. Also we have to be careful with the fork unless we cleanup the unbondings in postprocessing or fork before the relevant unbonding happened (~ height 300)

Stacktrace:

invariant broken: loose token invariance:
	pool.LooseTokens: 1284177.3999994578
	sum of account tokens: 1284173.0694983980" stack="goroutine 655 [running]:
runtime/debug.Stack(0xc0058575a8, 0xf4e7c0, 0xc0046eb2a0)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func2(0xc000137180, 0x12318b0)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:576 +0x57
panic(0xf4e7c0, 0xc0046eb2a0)
	/usr/lib/golang/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariantsOnContext(0xc000a40000, 0x13326a0, 0xc00121aba0, 0xc003624700, 0xc)
	/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:36 +0x35b
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).assertRuntimeInvariants(0xc000a40000)
	/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/invariants.go:28 +0xf6
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc000a40000, 0x13326a0, 0xc000e67140, 0xc00368cfc0, 0xd, 0x2cb3, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:221 +0x241
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0x13326a0, 0xc000e67140, 0xc00368cfc0, 0xd, 0x2cb3, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:168 +0xc8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc000180600, 0x2cb3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:802 +0x105
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc000077680, 0x2cb3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:239 +0xea
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc0007fac90, 0x2cb3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x51
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0x1333520, 0xc000923120, 0x1338680, 0xc0007fac90, 0xc0074f6960, 0xc000e66ae0, 0x133d200, 0xc0001a4128, 0xdfdd, 0xc0032dcbe0, ...)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:265 +0x701
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc0000e8070, 0x8, 0x0, 0xc0008f82d6, 0x6, 0xc0008f82f0, 0xe, 0x2cb2, 0xdfdd, 0xc0032dcbe0, ...)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:96 +0x15f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc000137180, 0x2cb3)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1290 +0xa8c
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc000137180, 0x2cb3)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1221 +0x451
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc000137180, 0x0, 0x2cb3)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1167 +0x90
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc000137180, 0x2cb3, 0x0)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1198 +0x6b8
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc000137180, 0xc005863400, 0xc004378060, 0x28, 0x1232b70, 0x107, 0xc005859ad0)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1622 +0xc03
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc000137180, 0xc005863400, 0xc004378060, 0x28, 0xc0035b3320, 0x107, 0x107)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1469 +0x59
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc000137180, 0x1324d40, 0xc005d8dd68, 0xc004378060, 0x28)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:650 +0x696
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc000137180, 0x0)
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:607 +0x670
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
	/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:300 +0x132

A unbonding period ended in the said block. Pretty sure this caused the invariant to break.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[alexanderbez]

/cc @cwgoes @rigelrozanski