miner, consensus/clique: avoid memory leak during block stasis

[holiman]

When a clique network is not progressing, we see a memory leak:

This is pretty bad, because whenever one signer falls off, then the rest of them eventually will crash too -- so one failing sealer will wreak havoc, causing other sealers to crash and making it very difficult to get the network live again.

Every 3 seconds (miner.recommit interval) we shove a new task to be handed, where we e.g. deep-copy the receipts and other stuff and send it over a channel w.taskCh <- &task{receipts: receipts, state: s, block: block, createdAt: time.Now()}:

That get's shoved into a map

             w.pendingTasks[sealHash] = task

So every 3 seconds (miner recommit), we leak a block into memory, basically.
The routine that would have cleaned that up :

    clearPending := func(number uint64) {
        w.pendingMu.Lock()
        for h, t := range w.pendingTasks {
            if t.block.NumberU64()+staleThreshold <= number {
                delete(w.pendingTasks, h)
            }
        }
        w.pendingMu.Unlock()
    }

Won't clean close to head, and besides, it's not executed unless blocks progress.

When we hit this case:

Nov 05 15:08:18 rinkeby-aws-eu-west-3-001 geth INFO [11-05|14:08:18.254] Signed recently, must wait for others
Nov 05 15:08:18 rinkeby-aws-eu-west-3-001 geth INFO [11-05|14:08:18.254] Commit new mining work number=9,589,541 sealhash=df3e3b..b20b0c uncles=0 txs=237 gas=29,998,852 fees=5.408818027 elapsed=74.878ms
Nov 05 15:08:21 rinkeby-aws-eu-west-3-001 geth INFO [11-05|14:08:21.253] Signed recently, must wait for others
Nov 05 15:08:21 rinkeby-aws-eu-west-3-001 geth INFO [11-05|14:08:21.253] Commit new mining work number=9,589,541 sealhash=16ca1f..b229ee uncles=0 txs=237 gas=29,998,852 fees=5.408818027 elapsed=73.620ms
Nov 05 15:08:24 rinkeby-aws-eu-west-3-001 geth INFO [11-05|14:08:24.253] Signed recently, must wait for others
Nov 05 15:08:24 rinkeby-aws-eu-west-3-001 geth INFO [11-05|14:08:24.254] Commit new mining work number=9,589,541 sealhash=c3e488..6dd7a3 uncles=0 txs=237 gas=29,998,852 fees=5.408818027 elapsed=72.891ms

Then Signed recently, must wait for others means that we did not actually start a sealing-task at all, so there's no point tracking the work package. This PR changes the behavior so that

1. We return an error from Clique, instead of logging + return nil,
2. When the engine returns an error, we un-track the work package.

[holiman]

Was running latest stable until 15:40, then restarted with this PR. So seems to work fine:

[karalabe]

SGTM