core: txpool stable underprice drop order, perf fixes

[karalabe]

This PR fixes 3 performance instabilities in the transaction pool during high transaction churn.

The primary issue (for which the newly added test was written) is when the transaction pool is full of spam, and a new proper transaction is added. In this case, the desired behavior is that a cheap spam transaction is removed in favor of the new expensive transaction. The corner case in the current behavior is that if all the spam transactions are at the same price point, a random one will be dropped. This can cause a nonce gap, moving all subsequent spam transactions into the non-executable queue. Further ones will be dropped as a result (since there are stricter limits on non-executable transactions). This opens up many new slots in the pool, which in itself is not a problem, unless they get filled by spam again. In that case, a new proper transaction will again potentially remove hundreds of txs in one go. The end result is that transactions start to rotate in the pool and even in the network. The root cause is that one transaction may kick out many from the pool. Note, this is not really a DoS vector, only a networking annoyance and performance hit.

The solution is fairly simple. We already maintain a heap of the transactions sorted by price to know which to discard. This PR extends that heap so that same-price txs get sorted by nonce (larger is worse). When searching for cheap transactions to discard, we always discard the highest nonce at the cheapest price point. This is not particularly fair as accounts with high activity get hit first, but assuming the discarded transaction is spam, we don't care much.

---

Changing the above behavior surfaced two optimization bugs in the transaction price heap implementation:

• When capping the price pool (miner.setGasPrice or discard due to expensive tx), we counted the removal event twice: once in txPricedList.Cap/Discard and once in txpool.removeTx. This caused the price heap to assume a higher churn rate than in reality, causing higher resorting of transactions internally. This issue is an annoyance only, since it's a tiny performance hit. Nonetheless the PR fixes it by passing an outofbound flag to removeTx, signalling whether the transaction being removed was dropped at random (notify the price pool), or cleanly (don't notify the price pool).
• When a price cap (miner.setGasPrice or discard due to expensive tx) produced a gap, all the transactions moved from the pending pool to the queue were added to the price heap, duplicating existing ones. This was because txpool.enqueueTx assumed it's called on new transactions only, but it is actually called on old ones too when gaps are created. The PR fixes it by ensuring txpool.enqueueTx only adds truly new transactions to the priced pool.

[holiman]

LGTM

[holiman]

s/primarilly/primarily