Improve Inactive Reconciler Performance when --lookup-balance-by-block=false

[patrick-ogrady]

Motivation

Fix #65

Solution

• Ensure all seen accounts are added to the inactiveAccountQueue even if we are far behind tip
• Ensure accounts aren't leaked from the inactiveAccountQueue (could occur if accounts
• Improve performance by changing frequent operation (seenAccounts check on each block) from O(n) to ~O(1) (caused reconciler to get much slower as more accounts seen).
• Ensure inactive reconciler respects the highWaterMark (so useless queries are not performed during syncing).
• Fix edge case issue where an account could be added to the queue multiple times (in different go routines). This fix was possible because a lookup is performed while holding the lock and that lookup is now ~O(1) time now.

Other Thoughts

We could further optimize the reconciler to cache balances it retrieved when behind tip so that reconciliation could begin before making it to tip. This would require some careful thinking and testing and should be attempted in another PR.

[coveralls]

Pull Request Test Coverage Report for Build 2540

• 14 of 96 (14.58%) changed or added relevant lines in 2 files are covered.
• 3 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-1.0%) to 64.759%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
reconciler/configuration.go	1	9	11.11%
reconciler/reconciler.go	13	87	14.94%

Files with Coverage Reduction	New Missed Lines	%
reconciler/reconciler.go	3	27.7%

Totals
Change from base Build 2433:	-1.0%
Covered Lines:	1867
Relevant Lines:	2883

---

💛 - Coveralls

[heimdall-asguard]

Review Error for juliankoh @ 2020-07-15 11:03:36 UTC
User failed mfa authentication, public email is not set on your github profile. see go/mfa-help

[heimdall-asguard]

Review Error for juliankoh @ 2020-07-15 11:04:24 UTC
User failed mfa authentication, public email is not set on your github profile. see go/mfa-help