Catchpoints: Enrich catchpoint generation and status with KV metadata

[michaeldiamant]

Enriches catchpoints with the following KV metadata mirroring accounts:

• Catchpoint creation (writes) - Add total KV count to:
  • CatchpointFileHeader
  • CatchpointGenerationEvent
• Catchpoint status (reads) - Add total, processed, and verified KV counts to:
  • /v2/status
  • goal node status

Prior to the PR, account and KV hashes are not disambiguated. Consequently, it's possible to see catchpoint accounts verified > total accounts like the example below:

❯ goal node status -w 1000 -d b4
Last committed block: 0
Sync Time: 4.4s
Catchpoint: 21973000#PJIVBPJMRDKP65YUNCUWJ6WO2ZQ4EKOLFG4S3LA3INYTY4WBCWBQ
Catchpoint total accounts: 24040
Catchpoint accounts processed: 24040
Catchpoint accounts verified: 24207    # non-sensical:  verified > total accounts
...

[jannotti]

I think this is the right approach. I suspect it would be possible to get the number of kvs for the catchpoint from the iterator rather that the select count, but if it takes any effort to restructure, this seems fine.

[codecov]

Codecov Report

Merging #4808 (56b4a54) into master (9202a43) will increase coverage by 0.01%.
The diff coverage is 58.97%.

@@            Coverage Diff             @@
##           master    #4808      +/-   ##
==========================================
+ Coverage   54.65%   54.67%   +0.01%     
==========================================
  Files         416      417       +1     
  Lines       53710    53734      +24     
==========================================
+ Hits        29357    29379      +22     
+ Misses      21923    21922       -1     
- Partials     2430     2433       +3     

Impacted Files	Coverage Δ
catchup/catchpointService.go	7.73% <0.00%> (-0.08%)	⬇️
cmd/goal/node.go	12.67% <0.00%> (-0.04%)	⬇️
daemon/algod/api/server/v2/handlers.go	0.91% <0.00%> (-0.01%)	⬇️
daemon/algod/api/server/v2/test/helpers.go	73.94% <ø> (ø)
node/node.go	4.37% <0.00%> (-0.03%)	⬇️
ledger/accountdb.go	72.35% <46.15%> (-0.17%)	⬇️
ledger/catchpointwriter.go	59.37% <50.00%> (-0.31%)	⬇️
ledger/catchpointtracker.go	59.02% <73.33%> (-0.17%)	⬇️
ledger/catchupaccessor.go	63.83% <92.85%> (+0.47%)	⬆️
ledger/hashkind_string.go	100.00% <100.00%> (ø)
... and 9 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[michaeldiamant]

As of 7e1f45c, here's a proof point against betanet.

Generating catchpoints with Total KVs in file header:

❯ catchpointdump file -t 21980000.catchpoint.tar | grep ^Total
Total Accounts: 24041
Total KVs: 358
Total Chunks: 48

Applying a catchpoint generated by a node running the PR:

❯ goal node status -w 1000 -d b5
Last committed block: 0
Sync Time: 15.2s
Catchpoint: 21980000#T6KDPI5OLWU4T3WB7QUKPLGF2NMAJM6WE7ZBFAJEGXYQDQT66NKA
Catchpoint total accounts: 24041
Catchpoint accounts processed: 24041
Catchpoint accounts verified: 1188
Catchpoint total KVs: 358
Catchpoint KVs processed: 358
Catchpoint KVs verified: 0
Catchpoint total blocks: 1321
Catchpoint downloaded blocks: 452
Genesis ID: betanet-v1.0
Genesis hash: mFgazF+2uRS1tMiL9dsj01hJGySEmPN28B/TjjvpVW0=

[michaeldiamant]

As of 85e25e9, applying a catchpoint generated by a node running the PR yields the processed == verified counts:

❯ goal node status -w 1000 -d b5
Last committed block: 0
Sync Time: 4.9s
Catchpoint: 21997000#4AR2347S3M6CFRVZVFML4ZCIZWMWWZQTZQ5C3AQUTACM573B7HDQ
Catchpoint total accounts: 24061
Catchpoint accounts processed: 24061
Catchpoint accounts verified: 24061
Catchpoint total KVs: 358
Catchpoint KVs processed: 358
Catchpoint KVs verified: 358
Catchpoint total blocks: 1321
Catchpoint downloaded blocks: 32
Genesis ID: betanet-v1.0
Genesis hash: mFgazF+2uRS1tMiL9dsj01hJGySEmPN28B/TjjvpVW0=

[jannotti]

I don't feel strongly about swapping the order of kv, accounts all over the place, so leaving an approval.

Earlier we talked about not exporting some stuff, and the KV had to remain exported for some reason. With that in mind, I'd probably add something to those names, as that means Account is now an exported symbol. Seems pretty weird to me. Maybe AccountHashKind, etc?

I hate going overboard on long names - but these are weird little things that don't feel like they deserve such important names.

[algorandskiy]

could you make a minimal diff by re-merging master after initializeHashes gets it?

[algorandskiy]

The change looks good, some code-style comments