zip_read is filling disk with lots of txhashset.zip files and temp dirs

[antiochp]

Every time a peer syncs the chain state from our node we -

• copy our entire txhashset dir into a temp dir
• zip the temp dir up

In floonet right now the txhashset.zip is running at around 35MB.
So every peer coming online and syncing from us burns an additional 70MB of local disk space.

-rw-r--r--  1 root root  33M Jan  5 17:34 txhashset_snapshot_184053.zip
-rw-r--r--  1 root root  35M Jan  7 01:52 txhashset_snapshot_206426.zip
-rw-r--r--  1 root root  34M Jan  6 16:03 txhashset_snapshot_210624.zip
-rw-r--r--  1 root root  33M Jan  5 22:46 txhashset_snapshot_243020.zip
-rw-r--r--  1 root root  33M Jan  6 08:41 txhashset_snapshot_258549.zip
-rw-r--r--  1 root root  31M Jan  5 03:52 txhashset_snapshot_286784.zip
-rw-r--r--  1 root root  33M Jan  6 00:02 txhashset_snapshot_376418.zip
-rw-r--r--  1 root root  35M Jan  7 00:40 txhashset_snapshot_377277.zip
-rw-r--r--  1 root root  34M Jan  6 12:05 txhashset_snapshot_43901.zip
-rw-r--r--  1 root root  32M Jan  5 13:46 txhashset_snapshot_463581.zip
-rw-r--r--  1 root root  36M Jan  7 03:50 txhashset_snapshot_479246.zip
-rw-r--r--  1 root root  34M Jan  6 19:13 txhashset_snapshot_605382.zip
-rw-r--r--  1 root root  32M Jan  5 13:17 txhashset_snapshot_646275.zip
-rw-r--r--  1 root root  31M Jan  5 02:39 txhashset_snapshot_649741.zip
-rw-r--r--  1 root root  32M Jan  5 13:04 txhashset_snapshot_658632.zip
drwxr-xr-x  5 root root 4.0K Jan  5 17:34 txhashset_zip_184053
drwxr-xr-x  5 root root 4.0K Jan  7 01:52 txhashset_zip_206426
drwxr-xr-x  5 root root 4.0K Jan  6 16:03 txhashset_zip_210624
drwxr-xr-x  5 root root 4.0K Jan  5 22:46 txhashset_zip_243020
drwxr-xr-x  5 root root 4.0K Jan  6 08:41 txhashset_zip_258549
drwxr-xr-x  5 root root 4.0K Jan  5 03:52 txhashset_zip_286784
drwxr-xr-x  5 root root 4.0K Jan  6 00:02 txhashset_zip_376418
drwxr-xr-x  5 root root 4.0K Jan  7 00:40 txhashset_zip_377277
drwxr-xr-x  5 root root 4.0K Jan  6 12:05 txhashset_zip_43901
drwxr-xr-x  5 root root 4.0K Jan  5 13:46 txhashset_zip_463581
drwxr-xr-x  5 root root 4.0K Jan  7 03:50 txhashset_zip_479246
drwxr-xr-x  5 root root 4.0K Jan  6 19:13 txhashset_zip_605382
drwxr-xr-x  5 root root 4.0K Jan  5 13:17 txhashset_zip_646275
drwxr-xr-x  5 root root 4.0K Jan  5 02:39 txhashset_zip_649741
drwxr-xr-x  5 root root 4.0K Jan  5 13:04 txhashset_zip_658632

This is problematic at 35MB over time if lots of new peers sync from us. This is likely really problematic for any seed nodes out there if they support connections with 100s of peers.

At 1GB this is likely catastrophic.

I think we lost sight of the original intent with the state sync here over time.
We should be able to provide a txhashset.zip as cheaply as possible (in terms of local cpu and disk as well as bytes over the network).

Things to think about or consider (some are mutually exclusive approaches) -

• Is it possible to build the zip (or tar) on the fly when sending it over (and never actually write it to disk)?
• Can we (via zip -i or rust equivalent) explicitly list everything to zip up, rather than removing them from a temp dir?
• Use some kind of snapshot logic - we don't need to be necessarily build a txhashset.zip for any arbitrary header. Could we build one (and reuse it) at the closest hour (60 blocks) or day (1440 blocks)?
• Clean up our zip files (and temp dirs if we still have them) once they are no longer needed?

[sesam]

related: #1928

[quentinlesceller]

Fixed by #2575.