txhashset snapshot zips and dirs still using up a lot of disk space

[antiochp]

We clean up "old" fast sync snapshot dirs and zip files but I suspect we need to be more aggresive with these and only keep the last n around.
Ideally we can get rid of them shortly after using them. Maybe our node can determine this based on how we serve the txhashset request to the syncing peer.

-rw-r--r--  1 root root    0 Feb 20 16:33 grin.lock
drwxr-xr-x  4 root root 4.0K Jan 15 15:38 header
drwxr-xr-x  2 root root 4.0K Jan 15 15:38 lmdb
drwxr-xr-x  2 root root 4.0K Feb 25 09:17 peer
drwxr-xr-x  5 root root 4.0K May  5 12:03 txhashset
-rw-r--r--  1 root root 242M May  5 16:34 txhashset_snapshot_100376.zip 
-rw-r--r--  1 root root 241M May  5 14:39 txhashset_snapshot_123868.zip 
-rw-r--r--  1 root root 241M May  5 15:13 txhashset_snapshot_15110.zip
-rw-r--r--  1 root root 241M May  5 15:03 txhashset_snapshot_329292.zip
-rw-r--r--  1 root root 242M May  5 18:57 txhashset_snapshot_454730.zip
-rw-r--r--  1 root root 243M May  6 01:33 txhashset_snapshot_454978.zip
-rw-r--r--  1 root root 243M May  5 22:22 txhashset_snapshot_475856.zip
-rw-r--r--  1 root root 241M May  5 13:03 txhashset_snapshot_527162.zip 
-rw-r--r--  1 root root 241M May  5 14:52 txhashset_snapshot_694734.zip
-rw-r--r--  1 root root 243M May  6 00:44 txhashset_snapshot_69812.zip
-rw-r--r--  1 root root 244M May  6 05:22 txhashset_snapshot_833786.zip
-rw-r--r--  1 root root 242M May  5 21:29 txhashset_snapshot_85703.zip
-rw-r--r--  1 root root 243M May  6 02:49 txhashset_snapshot_879868.zip
-rw-r--r--  1 root root 242M May  5 18:05 txhashset_snapshot_896996.zip
drwxr-xr-x  5 root root 4.0K May  5 16:34 txhashset_zip_100376 
drwxr-xr-x  5 root root 4.0K May  5 14:38 txhashset_zip_123868 
drwxr-xr-x  5 root root 4.0K May  5 15:13 txhashset_zip_15110
drwxr-xr-x  5 root root 4.0K May  5 15:03 txhashset_zip_329292
drwxr-xr-x  5 root root 4.0K May  5 18:57 txhashset_zip_454730
drwxr-xr-x  5 root root 4.0K May  6 01:32 txhashset_zip_454978
drwxr-xr-x  5 root root 4.0K May  5 22:22 txhashset_zip_475856
drwxr-xr-x  5 root root 4.0K May  5 13:03 txhashset_zip_527162 
drwxr-xr-x  5 root root 4.0K May  5 14:52 txhashset_zip_694734
drwxr-xr-x  5 root root 4.0K May  6 00:44 txhashset_zip_69812
drwxr-xr-x  5 root root 4.0K May  6 05:22 txhashset_zip_833786
drwxr-xr-x  5 root root 4.0K May  5 21:29 txhashset_zip_85703
drwxr-xr-x  5 root root 4.0K May  6 02:49 txhashset_zip_879868
drwxr-xr-x  5 root root 4.0K May  5 18:05 txhashset_zip_896996

I believe we clean these up approx every 24 hours, but we can easily receive a lot of fast sync requests in a 24 hour period.

Related -

• zip_read is filling disk with lots of txhashset.zip files and temp dirs #2311
• Automatically clean up old txhashset snapshots #2671

[garyyu]

@antiochp are you running the old version? the new version has different file name convention which will use block hash instead of a random number as the suffix.

[antiochp]

How recent?

Edit: Looks like this was a previous version, this is what I'm seeing now on that node -

-rw-r--r-- 1 root root    0 Feb 20 16:33 grin.lock
drwxr-xr-x 4 root root 4.0K Jan 15 15:38 header
drwxr-xr-x 2 root root 4.0K Jan 15 15:38 lmdb
drwxr-xr-x 2 root root 4.0K Feb 25 09:17 peer
drwxr-xr-x 5 root root 4.0K May  5 12:03 txhashset
-rw-r--r-- 1 root root 256M May  8 05:43 txhashset_snapshot_00000564507f.zip

I think this is orthogonal to this issue though - we may still see a large number of zip files generated in a 24 hour period.
Or do we now clean them up more aggressively?

[garyyu]

Currently it's 1 hour.
With 24 hours, I will need for example 256M*100=25.6G for these zip files in my seed node.

[antiochp]

Currently it's 1 hour.

👍

[antiochp]

Closing - 1 hour I think is fine for now.
And #2813 will solve this in a different way.