pruning archive node cannot breach more than 2.0blk/s super slow 2.7.2

[ApolloMuses]

• Parity Ethereum version: 2.7.2
• Operating system: Linux
• Installation: one-line installer
• Fully synchronized: no*
• Network: ethereum
• Restarted: yes

Configuration script used:
parity --allow-ips=public --max-peers=128 --max-pending-peers=128 --cache-size=32768 --no-serve-light --no-periodic-snapshot --no-secretstore --scale-verifiers --jsonrpc-apis="web3,eth,pubsub,net,parity,parity_pubsub,traces,rpc,secretstore,shh,shh_pubsub" --jsonrpc-hosts="all" --pruning archive

Currently synced to ~7,945,500 Blocks and going at 1-2blk/s on average.

Hardware:
Ec2: M5.8xlarge w/ a 3.5Tb attached EBS volume.
This is equivalent of:
32vCPU
128Gb of memory
Bandwidth of 10Gps

Prior to upgrading to v 2.7.2, the node was crashing due to memory leaks, but upgrading seem to have fixed the issue, having said that the sync sleep is still agonizingly slow.

Anyone else experiencing this or have any idea how to speed up the sync time while still maintaining an archival node?

Before anyone ask, I've also tried the configurations on c & r typed instances and noticed marginal if not equivalent performance.

Any comments or suggestions would be much appreciated. Thanks in advance!

*the node was previously at one point fully synced but that was with a m5.24xlarge instance, but later on the node was downgraded to a less powerful instance which couldn't keep up with the network.

[dvdplm]

@ApolloMuses we are seeing several reports from other users on tracing and/or archive nodes with similar issues. We are still investigating the issue and collecting information from reporters, see e.g. #11494 or #11503. Is your node also under RPC traffic while syncing? Can you try running the node without specifying the cache size (i.e. remove --cache-size)? I don't expect it to make a difference for syncing but worth a try.
Also, do you mind sharing the rocksdb LOG file found in the database directory (/your/path/to/chains/ethereum/db/<your id>/overlayrecent/db/LOG)?
Further questions: do you have any CPU/IO/RAM monitoring on the node, anything interesting you can share with us from that?

[ApolloMuses]

Thanks for the quick response.

Here's the log below:

Note the path was a bit different, I didn't have the overlayrecent folder but rather the archive folder that led to the same downstream file path.

2020/02/19-00:04:34.404895 7fecefaed600 RocksDB version: 6.2.4
2020/02/19-00:04:34.404914 7fecefaed600 Git sha rocksdb_build_git_sha:@76a56d89a7740f8dbb01edabf1ea5abc95a67657@
2020/02/19-00:04:34.404916 7fecefaed600 Compile date Feb  6 2020
2020/02/19-00:04:34.404918 7fecefaed600 DB SUMMARY
2020/02/19-00:04:34.433077 7fecefaed600 CURRENT file:  CURRENT
2020/02/19-00:04:34.433080 7fecefaed600 IDENTITY file:  IDENTITY
2020/02/19-00:04:34.433085 7fecefaed600 MANIFEST file:  MANIFEST-1174770 size: 3951069 Bytes
2020/02/19-00:04:34.433087 7fecefaed600 SST files in /home/ubuntu/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d/archive/db dir,$
2020/02/19-00:04:34.433089 7fecefaed600 Write Ahead Log file in /home/ubuntu/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d/arch$
2020/02/19-00:04:34.433133 7fecefaed600                         Options.error_if_exists: 0
2020/02/19-00:04:34.433135 7fecefaed600                       Options.create_if_missing: 1
2020/02/19-00:04:34.433136 7fecefaed600                         Options.paranoid_checks: 1
2020/02/19-00:04:34.433137 7fecefaed600                                     Options.env: 0x562e57737670
2020/02/19-00:04:34.433138 7fecefaed600                                Options.info_log: 0x562e580cb110
2020/02/19-00:04:34.433139 7fecefaed600                Options.max_file_opening_threads: 16
2020/02/19-00:04:34.433140 7fecefaed600                              Options.statistics: (nil)
2020/02/19-00:04:34.433141 7fecefaed600                               Options.use_fsync: 0
2020/02/19-00:04:34.433142 7fecefaed600                       Options.max_log_file_size: 0
2020/02/19-00:04:34.433143 7fecefaed600                  Options.max_manifest_file_size: 1073741824
2020/02/19-00:04:34.433144 7fecefaed600                   Options.log_file_time_to_roll: 0
2020/02/19-00:04:34.433145 7fecefaed600                       Options.keep_log_file_num: 1
2020/02/19-00:04:34.433146 7fecefaed600                    Options.recycle_log_file_num: 0
2020/02/19-00:04:34.433147 7fecefaed600                         Options.allow_fallocate: 1
2020/02/19-00:04:34.433148 7fecefaed600                        Options.allow_mmap_reads: 0
2020/02/19-00:04:34.433148 7fecefaed600                       Options.allow_mmap_writes: 0
2020/02/19-00:04:34.433149 7fecefaed600                        Options.use_direct_reads: 0
2020/02/19-00:04:34.433150 7fecefaed600                        Options.use_direct_io_for_flush_and_compaction: 0
2020/02/19-00:04:34.433151 7fecefaed600          Options.create_missing_column_families: 0
2020/02/19-00:04:34.433152 7fecefaed600                              Options.db_log_dir:
2020/02/19-00:04:34.433153 7fecefaed600                                 Options.wal_dir: /home/ubuntu/.local/share/io.parity.ethereum/chains/ethereum$
2020/02/19-00:04:34.433154 7fecefaed600                Options.table_cache_numshardbits: 6
2020/02/19-00:04:34.433155 7fecefaed600                      Options.max_subcompactions: 1
2020/02/19-00:04:34.433156 7fecefaed600                  Options.max_background_flushes: -1
2020/02/19-00:04:34.433157 7fecefaed600                         Options.WAL_ttl_seconds: 0
2020/02/19-00:04:34.433158 7fecefaed600                       Options.WAL_size_limit_MB: 0
2020/02/19-00:04:34.433159 7fecefaed600             Options.manifest_preallocation_size: 4194304
2020/02/19-00:04:34.433160 7fecefaed600                     Options.is_fd_close_on_exec: 1
2020/02/19-00:04:34.433161 7fecefaed600                   Options.advise_random_on_open: 1
2020/02/19-00:04:34.433162 7fecefaed600                    Options.db_write_buffer_size: 0
2020/02/19-00:04:34.433163 7fecefaed600                    Options.write_buffer_manager: 0x562e580c6460
2020/02/19-00:04:34.433163 7fecefaed600         Options.access_hint_on_compaction_start: 1
2020/02/19-00:04:34.433164 7fecefaed600  Options.new_table_reader_for_compaction_inputs: 0
2020/02/19-00:04:34.433165 7fecefaed600           Options.random_access_max_buffer_size: 1048576
2020/02/19-00:04:34.433176 7fecefaed600                      Options.use_adaptive_mutex: 0
2020/02/19-00:04:34.433178 7fecefaed600                            Options.rate_limiter: (nil)
2020/02/19-00:04:34.433179 7fecefaed600     Options.sst_file_manager.rate_bytes_per_sec: 0
2020/02/19-00:04:34.433180 7fecefaed600                       Options.wal_recovery_mode: 2
2020/02/19-00:04:34.433181 7fecefaed600                  Options.enable_thread_tracking: 0
2020/02/19-00:04:34.433182 7fecefaed600                  Options.enable_pipelined_write: 0
2020/02/19-00:04:34.433183 7fecefaed600         Options.allow_concurrent_memtable_write: 1
2020/02/19-00:04:34.433183 7fecefaed600      Options.enable_write_thread_adaptive_yield: 1
2020/02/19-00:04:34.433184 7fecefaed600             Options.write_thread_max_yield_usec: 100
2020/02/19-00:04:34.433185 7fecefaed600            Options.write_thread_slow_yield_usec: 3
2020/02/19-00:04:34.433186 7fecefaed600                               Options.row_cache: None
2020/02/19-00:04:34.433187 7fecefaed600                              Options.wal_filter: None
2020/02/19-00:04:34.433188 7fecefaed600             Options.avoid_flush_during_recovery: 0
2020/02/19-00:04:34.433189 7fecefaed600             Options.allow_ingest_behind: 0
2020/02/19-00:04:34.433190 7fecefaed600             Options.preserve_deletes: 0
2020/02/19-00:04:34.433191 7fecefaed600             Options.two_write_queues: 0
2020/02/19-00:04:34.433192 7fecefaed600             Options.manual_wal_flush: 0
2020/02/19-00:04:34.433193 7fecefaed600             Options.atomic_flush: 0
2020/02/19-00:04:34.433193 7fecefaed600             Options.avoid_unnecessary_blocking_io: 0
2020/02/19-00:04:34.433194 7fecefaed600             Options.max_background_jobs: 16
2020/02/19-00:04:34.433195 7fecefaed600             Options.max_background_compactions: -1
2020/02/19-00:04:34.433196 7fecefaed600             Options.avoid_flush_during_shutdown: 0
2020/02/19-00:04:34.433197 7fecefaed600           Options.writable_file_max_buffer_size: 1048576
2020/02/19-00:04:34.433198 7fecefaed600             Options.delayed_write_rate : 16777216
2020/02/19-00:04:34.433199 7fecefaed600             Options.max_total_wal_size: 0
2020/02/19-00:04:34.433199 7fecefaed600             Options.delete_obsolete_files_period_micros: 21600000000
2020/02/19-00:04:34.433200 7fecefaed600                   Options.stats_dump_period_sec: 600
2020/02/19-00:04:34.433201 7fecefaed600                 Options.stats_persist_period_sec: 600
2020/02/19-00:04:34.433202 7fecefaed600                 Options.stats_history_buffer_size: 1048576
2020/02/19-00:04:34.433203 7fecefaed600                          Options.max_open_files: 512
2020/02/19-00:04:34.433204 7fecefaed600                          Options.bytes_per_sync: 1048576
2020/02/19-00:04:34.433205 7fecefaed600                      Options.wal_bytes_per_sync: 0
2020/02/19-00:04:34.433206 7fecefaed600                   Options.strict_bytes_per_sync: 0
2020/02/19-00:04:34.433207 7fecefaed600       Options.compaction_readahead_size: 0
2020/02/19-00:04:34.433208 7fecefaed600 Compression algorithms supported:
2020/02/19-00:04:34.433209 7fecefaed600         kZSTDNotFinalCompression supported: 0
2020/02/19-00:04:34.433210 7fecefaed600         kZSTD supported: 0
2020/02/19-00:04:34.433211 7fecefaed600         kXpressCompression supported: 0
2020/02/19-00:04:34.433212 7fecefaed600         kLZ4HCCompression supported: 0
2020/02/19-00:04:34.433213 7fecefaed600         kLZ4Compression supported: 0
2020/02/19-00:04:34.433214 7fecefaed600         kBZip2Compression supported: 0
2020/02/19-00:04:34.433216 7fecefaed600         kZlibCompression supported: 0
2020/02/19-00:04:34.433217 7fecefaed600         kSnappyCompression supported: 1
2020/02/19-00:04:34.433221 7fecefaed600 Fast CRC32 supported: Supported on x86
2020/02/19-00:04:34.433332 7fecefaed600 [db/version_set.cc:4053] Recovering from manifest file: /home/ubuntu/.local/share/io.parity.ethereum/chains/e$
2020/02/19-00:04:34.433366 7fecefaed600 [db/column_family.cc:482] --------------- Options for column family [default]:
2020/02/19-00:04:34.433373 7fecefaed600               Options.comparator: leveldb.BytewiseComparator
2020/02/19-00:04:34.433374 7fecefaed600           Options.merge_operator: None
2020/02/19-00:04:34.433375 7fecefaed600        Options.compaction_filter: None
2020/02/19-00:04:34.433376 7fecefaed600        Options.compaction_filter_factory: None
2020/02/19-00:04:34.433376 7fecefaed600         Options.memtable_factory: SkipListFactory
2020/02/19-00:04:34.433377 7fecefaed600            Options.table_factory: BlockBasedTable
2020/02/19-00:04:34.433397 7fecefaed600            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562e57fc17c0)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  pin_top_level_index_and_filter: 1
  index_type: 0
  data_block_index_type: 0

I'll try the remove cache again to see if it makes a difference, I removed it previously and it didn't really affect syncing performance.

In terms of usage:

CPU: ~3%. Quite low.

RAM:

total        used        free      shared  buff/cache   available
Mem:         126123       11003       78842           8       36277      114281
Swap:          8191           0        8191

IO:

Network:

[ApolloMuses]

The node has minimal to no RPC traffic. So I can't imagine that being the same issue.

After removing the cache, the parity process hangs. Now I can't even get the process to start with or without the cache tag...

2020-02-19 21:58:10 UTC Starting Parity-Ethereum/v2.7.2-stable-2662d19-20200206/x86_64-unknown-linux-gnu/rustc1.41.0
2020-02-19 21:58:10 UTC Keys path /home/ubuntu/.local/share/io.parity.ethereum/keys/ethereum
2020-02-19 21:58:10 UTC DB path /home/ubuntu/.local/share/io.parity.ethereum/chains/ethereum/db/906a34e69aec8c0d
2020-02-19 21:58:10 UTC State DB configuration: archive
2020-02-19 21:58:10 UTC Operating mode: active
2020-02-19 21:58:10 UTC Warning: Warp Sync is disabled because of non-default pruning mode.

[ApolloMuses]

@dvdplm David, any ideas on how this can be resolved?