Witness_node Memory Management Discussion

[vikramrajkumar]

From @abitmore on March 14, 2016 10:42

This ticket is for discussing potential optimization on memory management of witness_node.

Currently, 5 months after BTS 2.0 launch,

• head_block_num is about 4,300,000,
• a typical witness_node running in Windows 7 64 bit uses about 2GB of memory (physical+virtual),
• a typical witness_node running in Ubuntu 14.04 LTS amd64 uses about 2GB of memory as well (VIRT>2,000,000 and RES=1.16g on one of my node).

Perhaps we'll have to face scalability issues rather soon.

Ideas?

Copied from original issue: cryptonomex/graphene#621

[vikramrajkumar]

From @theoreticalbts on March 14, 2016 14:54

Originally BitShares was created with architecture of core (bare minimum needed for validation) and plugins (containing data not needed for validation, e.g. operation history). As time went on, more and more code stopped following this architecture. There's a lot of stuff in the core that needs to be moved to a plugin.

For example:

• Block producing witnesses need not track operation history at all
• Single-user wallets need only track operation history for user's account
• Hosted wallets need to track all operation history

Related: #246 #422 #571

As a large portion of the objects are never modified, as a stopgap you can simply run the witness node on a system with a large swap file.

[vikramrajkumar]

From @abitmore on March 14, 2016 18:46

Using larger swap file seems to be a good idea. However, when a swapped out object need to be read (but not be modified), it still need to be swapped in, this will more or less affect performance.

We need analysis to show what objects are in memory, how big is each of them, how useful is each of them etc. As @theoreticalbts mentioned, market history is one of useless objects for block producers.

[vikramrajkumar]

From @pmconrad on March 25, 2016 9:3

The "large swap" idea may not work if rarely used objects in memory are interleaved with often-used objects. A single often-used object in a memory page will effectively lock that page in RAM.

[abitmore]

Linking related discussions: #65, #116, #157, #233.

Currently the head block number is around 16,800,000, and typical memory usage is more than 10GB.

When I was writing issue 233 I thought the biggest part of RAM is consumed by raw blocks, however it's not true. The biggest part is consumed by history objects and related data: 1.11.X, 2.9.X and etc. After removed these two lines, a node can run with less than 2G of RAM, although it can't serve any history related API call, that's fine for block producing nodes and seed nodes.

Actually 1.11.X objects aren't consensus data, because they aren't referenced by any operation nor evaluator nor other consensus related code. However, there are some businesses built on top of BitShares rely on consistent numbering of 1.11.X objects, E.G. bittrex.com, for compatibility we can't simply remove them (by default). It's best that we introduce a switch, enabled by default, so people who don't need them can disable it manually, either when compiling or when running, but don't need to modify the code every time.

@vikramrajkumar @oxarbitrage thoughts?

[oxarbitrage]

i agree on everything, memory usage is too much and most nodes probably don't need to be exposing history. let me know if you want me to add the command line option to the node. need to be well documented all around that to enable history you will need a new command line argument explicit.

[pmconrad]

See #288 for a config variable to en/disable plugins at runtime.

In addition, we could add an additional switch to the account_history plugin that preserves 1.11.x object only if impacted is not empty and/or contains only tracked accounts. That would allow tracking specific accounts without the bloat of having a complete operation history. -> #289

[pmconrad]

It seems that the create-remove trick has horrible performance if the result is populated sparsely. I'm now sitting at 13 hours for a replay, and it's still at 95%. But memory is at 1GB, so it's working at least.
Will try a workaround by calling index.use_next_id instead of create+remove.

[abitmore]

#292 is related.

[oxarbitrage]

With head_block 20,355,040 we need more than 40 gigs of ram to run a full node with all the history. Almost all the ram is consumed by account_history_plugin specially automated bots accounts with millions of operations. improvements were made on this, some options were added to the cli startup command and they are documented here https://github.com/bitshares/bitshares-core/wiki/Memory-reduction-for-nodes however we were discussing a few days in telegram on what will be the best way to store this history on disk. a database makes sense, elasticsearch was suggested, could be mongodb or any other.

We may use this thread to discuss what will be the best way to do it and start working on it.

[abitmore]

More info: after fixed #454 and #472 with #478, and moving to ElasticSearch, I think it's safe to close this ticket.