Autodetect lmdb storage allocation based on the env

[0xmichalis]

The hardcoded allocation in here is causing trouble in low-end devices such as ARM 64bit boards (eg. odroid c2). It should either be autodetected based on the environment where grin runs or at the very least driven by a config option.

[0xmichalis]

I saw this issue on an odroid c2, another occurence of it can be found in https://www.grin-forum.org/t/problems-building-on-windows-linux/184/13

[i1skn]

It also causes an issue with cargo lipo for iOS device. So far I manage to run it with /8192

[mcdallas]

I tried to compile from source on a raspberry pi 3 (1GB RAM) with 300GB HD and 3GB swap.
I changed this value to 1GB

grin/store/src/lmdb.rs

Line 75 in fcec70d

env_builder.set_mapsize(549755813888).unwrap_or_else(|e| {

and I got this error when I try to run grin:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error::Code(12, 'Cannot allocate memory')': libcore/result.rs:1009stack backtrace:
   0:   0xccac97 - backtrace::backtrace::trace::h1e6b3cbff53b4eca
   1:   0xccaa43 - <backtrace::capture::Backtrace as core::default::Default>::default::h8ede5ca42cdc5af6
   2:   0xccaa9f - backtrace::capture::Backtrace::new::hee8cfb25433837a0
   3:   0xc41077 - grin_util::logger::send_panic_to_log::{{closure}}::hec8b726050880d32
   4:   0xd6c34b - std::panicking::rust_panic_with_hook::hb4c40b3721cfbfb0
                at libstd/panicking.rs:480
   5:   0xd6bf2b - std::panicking::continue_panic_fmt::h7e84a0a182ce9c2d
                at libstd/panicking.rs:390
   6:   0xd6be13 - rust_begin_unwind
                at libstd/panicking.rs:325
   7:   0xdc2ccf - core::panicking::panic_fmt::h0c93626b89c38af6
                at libcore/panicking.rs:77
   8:   0xbdd1b7 - core::result::unwrap_failed::hd301dc9e6e84f390
   9:   0xbd7a63 - grin_store::lmdb::new_named_env::h6b9fc91b70a994da
  10:   0x75cd83 - grin_servers::grin::server::Server::new::hf793ba76a2fce3f1
  11:   0x5f51af - grin::cmd::server::start_server::h70ce3659d88b1ee9
  12:   0x5f658f - grin::cmd::server::server_command::hd9bac72fdacd5914
  13:   0x5c6a97 - grin::real_main::h8d10e3b50895c646
  14:   0x5c5757 - grin::main::h8da7a1b283f4c24c
  15:   0x584c07 - std::rt::lang_start::{{closure}}::ha7d185b75f8ab45f
  16:   0xd6bd8b - std::rt::lang_start_internal::{{closure}}::h6b5f7ae7c14f8493
                at libstd/rt.rs:59
                 - std::panicking::try::do_call::he4712eb84191fb3b
                at libstd/panicking.rs:310
  17:   0xd8cfcf - __rust_maybe_catch_panic
                at libpanic_unwind/lib.rs:102
  18:   0xd6bcd7 - std::panicking::try::hd44c69d8635b9942
                at libstd/panicking.rs:289
  19:   0xd65107 - std::panic::catch_unwind::h4424b54e4de01dca
                at libstd/panic.rs:392
                 - std::rt::lang_start_internal::h76f990c2b2c74932
                at libstd/rt.rs:58
  20:   0x5c9edb - main
  21: 0x76d43677 - __libc_start_main

After changing it to 256MB it started running but got a segfault during block download.

I am a bit confused, I thought this value is about disk space (it defaults to 500GB) but it complains about memory?

[ignopeverell]

Both LMDB and our state storage use (mmap)[https://www.hbg.com/en-us/application-center/authentication/], which is a way to map virtual memory space onto disk. It'd be good to further track down the segfault.

[mcdallas]

This is the backtrace

#0  0x00ca16bc in secp256k1_ge_set_xquad (r=r@entry=0x0, x=x@entry=0x4c7fb250) at depend/secp256k1-zkp/src/group_impl.h:224
#1  0x00cab2dc in secp256k1_pedersen_commitment_load (commit=0x50660040, ge=0x0) at depend/secp256k1-zkp/src/modules/commitment/main_impl.h:32
#2  secp256k1_bulletproof_rangeproof_verify_multi (ctx=0x2952be8f, scratch=0xb4229db5, gens=0x5077f000, proof=0x50623f90, n_proofs=1, plen=675, min_value=0x0,
    commit=0x5062e348, n_commits=1, nbits=64, value_gen=0x50660000, extra_commit=0x5062e350, extra_commit_len=0x5062e358)
    at depend/secp256k1-zkp/src/modules/bulletproofs/main_impl.h:154
#3  0x00c8ac68 in secp256k1zkp::pedersen::_$LT$impl$u20$secp256k1zkp..Secp256k1$GT$::verify_bullet_proof_multi::h6b4776a370860842 ()
#4  0x00be5988 in grin_core::core::transaction::Output::batch_verify_proofs::hf9dd4aeb2ffa0c37 ()
#5  0x00be1b90 in grin_core::core::transaction::TransactionBody::validate::h98d6666409ea7ac6 ()
#6  0x00bf7344 in grin_core::core::block::Block::validate::hdfd8ed8d9c3d69e5 ()
#7  0x00b9cf88 in grin_chain::pipe::process_block::heeafc8ef5ee5f255 ()
#8  0x00b87c14 in grin_chain::chain::Chain::process_block_single::h5f5be89a471f5135 ()
#9  0x00b876b0 in grin_chain::chain::Chain::process_block::h78fa9a0bae1f163a ()
#10 0x00766fa4 in grin_servers::common::adapters::NetToChainAdapter::process_block::h2bff768004034a92 ()
#11 0x00762df8 in _$LT$grin_servers..common..adapters..NetToChainAdapter$u20$as$u20$grin_p2p..types..ChainAdapter$GT$::block_received::hf86a4f5bf734e1b0 ()
#12 0x00b45a7c in _$LT$grin_p2p..peers..Peers$u20$as$u20$grin_p2p..types..ChainAdapter$GT$::block_received::h4b5c89a7e7e3f9bb ()
#13 0x00b513b8 in _$LT$grin_p2p..peer..TrackingAdapter$u20$as$u20$grin_p2p..types..ChainAdapter$GT$::block_received::ha283817f993a951a ()
#14 0x00b35c28 in _$LT$grin_p2p..protocol..Protocol$u20$as$u20$grin_p2p..conn..MessageHandler$GT$::consume::h0ebd0da41bb1be95 ()
#15 0x52385ca4 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[kravens]

With Raspberry Pi 3 B+ this is indeed the issue:

Compiling grin_store v1.0.0 (/home/pi/grin/store)
warning: unused import: lmdb_zero
--> store/src/lib.rs:23:5
|
23 | use lmdb_zero;
| ^^^^^^^^^
|
= note: #[warn(unused_imports)] on by default

warning: unused import: memmap
--> store/src/lib.rs:24:5
|
24 | use memmap;
| ^^^^^^

error: literal out of range for usize
--> store/src/lmdb.rs:77:16
|
77 | .set_mapsize(549_755_813_888)
| ^^^^^^^^^^^^^^^
|
= note: #[deny(overflowing_literals)] on by default

error: aborting due to previous error

error: Could not compile grin_store.
warning: build failed, waiting for other jobs to finish...
error: build failed

[0xmichalis]

@yeastplume is this issue fixed by #2605? Seems you have removed the static allocation but I cannot run HEAD on my board just yet for reasons.