Fix limits

[pmconrad]

Limit allocation during deserialization

[jmjatlanta]

Please validate my interpretation of this change below. Am I missing some other benefit? If there is more discussion at STEEM, would you please provide a link?

--
While unpacking, we guess at the number of elements in a packed vector. We can easily overestimate, and reserve too many elements, thereby wasting memory, which will be reclaimed when we resize later in the process.

To reduce over-reserving, we now pick the minimum of (the estimated size or an arbitrary number of 200 ). While this may cause re-allocations, it is better than over-allocating.

[pmconrad]

Yes, basically. Related STEEM PRs:

• FC unpack fixes (master) steemit/steem#3201
• When unpacking containers, clear the structures (master) steemit/steem#3208

[abitmore]

It's mentioned in steemit/steem#3201 that resize() initializes but not only pre-allocates, thus may impact performance. Perhaps we can change them all to reserve()?

[pmconrad]

I don't see how to get around the initialization.
One option would be to use reserve and a temporary variable for the nested unpack call, then emplace_back( std::move(tmp) ), similar to

bitshares-fc/include/fc/io/raw.hpp

Lines 478 to 484 in 90f8ad5

	value.reserve( std::min( size.value, FC_MAX_PREALLOC_SIZE ) );
	for( uint32_t i = 0; i < size.value; ++i )
	{
	std::pair<K,V> tmp;
	fc::raw::unpack( s, tmp, _max_depth );
	value.insert( std::move(tmp) );
	}

. Obviously, creating the temporary variable also means one empty initialization for every element to be deserialized, so this is not an improvement.
Better ideas?

[abitmore]

Makes sense.

[pmconrad]

Thanks!