Parachain runtime instance cache #1687

Harrm · 2023-07-11T09:32:15Z

Referenced issues

Description of the Change

Adds a parachain runtime instance cache, so that parachains don't instantiate a WASM module every time they want to call a runtime method. Also, respect the memory page limit returned by session_executor_params runtime call.

Benefits

Faster PVF instantiation.

Possible Drawbacks

Increased memory consumption.

…achain-runtime-instance-cache

codecov · 2023-07-24T10:36:31Z

Codecov Report

Merging #1687 (2cbe0a3) into master (0cd656a) will increase coverage by 1.79%.
The diff coverage is 37.75%.

❗ Current head 2cbe0a3 differs from pull request most recent head 3a1bbca. Consider uploading reports for the commit 3a1bbca to get more accurate results

@@            Coverage Diff             @@
##           master    #1687      +/-   ##
==========================================
+ Coverage   21.56%   23.36%   +1.79%     
==========================================
  Files         745      708      -37     
  Lines       31935    29284    -2651     
  Branches    16586    15090    -1496     
==========================================
- Hits         6887     6841      -46     
+ Misses      19286    16722    -2564     
+ Partials     5762     5721      -41

Files Changed	Coverage Δ
core/api/service/system/impl/system_api_impl.cpp	`39.02% <0.00%> (ø)`
core/application/app_configuration.hpp	`100.00% <ø> (ø)`
core/application/impl/app_configuration_impl.hpp	`23.86% <0.00%> (ø)`
core/authorship/impl/block_builder_impl.hpp	`100.00% <ø> (ø)`
core/benchmark/block_execution_benchmark.cpp	`0.00% <ø> (ø)`
core/consensus/grandpa/impl/environment_impl.cpp	`12.03% <0.00%> (-5.34%)`	⬇️
core/injector/calculate_genesis_state.hpp	`31.81% <0.00%> (+1.38%)`	⬆️
core/network/impl/protocols/light.hpp	`0.00% <ø> (ø)`
core/network/impl/state_sync_request_flow.cpp	`0.00% <0.00%> (ø)`
core/network/types/collator_messages.hpp	`15.38% <0.00%> (+15.38%)`	⬆️
... and 62 more

... and 110 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

core/runtime/runtime_api/parachain_host_types.hpp

core/runtime/common/memory_allocator.hpp

core/injector/calculate_genesis_state.hpp

core/runtime/common/executor.cpp

turuslan · 2023-07-26T06:50:24Z

core/runtime/common/memory_allocator.cpp

      logger_->error(
          "Memory size exceeded when growing it on {} bytes, offset was 0x{:x}",
          chunk_sz,
          offset_);
      return 0;
    }
-    resize(offset_ + chunk_sz);
+    auto new_size = offset_ + chunk_sz;


Suggested change

auto new_size = offset_ + chunk_sz;

auto new_size = new_pages_num * kMemoryPageSize;

What for? resize() aligns the new size anyway.

turuslan · 2023-07-26T07:00:15Z

core/runtime/runtime_context.hpp

+  class RuntimeContext {
+   public:
+    // should be created from runtime contex factory
+    RuntimeContext() = delete;
+    RuntimeContext(const RuntimeContext &) = delete;
+    RuntimeContext &operator=(const RuntimeContext &) = delete;
+
+    RuntimeContext(RuntimeContext &&) = default;
+
+    // constructor for tests
+    static RuntimeContext create_TEST(
+        std::shared_ptr<ModuleInstance> module_instance) {
+      return RuntimeContext{module_instance};
+    }
+
+    struct ContextParams {
+      ContextParams() = delete;
+
+      MemoryLimits memory_limits;
+    };
+
+    const std::shared_ptr<ModuleInstance> module_instance;
+
+   private:
+    friend class RuntimeContextFactoryImpl;
+    friend class RuntimeContextFactory;
+    RuntimeContext(std::shared_ptr<ModuleInstance> module_instance);
+  };


Why not use ModuleInstance directly?

Ah, this should be noted in comments. ModuleInstances are reused, but before every call there should be some setup. RuntimeContext enforces this setup. Resetting memory, setting the correct batch.

core/runtime/runtime_context.hpp

turuslan · 2023-07-26T07:22:44Z

core/runtime/executor.hpp

+    virtual outcome::result<Buffer> callWithCtx(RuntimeContext &ctx,
+                                                std::string_view name,
+                                                const Buffer &encoded_args);


Move this wrapper method to ModuleInstance

core/parachain/pvf/pvf_impl.cpp

core/parachain/pvf/pvf_impl.hpp

core/parachain/pvf/pvf_impl.cpp

turuslan · 2023-07-28T06:52:01Z

core/parachain/pvf/pvf_impl.cpp

+    OUTCOME_TRY(
+        parent_hash,
+        block_header_repository_->getHashByNumber(params.relay_parent_number));


Suggested change

OUTCOME_TRY(

parent_hash,

block_header_repository_->getHashByNumber(params.relay_parent_number));

Not used in rust, also num -> hash is ambigous

core/parachain/pvf/pvf_impl.cpp

turuslan · 2023-07-31T05:10:01Z

core/parachain/pvf/pvf_runtime_cache.hpp

+    std::unordered_map<ParachainId, Entry> instance_cache_;
+    std::map<uint64_t, ParachainId> last_usage_time_;
+    const uint32_t instances_limit_ = 43;
+    std::atomic<uint64_t> time_ = 0;
+
+    void erase(decltype(instance_cache_)::iterator it);


Lru.

Keep 2 codes for para near upgrade, also there are less codes then paras.

Suggested change

std::unordered_map<ParachainId, Entry> instance_cache_;

std::map<uint64_t, ParachainId> last_usage_time_;

const uint32_t instances_limit_ = 43;

std::atomic<uint64_t> time_ = 0;

void erase(decltype(instance_cache_)::iterator it);

Lru<ValidationCodeHash, Entry> instance_cache_;

core/parachain/pvf/pvf_runtime_cache.hpp

turuslan · 2023-07-31T05:19:36Z

core/parachain/pvf/pvf_runtime_cache.hpp

+    using SafeInstance = SafeObject<std::shared_ptr<runtime::ModuleInstance>>;
+    using SafeInstanceRef = std::reference_wrapper<SafeInstance>;


Suggested change

using SafeInstance = SafeObject<std::shared_ptr<runtime::ModuleInstance>>;

using SafeInstanceRef = std::reference_wrapper<SafeInstance>;

turuslan · 2023-07-31T05:20:20Z

core/parachain/pvf/pvf_runtime_cache.hpp

+    outcome::result<SafeInstanceRef> requestInstance(
+        ParachainId para_id,
+        const common::Hash256 &code_hash,
+        const ParachainRuntime &code_zstd);


~BorrowedInstance() will call pool(self.borrowed)

Suggested change

outcome::result<SafeInstanceRef> requestInstance(

ParachainId para_id,

const common::Hash256 &code_hash,

const ParachainRuntime &code_zstd);

outcome::result<std::shared_ptr<runtime::ModuleInstance>> requestInstance(

const common::Hash256 &code_hash,

const ParachainRuntime &code_zstd);

pool(std::shared_ptr<runtime::ModuleInstance>);

turuslan · 2023-07-31T05:35:30Z

core/parachain/pvf/pvf_runtime_cache.cpp

+  PvfRuntimeCache::requestInstance(ParachainId para_id,
+                                   const common::Hash256 &code_hash,
+                                   const ParachainRuntime &code_zstd) {
+    ++time_;
+    std::unique_lock lock{instance_cache_mutex_};
+    auto it = instance_cache_.find(para_id);
+
+    bool it_found = it != instance_cache_.end();
+
+    bool same_hash =
+        it_found && it->second.instance.sharedAccess([](const auto &instance) {
+          return instance->getCodeHash();
+        }) == code_hash;
+
+    if (!(it_found && same_hash)) {
+      ParachainRuntime code;
+      OUTCOME_TRY(runtime::uncompressCodeIfNeeded(code_zstd, code));
+      OUTCOME_TRY(runtime_module, module_factory_->make(code));
+      OUTCOME_TRY(instance, runtime_module->instantiate());
+      if (it_found) {
+        erase(it);
+      }
+      SafeObject safe_instance{instance};
+
+      auto [new_it, inserted] =
+          instance_cache_.emplace(std::piecewise_construct,
+                                  std::forward_as_tuple(para_id),
+                                  std::forward_as_tuple(instance, time_));
+      BOOST_ASSERT(inserted);
+      last_usage_time_.emplace(time_, para_id);
+      it = new_it;
+      if (instance_cache_.size() > instances_limit_) {
+        cleanup(lock);
+      }
+    } else {
+      auto lru_it = last_usage_time_.find(it->second.last_used);
+      BOOST_ASSERT(lru_it != last_usage_time_.end());
+      last_usage_time_.erase(lru_it);
+      last_usage_time_.emplace(time_, para_id);
+      it->second.last_used = time_;
+    }
+    BOOST_ASSERT(it != instance_cache_.end());
+    return it->second.instance;
+  }
+
+  void PvfRuntimeCache::cleanup(const std::unique_lock<std::mutex> &lock) {
+    BOOST_ASSERT(lock.owns_lock());
+    BOOST_ASSERT(lock.mutex() == &instance_cache_mutex_);
+
+    for (auto it = last_usage_time_.begin();
+         it != last_usage_time_.end()
+         && last_usage_time_.size() > instances_limit_;) {
+      instance_cache_.erase(it->second);
+      it = last_usage_time_.erase(it);
+    }
+  }
+
+  void PvfRuntimeCache::erase(decltype(instance_cache_)::iterator it) {
+    // instance can be used at this point, so we wait for exclusive access
+    // and then extract it from the map (we cannot erase it from inside
+    // the exclusive access callback because destroying a locked mutex is
+    // UB)
+    it->second.instance.exclusiveAccess(
+        [this, it](auto) {
+          last_usage_time_.erase(it->second.last_used);
+          return instance_cache_.extract(it);
+        });
+  }


Suggested change

PvfRuntimeCache::requestInstance(ParachainId para_id,

const common::Hash256 &code_hash,

const ParachainRuntime &code_zstd) {

++time_;

std::unique_lock lock{instance_cache_mutex_};

auto it = instance_cache_.find(para_id);

bool it_found = it != instance_cache_.end();

bool same_hash =

it_found && it->second.instance.sharedAccess([](const auto &instance) {

return instance->getCodeHash();

}) == code_hash;

if (!(it_found && same_hash)) {

ParachainRuntime code;

OUTCOME_TRY(runtime::uncompressCodeIfNeeded(code_zstd, code));

OUTCOME_TRY(runtime_module, module_factory_->make(code));

OUTCOME_TRY(instance, runtime_module->instantiate());

if (it_found) {

erase(it);

}

SafeObject safe_instance{instance};

auto [new_it, inserted] =

instance_cache_.emplace(std::piecewise_construct,

std::forward_as_tuple(para_id),

std::forward_as_tuple(instance, time_));

BOOST_ASSERT(inserted);

last_usage_time_.emplace(time_, para_id);

it = new_it;

if (instance_cache_.size() > instances_limit_) {

cleanup(lock);

}

} else {

auto lru_it = last_usage_time_.find(it->second.last_used);

BOOST_ASSERT(lru_it != last_usage_time_.end());

last_usage_time_.erase(lru_it);

last_usage_time_.emplace(time_, para_id);

it->second.last_used = time_;

}

BOOST_ASSERT(it != instance_cache_.end());

return it->second.instance;

}

void PvfRuntimeCache::cleanup(const std::unique_lock<std::mutex> &lock) {

BOOST_ASSERT(lock.owns_lock());

BOOST_ASSERT(lock.mutex() == &instance_cache_mutex_);

for (auto it = last_usage_time_.begin();

it != last_usage_time_.end()

&& last_usage_time_.size() > instances_limit_;) {

instance_cache_.erase(it->second);

it = last_usage_time_.erase(it);

}

}

void PvfRuntimeCache::erase(decltype(instance_cache_)::iterator it) {

// instance can be used at this point, so we wait for exclusive access

// and then extract it from the map (we cannot erase it from inside

// the exclusive access callback because destroying a locked mutex is

// UB)

it->second.instance.exclusiveAccess(

[this, it](auto) {

last_usage_time_.erase(it->second.last_used);

return instance_cache_.extract(it);

});

}

PvfRuntimeCache::requestInstance(const common::Hash256 &code_hash,

const ParachainRuntime &code_zstd) {

std::unique_lock lock{instance_cache_mutex_};

std::shared_ptr<runtime::ModuleInstance> instance;

if (auto entry = instance_cache_.get(code_hash)) {

if (entry.instances.empty()) {

BOOST_OUTCOME_TRY(instance, entry.module->instantiate());

} else {

instance = entry->instances.back();

entry->instances.pop_back();

}

} else {

ParachainRuntime code;

OUTCOME_TRY(runtime::uncompressCodeIfNeeded(code_zstd, code));

OUTCOME_TRY(runtime_module, module_factory_->make(code));

BOOST_OUTCOME_TRY(instance, runtime_module->instantiate());

cache_.put(code_hash, {module, {}});

}

return std::make_shared<runtime::BorrowedInstance>(weak_from_this(), instance);

}

void PvfRuntimeCache::pool(std::shared_ptr<runtime::ModuleInstance> instance) {

std::unique_lock lock{instance_cache_mutex_};

auto module = instance->getModule();

auto hash = module->getHash();

if (auto entry = instance_cache_.get(hash)) {

entry->instances.emplace_back(instance);

} else {

instance_cache_.put(hash, {module, {instance}});

}

}

…instance-cache

…achain-runtime-instance-cache

…:soramitsu/kagome into feature/parachain-runtime-instance-cache

Signed-off-by: Ruslan Tushov <[email protected]>

…:soramitsu/kagome into feature/parachain-runtime-instance-cache

turuslan · 2023-08-17T10:27:42Z

core/runtime/runtime_api/impl/core.cpp

  outcome::result<primitives::Version> CoreImpl::version(
-      RuntimeEnvironment &env) {
-    return executor_->call<primitives::Version>(env, "Core_version");
+      std::shared_ptr<ModuleInstance> instance) {
+    OUTCOME_TRY(genesis_hash, header_repo_->getHashByNumber(0));
+    OUTCOME_TRY(genesis_header, header_repo_->getBlockHeader(genesis_hash));
+    OUTCOME_TRY(ctx,
+                ctx_factory_->ephemeral(instance, genesis_header.state_root));
+    return executor_->decodedCallWithCtx<primitives::Version>(ctx,
+                                                              "Core_version");
  }


Remove this method, if only used by calculate_genesis_state, or remove ambigous argument

turuslan · 2023-08-17T10:32:17Z

core/parachain/pvf/pvf_runtime_cache.hpp

+  class PvfRuntimeCache {
+   public:
+    using SafeInstance = SafeObject<std::shared_ptr<runtime::ModuleInstance>>;
+    using SafeInstanceRef = std::reference_wrapper<SafeInstance>;


SafeInstanceRef will become invalid when lru evicts it

* Add cache for runtime instances in pvf_impl * Refactor runtime environment factory * Executor refactoring for testability Co-authored-by: Ruslan Tushov <[email protected]> Co-authored-by: Ruslan Tushov <[email protected]>

Harrm added 19 commits June 28, 2023 11:21

Add cache for runtime instances in pvf_impl

8c43770

Fix for instance compilation in parachains

1cd1a6f

Return back instance creation

56238b5

Refactor runtime environment factory

766c179

Merge with master

935505c

Refactor runtime

4d3eeec

Finish refactoring, add mutex to the cache

6d2215b

Working on parachain instance cache

3f2874a

Parachain cache and memory limits

2500521

Executor refactoring for testability in progress

82c4c14

Parachain instance cache and memory limits

f71c5d3

Merge branch 'master' of github.com:soramitsu/kagome into feature/par…

177f2fd

…achain-runtime-instance-cache

Fixing tests

2bcdb81

Fix tests

eaf22b1

Merge with master

1891e58

Fixing tests

2afb24c

Fixing runtime tests

018c7c5

Fix tests

2da902f

Fix memory_page_limit_

4e518b2

Harrm requested review from turuslan and xDimon July 24, 2023 08:18

Harrm marked this pull request as ready for review July 24, 2023 08:18

Merge with master

c7bc6ac

turuslan reviewed Jul 26, 2023

View reviewed changes

igor-egorov requested review from igor-egorov and removed request for xDimon July 27, 2023 08:43

turuslan reviewed Jul 28, 2023

View reviewed changes

turuslan requested changes Jul 31, 2023

View reviewed changes

Merge with master

3925881

igor-egorov approved these changes Aug 7, 2023

View reviewed changes

Harrm and others added 12 commits August 14, 2023 18:00

Merge remote-tracking branch 'origin' into feature/parachain-runtime-…

8c7e661

…instance-cache

Merge branch 'master' into feature/parachain-runtime-instance-cache

004fec0

Merge branch 'master' of github.com:soramitsu/kagome into feature/par…

ccd9a8c

…achain-runtime-instance-cache

Merge branch 'feature/parachain-runtime-instance-cache' of github.com…

6a36e35

…:soramitsu/kagome into feature/parachain-runtime-instance-cache

Fixes from review

7766497

Merge with master

f7d351e

parent hash

87361ba

Signed-off-by: Ruslan Tushov <[email protected]>

lru

138aaa4

Signed-off-by: Ruslan Tushov <[email protected]>

cache

c34a67b

Signed-off-by: Ruslan Tushov <[email protected]>

shared_ptr

75e0987

Signed-off-by: Ruslan Tushov <[email protected]>

Merge branch 'master' into feature/parachain-runtime-instance-cache

c845891

Merge branch 'feature/parachain-runtime-instance-cache' of github.com…

cb73ee0

…:soramitsu/kagome into feature/parachain-runtime-instance-cache

turuslan approved these changes Aug 21, 2023

View reviewed changes

Harrm added 4 commits August 21, 2023 11:24

Merge branch 'master' into feature/parachain-runtime-instance-cache

7e243c8

Merge branch 'master' into feature/parachain-runtime-instance-cache

f177e61

Fix typo in parachain test cmakelists

3625a3e

Update CMakeLists.txt

3a1bbca

Harrm merged commit 153a078 into master Aug 21, 2023

Harrm deleted the feature/parachain-runtime-instance-cache branch August 21, 2023 14:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parachain runtime instance cache #1687

Parachain runtime instance cache #1687

Harrm commented Jul 11, 2023 •

edited

Loading

codecov bot commented Jul 24, 2023 •

edited

Loading

turuslan Jul 26, 2023

Harrm Aug 14, 2023

turuslan Jul 26, 2023

Harrm Jul 26, 2023

turuslan Jul 26, 2023

turuslan Jul 28, 2023

Harrm Aug 15, 2023

turuslan Aug 15, 2023

turuslan Jul 31, 2023

turuslan Jul 31, 2023

turuslan Jul 31, 2023

turuslan Jul 31, 2023

turuslan Aug 17, 2023

turuslan Aug 17, 2023

	auto new_size = offset_ + chunk_sz;
	auto new_size = new_pages_num * kMemoryPageSize;

	OUTCOME_TRY(
	parent_hash,
	block_header_repository_->getHashByNumber(params.relay_parent_number));

		using SafeInstance = SafeObject<std::shared_ptr<runtime::ModuleInstance>>;
		using SafeInstanceRef = std::reference_wrapper<SafeInstance>;

Parachain runtime instance cache #1687

Parachain runtime instance cache #1687

Conversation

Harrm commented Jul 11, 2023 • edited Loading

Referenced issues

Description of the Change

Benefits

Possible Drawbacks

codecov bot commented Jul 24, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Harrm commented Jul 11, 2023 •

edited

Loading

codecov bot commented Jul 24, 2023 •

edited

Loading