Metering improvements

[webmaster128]

After migrating to Wasmer 1.0 and adding #668, there are a few smaller improvements we should do at some point:

• Use get_remaining_points/set_remaining_points from Wasmer directly (done in Upgrade to latest Wasmer (🎅 edition) #676)
• Handle out of gas correctly (VmError::RuntimeErr { .. } | VmError::GasDepletion { .. } => {} and #[ignore] // fails due to the inability to handle gas depletion (https://github.com/wasmerio/wasmer/issues/1931), see Allow users to handle metering errors wasmerio/wasmer#1931; done in Upgrade to latest Wasmer (🎅 edition) #676)
• Write specification for gas cost and adapt implementation. Currently we use arbitrary values (Integrate Wasmer Reborn Metering middleware #668 (comment))

[ethanfrey]

Currently we use random values

Maybe you can clarify. They are not random, but rather arbitrary. ie. They are fixed for all instances and runs, but are rather meaningless.

[webmaster128]

@ethanfrey If I do I understand correctly, every operator cost was 1 point flat in Wasmer 0.17: https://github.com/wasmerio/wasmer/blob/0.17.1/lib/middleware-common/src/metering.rs#L43-L113. This means we can trivially get similar or even the same gas results by using a constant 1 in the new cost function.

[ethanfrey]

Yes, it does look quite easy to get the same result.

Let's use this for the 0.13 release and then open a new issue for a more nuanced gas meter (and new sdk-wasm multiplier calculation) in a future release. Clearly adding 2 registers is faster than reading from memory.

[ethanfrey]

Write specification for gas cost and adapt implementation.

We should benchmark different op codes to get a feel for relative cost.

I took a look at NEAR, which uses wasmer in production, to see their gas metering strategy, and it seems like they charge a flat cost:

https://github.com/near/nearcore/blob/5d12069ca305cc781c77d63e3c932e1c11f06ac4/runtime/near-vm-runner/src/prepare.rs#L60-L66

Note they use pwasm_utils::inject_gas_counter so maybe that does more complex. Nope, second argument allows overriding cost for individual ops, otherwise, it just uses the default cost for each

[ethanfrey]

Note, I think this page is a decent introduction to gas pricing and we should do something similar for docs (the high-level aspects): https://docs.near.org/docs/concepts/gas

Notably, they have a huge difference between native token transfer and contract tokens...

Transfer native token (ETH or NEAR): 0.045 mN (milli-Near as fee)
Transfer a fungible token: 1.4 mN

So, around 35x more expensive for the contract (with CosmWasm + Cosmos SDK contracts are about 2x more expensive... maybe cuz the bank and auth modules are so inefficient? Or maybe cuz we have better caching?)

[uint]

I've been mulling this over and the best thing I can come up with is writing our own wasmer middleware called profiling. profiling would time operator executions and produce per-operator averages at the end. This seems a little tricky, but possible.

We could then write a cosmwasm-profiler CLI tool that would take a bunch of our example contracts and do various calls on them over and over with profiling enabled. Eventually it would combine the data and produce a map of gas costs that would roughly translate to our cost function.

Operators that were never profiled would probably get an expensive cost by default (I think?)

Does this sound like a good direction?

[webmaster128]

This sounds good. But isn't a middleware a Wasm -> Wasm mapping, such that we had to implement all profiling logic in Wasm?

[uint]

This sounds good. But isn't a middleware a Wasm -> Wasm mapping, such that we had to implement all profiling logic in Wasm?

Yeah. I don't know much about Wasm, so it's a bit tricky to me. I'm trying to figure out if we can measure time sensibly by injecting the right Wasm code between each (?) operator. I think then it's a matter of keeping some sort of global map of Operator -> Vec<Measurement> in Wasm, and once we have that, we can interpret the data in Rust.

[webmaster128]

Operators that were never profiled would probably get an expensive cost by default (I think?)

I think we can 1. try to include them in some contract, 2. compare with similar operations, 3. look into the spec to dertermine if they are potentially expensive.

[webmaster128]

This is similar to how the metering middleware works. This looks at blocks of code, not individual operations.

We could adapt this approach to call a tracing import every time the metering middleware would do accounting. The import is implemented in the host and has access to a clock.

[uint]

We could adapt this approach to call a tracing import every time the metering middleware would do accounting. The import is implemented in the host and has access to a clock.

Yeah, I studied it. That's a pretty neat idea. Wouldn't we want to be more granular though and do this per instruction? We can try to do some statistical magic with whole sequences of them timed, but I think timing each instruction would be more reliable.

[uint]

Yeah, I studied it. That's a pretty neat idea. Wouldn't we want to be more granular though and do this per instruction? We can try to do some statistical magic with whole sequences of them timed, but I think timing each instruction would be more reliable.

Or is it that most instructions will pretty much be a rounding error compared to calling something on the host?

[webmaster128]

Some thoughts:

1. The call to the host is expensive but pretty much a constant, so it should be possible to subtract it reliably
2. The Wasm operations are compiled into native code doing a little bit of optimization. So I expect code blocks to give more accurate results than individual operations.
3. We might not get the chance to precisely measure the runtime of the fast operations. But we should be able to detect the ones that are significantly more expensive (like 50x or 1000x the cost of an addition). This is probably good enough.

[uint]

Some thoughts:

I feel convinced ;)

[webmaster128]

I am super curious to learn what we get out of this. Maybe it's a dead end and we need to try something completely different. But it's worth a try.

[webmaster128]

Interesting links from NEAR above. The docs look cool. I like the idea of a target gas/millisecond and the high numbers to be able to adjust in both direction.

Here is their config:

• https://github.com/near/nearcore/blob/d95a1edf9f4a3f751876907f8487e6eefa6594ec/core/primitives-core/src/config.rs#L13-L16
• https://github.com/near/nearcore/blob/d95a1edf9f4a3f751876907f8487e6eefa6594ec/nearcore/res/genesis_config.json#L189-L190

The memory growth (per 64KiB page) seems ridiculously cheep such that I am afraid I missed something.

[uint]

Okay, it's probably about time for an update on what I've been up to. My attempts at creating a profiling tool (to assess how much time each op takes and what our cost function should look like) have failed so far, although I have the pieces and a clear idea of how it needs to work. I submitted what I have in a draft PR: #1042. Details are there!

[webmaster128]

After significant work on #1042 it became clear that we currently cannot measure the execution time of individual Wasm operations precise enough to charge different price for them. Also there is a runtime parameter after memory.grow and memory copy operations would be important to read and consider first.

In CosmWasm 1.0 gas price will be 150_000 times as large as in 0.16 and a target gas per second is defined. This should make future adjustment much easier than before.

Done for now. We'll follow up with more specific issues.