safrole traces: RAW kv state + bootstrap service implementation (#12)

1. safrole - traces: RAW kv state + state_snapshots: dev-friendly C1-C15 state

To support JAM implementers to derive each others state roots, extending to service related storage keys, here we publish a keyvals state trace format with a full range of 0x00..00 to 0xFF..FF in `traces` while leaving developer-friendly `state_snapshots` that focus on the C1-C15.  With both state dumps, we can have teams collaborate:
* deriving the stateroot from key-value pair combinations (which match the blocks parent state root) in `traces`
* making sure values in `traces` correspond to the JAM codec encoding of the data in `state_snapshots`

2. Implementation of bootstrap service code

* `main.rs` contains an implementation of the above bootstrap service, which is included in the [genesis state](../../../traces/safrole/genesis.json)
   - The `refine` service expects a `codehash` (32 bytes) starting at `0xFEFF0004`
   - The `accumulate` does a `new` and then a `write`
* `bootstrapblob.pvm` has the disassembleable code (see below `polkatool`) with "magic bytes"
* `bootstrap.pvm` has the JAM-ready code blob (no magic bytes).  We used `polkatool` to do this, with [PR #190](paritytech/polkavm#190) actually mapping `main.rs` into the above.  See links: [Building
 JAM Services in Rust](https://forum.polkadot.network/t/building-jam-services-in-rust/10161) for background


Our test does the following:
* submits a few service code blobs (fib, trib, megatron) in 3 work packages using the bootstrap service "0"
* for each work package, we retrieve the new service indexes left by the `write` after "rho" is cleared (the service has been accumulated)
* our test then runs work packages through fib + trib.  Our next goal is to do ordered accumulation with queues.

Author: sourabhniyogi

services/jam-duna/bootstrap/README.md

@@ -12,43 +12,75 @@
 5. Validators (who will author a block _soon_ in a smart implementation, or all validators in a _lazy_ implementation) then request the actual preimage from N6 with **[CE143: Preimage Request](https://github.com/zdave-parity/jam-np/blob/main/simple.md#ce-143-preimage-request)** with the Preimage Hash received previously and receive the preimage from N6, checking against the Hash, and if it matches, forms (42, 242 bytes of code hashing to the preimage `0x1234...4321`).  This will be aggregated with other preimages and included in ${\bf E}_P$.   Alice can check new blocks as they are proposed / finalized for inclusion of her preimage and/or look in storage $C(255, 42)$ for the newly created service.
 6. Alice's service is available for everyone to submit work packages against at the new serviceID (e.g. 42).  
 
-### Sketch of bootstrap service code
+### Implementation of bootstrap service code
 
-This is a sketch of what the genesis bootstrap service described above.  This has not been tested yet but we believe this can be executed against.  
+* `main.rs` contains an implementation of the above bootstrap service, which is included in the [genesis state](../../../traces/safrole/genesis.json)
+   - The `refine` service expects a `codehash` (32 bytes) starting at `0xFEFF0004`
+   - The `accumulate` does a `new` and then a `write`
+* `bootstrapblob.pvm` has the disassembleable code (see below `polkatool`) with "magic bytes"
+* `bootstrap.pvm` has the JAM-ready code blob (no magic bytes).  We used `polkatool` to do this, with [PR #190](https://github.com/paritytech/polkavm/pull/190) actually mapping `main.rs` into the above.  See links: [Building JAM Services in Rust](https://forum.polkadot.network/t/building-jam-services-in-rust/10161) for background
 
-Notes:
-* `refine` has `codehash` expecting 32 byte payload starting at `0xFEFF0004` because there is 4 bytes of 0 right before the payload ${\bf y}$
-* `accumulate` does a `new` and then a `write`
+
+Our test does the following:
+* submits a few service code blobs (fib, trib, megatron) in 3 work packages using the bootstrap service "0"
+* for each work package, we retrieve the new service indexes left by the `write` after "rho" is cleared (the service has been accumulated)
+* our test then runs work packages through fib + trib.  Our next goal is to do ordered accumulation with queues.
 
 
 ```
-pub @refine:
-    a10 = 0xFEFF0004 // preceded by $s$
-    a11 = 32         // output is 32 bytes, containing the code hash eg 0x1234...4321
-    trap
-
-pub @accumulate:
-    a7 = 0xFEFF0000 // input is the output of @refine
-    a8 = 0          // Figure out sane values of this
-    a9 = 0x2000     
-    a10 = 0x1000
-    a11 = 0x3000
-    a12 = 0x4000
-    ecalli 9        // new will output a new service key to a7
-
-    // write storage key (0) with new service key value copied from a7 into 0xFEFE0004
-    u32 [0xFEFE0000] = 0    // key
-    u32 [0xFEFE0004] = a7   // value
-    a7 = 0xFEFE0000         // key is 0
-    a8 = 4                  // key is 4 bytes
-    a9 = 0xFEFE0004         // value is the new serviceID
-    a10 = 4                 // value is 4 bytes
-    ecalli 3         
-    trap
-
-pub @authorization:
-    fallthrough
-
-pub @on_transfer:
-    fallthrough
+# cargo run -p polkatool disassemble ~/go/src/github.com/jam-duna/jamtestnet/services/jam-duna/bootstrap/bootstrapblob.pvm  --show-raw-bytes
+warning: /root/go/src/github.com/colorfulnotion/polkavm/Cargo.toml: unused manifest key: workspace.lints.rust.unexpected_cfgs.check-cfg
+    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
+     Running `target/debug/polkatool disassemble /root/go/src/github.com/jam-duna/jamtestnet/services/jam-duna/bootstrap/bootstrapblob.pvm --show-raw-bytes`
+// RO data = 0/0 bytes
+// RW data = 0/0 bytes
+// Stack size = 4096 bytes
+// Jump table entry point size = 0 bytes
+// RO data = []
+// RW data = []
+// Instructions = 34
+// Code size = 115 bytes
+
+      :                          @0
+     0: 05 11 00 00 00           jump @4
+      :                          @1
+     5: 05 10 00 00 00           jump @5
+      :                          @2
+    10: 05 18 00 00 00           jump @6
+      :                          @3
+    15: 05 60                    jump @7
+      :                          @4 [export #0: 'is_authorized']
+    17: 04 07                    a0 = 0x0
+    19: 13 00                    ret
+      :                          @5 [export #1: 'refine']
+    21: 04 0a 04 00 ff fe        a3 = 0xfeff0004
+    27: 04 0b 20                 a4 = 0x20
+    30: 04 07                    a0 = 0x0
+    32: 13 00                    ret
+      :                          @6 [export #2: 'accumulate']
+    34: 02 11 f8                 sp = sp - 8
+    37: 03 10 04                 u32 [sp + 4] = ra
+    40: 03 15                    u32 [sp] = s0
+    42: 04 05 00 00 ff fe        s0 = 0xfeff0000
+    48: 04 07 00 00 ff fe        a0 = 0xfeff0000
+    54: 04 08 ce 00              a1 = 0xce
+    58: 04 09 00 20              a2 = 0x2000
+    62: 04 0a 00 10              a3 = 0x1000
+    66: 04 0b 00 30              a4 = 0x3000
+    70: 04 0c 00 40              a5 = 0x4000
+    74: 4e 09                    ecalli 9 // 'new'
+    76: 0d 05                    u32 [s0 + 0] = 0
+    78: 03 57 04                 u32 [s0 + 4] = a0
+    81: 04 09 04 00 ff fe        a2 = 0xfeff0004
+    87: 04 07 00 00 ff fe        a0 = 0xfeff0000
+    93: 04 08 04                 a1 = 0x4
+    96: 04 0a 04                 a3 = 0x4
+    99: 4e 03                    ecalli 3 // 'write'
+   101: 01 10 04                 ra = u32 [sp + 4]
+   104: 01 15                    s0 = u32 [sp]
+   106: 02 11 08                 sp = sp + 0x8
+   109: 13 00                    ret
+      :                          @7 [export #3: 'on_transfer']
+   111: 04 07                    a0 = 0x0
+   113: 13 00                    ret
 ```

services/jam-duna/bootstrap/bootstrap.pvm

@@ -0,0 +1,51 @@
+// RO data = 0/0 bytes
+// RW data = 0/0 bytes
+// Stack size = 4096 bytes
+// Jump table entry point size = 0 bytes
+// RO data = []
+// RW data = []
+// Instructions = 34
+// Code size = 115 bytes
+
+      :                          @0
+     0: 05 11 00 00 00           jump @4
+      :                          @1
+     5: 05 10 00 00 00           jump @5
+      :                          @2
+    10: 05 18 00 00 00           jump @6
+      :                          @3
+    15: 05 60                    jump @7
+      :                          @4 [export #0: 'is_authorized']
+    17: 04 07                    a0 = 0x0
+    19: 13 00                    ret
+      :                          @5 [export #1: 'refine']
+    21: 04 0a 04 00 ff fe        a3 = 0xfeff0004
+    27: 04 0b 20                 a4 = 0x20
+    30: 04 07                    a0 = 0x0
+    32: 13 00                    ret
+      :                          @6 [export #2: 'accumulate']
+    34: 02 11 f8                 sp = sp - 8
+    37: 03 10 04                 u32 [sp + 4] = ra
+    40: 03 15                    u32 [sp] = s0
+    42: 04 05 00 00 ff fe        s0 = 0xfeff0000
+    48: 04 07 00 00 ff fe        a0 = 0xfeff0000
+    54: 04 08 ce 00              a1 = 0xce
+    58: 04 09 00 20              a2 = 0x2000
+    62: 04 0a 00 10              a3 = 0x1000
+    66: 04 0b 00 30              a4 = 0x3000
+    70: 04 0c 00 40              a5 = 0x4000
+    74: 4e 09                    ecalli 9 // 'new'
+    76: 0d 05                    u32 [s0 + 0] = 0
+    78: 03 57 04                 u32 [s0 + 4] = a0
+    81: 04 09 04 00 ff fe        a2 = 0xfeff0004
+    87: 04 07 00 00 ff fe        a0 = 0xfeff0000
+    93: 04 08 04                 a1 = 0x4
+    96: 04 0a 04                 a3 = 0x4
+    99: 4e 03                    ecalli 3 // 'write'
+   101: 01 10 04                 ra = u32 [sp + 4]
+   104: 01 15                    s0 = u32 [sp]
+   106: 02 11 08                 sp = sp + 0x8
+   109: 13 00                    ret
+      :                          @7 [export #3: 'on_transfer']
+   111: 04 07                    a0 = 0x0
+   113: 13 00                    ret

services/jam-duna/bootstrap/bootstrap.txt

@@ -0,0 +1,139 @@
+#![no_std]
+#![no_main]
+
+extern crate alloc;
+
+use alloc::vec::Vec;
+use simplealloc::SimpleAlloc;
+
+#[global_allocator]
+static ALLOCATOR: SimpleAlloc<4096> = SimpleAlloc::new();
+
+#[panic_handler]
+fn panic(_info: &core::panic::PanicInfo) -> ! {
+    unsafe {
+        core::arch::asm!("unimp", options(noreturn));
+    }
+}
+
+#[polkavm_derive::polkavm_import]
+extern "C" {
+    #[polkavm_import(index = 0)]
+    pub fn gas() -> i64;
+    #[polkavm_import(index = 1)]
+    pub fn lookup(service: u32, hash_ptr: *const u8, out: *mut u8, out_len: u32) -> u32;
+    #[polkavm_import(index = 2)]
+    pub fn read(service: u32, key_ptr: *const u8, key_len: u32, out: *mut u8, out_len: u32) -> u32;
+
+
+    // #[polkavm_import(index = 3)]
+    // pub fn write(key_ptr: *const u8, key_len: u32, value: *const u8, value_len: u32) -> u32;
+
+    #[polkavm_import(index = 3)]
+    pub fn write(ko: u32, kz: u32, bo: u32, bz: u32) -> u32;
+
+
+    #[polkavm_import(index = 4)]
+    pub fn info(service: u32, out: *mut u8) -> u32;
+    #[polkavm_import(index = 5)]
+    pub fn empower(m: u32, a: u32, v: u32, o: u32, n: u32) -> u32;
+    #[polkavm_import(index = 6)]
+    pub fn assign(c: u32, out: *mut u8) -> u32;
+    #[polkavm_import(index = 7)]
+    pub fn designate(out: *mut u8) -> u32;
+    #[polkavm_import(index = 8)]
+    pub fn checkpoint() -> u64;
+
+    // #[polkavm_import(index = 9)]
+    // pub fn new(service: u32, hash_ptr: *const u8, out: *mut u8, out_len: u32) -> u32;
+
+    #[polkavm_import(index = 9)]
+    pub fn new(o: u32, l: u32, gl: u32, gh: u32, ml: u32, mh: u32) -> u32;
+
+
+
+    #[polkavm_import(index = 10)]
+    pub fn upgrade(out: *const u8, g: u64, m: u64) -> u32;
+    #[polkavm_import(index = 11)]
+    pub fn transfer(d: u32, a: u64, g: u64, out: *mut u8) -> u32;
+    #[polkavm_import(index = 12)]
+    pub fn quit(d: u32, a: u64, g: u64, out: *mut u8) -> u32;
+    #[polkavm_import(index = 13)]
+    pub fn solicit(hash_ptr: *const u8, z: u32) -> u32;
+    #[polkavm_import(index = 14)]
+    pub fn forget(hash_ptr: *const u8, z: u32) -> u32;
+    #[polkavm_import(index = 15)]
+    pub fn historical_lookup(service: u32, hash_ptr: *const u8, out: *mut u8, out_len: u32) -> u32;
+    #[polkavm_import(index = 16)]
+    pub fn import(import_index: u32, out: *mut u8, out_len: u32) -> u32;
+    #[polkavm_import(index = 17)]
+    pub fn export(out: *const u8, out_len: u32) -> u32;
+    #[polkavm_import(index = 18)]
+    pub fn machine(out: *const u8, out_len: u32) -> u32;
+    #[polkavm_import(index = 19)]
+    pub fn peek(out: *const u8, out_len: u32, i: u32) -> u32;
+    #[polkavm_import(index = 20)]
+    pub fn poke(n: u32, a: u32, b: u32, l: u32) -> u32;
+    #[polkavm_import(index = 21)]
+    pub fn invoke(n: u32, out: *mut u8) -> u32;
+    #[polkavm_import(index = 22)]
+    pub fn expunge(n: u32) -> u32;
+    #[polkavm_import(index = 99)]
+    pub fn blake2b(data: *const u8, data_len: u32, hash_ptr: *mut u8) -> u32;
+    #[polkavm_import(index = 100)]
+    pub fn blake2s(data: *const u8, data_len: u32, hash_ptr: *mut u8) -> u32;
+    #[polkavm_import(index = 101)]
+    pub fn ecrecover(h: *const u8, v: *const u8, r: *const u8, s: *const u8, out: *mut u8) -> u32;
+    #[polkavm_import(index = 102)]
+    pub fn sha2_256(data: *const u8, data_len: u32, hash_ptr: *mut u8) -> u32;
+}
+
+#[polkavm_derive::polkavm_export]
+extern "C" fn is_authorized() -> u32 {
+    0
+}
+
+#[polkavm_derive::polkavm_export]
+extern "C" fn refine() -> u32 {
+    unsafe {
+        core::arch::asm!(
+            "li a3, 0xFEFF0004",
+            "li a4, 0x20",
+        );
+    }
+    0
+}
+
+#[polkavm_derive::polkavm_export]
+extern "C" fn accumulate() -> u32 {
+    let mut omega_7: u32 = 0xFEFF0000;
+    let omega_8: u32 = 206;
+    let omega_9: u32 = 0x2000;
+    let omega_10: u32 = 0x1000;
+    let omega_11: u32 = 0x3000;
+    let omega_12: u32 = 0x4000;
+
+    let result = unsafe { new(omega_7, omega_8, omega_9, omega_10, omega_11, omega_12) };
+
+    unsafe {
+        let ptr1 = 0xFEFF0000 as *mut u32;
+        *ptr1 = 0;
+
+        let ptr2 = 0xFEFF0004 as *mut u32;
+        *ptr2 = result;
+    }
+
+    let mut omega_7: u32 = 0xFEFF0000;
+    let omega_8: u32 = 4;
+    let omega_9: u32 = 0xFEFF0004;
+    let omega_10: u32 = 4;
+
+    let result = unsafe { write(omega_7, omega_8, omega_9, omega_10) };
+    omega_7 = result;
+    result
+}
+
+#[polkavm_derive::polkavm_export]
+extern "C" fn on_transfer() -> u32 {
+    0
+}