WASI journal and stateful persistence

docs/snapshots.md

@@ -0,0 +1,105 @@
+# WASM Snapshot Functionality
+
+Wasmer now supports snapshots of the current running process into a journal
+log file which allows for the resumption from an earlier point in time.
+
+# Triggers
+
+Various triggers are possible that will cause a snapshot to be taken at
+a specific point in time, these are:
+
+## On Idle
+
+Triggered when all the threads in the process goes idle.
+
+## On Listen
+
+Triggered when a listen syscall is invoked on a socket.
+
+## On Stdin
+
+Triggered when the process reads stdin for the first time
+
+## On Timer
+
+Triggered periodically based on a timer (default 10 seconds) which can be specified using the `snapshot-timer` option
+
+## On Sigint (Ctrl+C)
+
+Issued if the user sends an interrupt signal (Ctrl + C).
+
+## On Sigalrm
+
+Alarm clock signal (used for timers)
+(see `man alarm`)
+
+## On Sigtstp
+
+The SIGTSTP signal is sent to a process by its controlling terminal to request it to stop temporarily. It is commonly initiated by the user pressing Ctrl-Z.
+
+# On Sigstop
+
+The SIGSTOP signal instructs the operating system to stop a process for later resumption
+
+# On Non Deterministic Call
+
+When a non-determinstic call is made from WASM
+
+# Limitations
+
+- The WASM process must have had the `asyncify` post processing step applied to the binary.
+- Taking a snapshot can consume large amounts of memory while its processing.
+- Snapshots are not instant and have overhead when generating.
+- The layout of the memory must be known by the runtime in order to take snapshots.
+
+# Design
+
+On startup if the restore snapshot file is specified then the runtime will restore the
+state of the WASM process by reading and processing the log entries in the snapshot
+journal. This restoration will bring the memory and the thread stacks back to a previous
+point in time and then resume all the threads.
+
+When a trigger occurs a new snapshot will be taken of the WASM process which will
+take the following steps:
+
+1. Pause all threads
+2. Capture the stack of each thread
+3. Write the thread state to the journal
+4. Write the memory (excluding stacks) to the journal
+5. Resume execution.
+
+The implementation is currently able to save and restore the following:
+
+- WASM Memory
+- Stack memory
+- Call stack
+- Open sockets
+- Open files
+- Terminal text
+
+# Capturer Implementations
+
+## Log File Capturer
+
+Writes the log events to a linear log file on the local file system
+as they are received by the trait. Log files can be concatenated
+together to make larger log files.
+
+## Unsupported Capturer
+
+The default implementation does not support snapshots and will error
+out if an attempt is made to send it events. Using the unsupported
+capturer as a restoration point will restore nothing but will not
+error out.
+
+## Compacting Capturer
+
+Deduplicates memory and stacks to reduce the number of volume of
+log events sent to its inner capturer. Compacting the events occurs
+in line as the events are generated
+
+## Filtered Capturer
+
+Filters out a specific set of log events and drops the rest, this
+capturer can be useful for restoring to a previous call point but
+retaining the memory changes (e.g. WCGI runner).

lib/api/src/externals/memory_view.rs

@@ -2,6 +2,7 @@ use super::memory::{Memory, MemoryBuffer};
 use crate::store::AsStoreRef;
 use crate::MemoryAccessError;
 use std::mem::MaybeUninit;
+use std::ops::Range;
 use wasmer_types::Pages;
 
 #[cfg(feature = "js")]
@@ -145,11 +146,17 @@ impl<'a> MemoryView<'a> {
 
     /// Copies the memory and returns it as a vector of bytes
     pub fn copy_to_vec(&self) -> Result<Vec<u8>, MemoryAccessError> {
+        self.copy_range_to_vec(0..self.data_size())
+    }
+
+    /// Copies a range of the memory and returns it as a vector of bytes
+    pub fn copy_range_to_vec(&self, range: Range<u64>) -> Result<Vec<u8>, MemoryAccessError> {
         let mut new_memory = Vec::new();
-        let mut offset = 0;
+        let mut offset = range.start;
+        let end = range.end.min(self.data_size());
         let mut chunk = [0u8; 40960];
-        while offset < self.data_size() {
-            let remaining = self.data_size() - offset;
+        while offset < end {
+            let remaining = end - offset;
             let sublen = remaining.min(chunk.len() as u64) as usize;
             self.read(offset, &mut chunk[..sublen])?;
             new_memory.extend_from_slice(&chunk[..sublen]);

lib/api/src/js/externals/memory_view.rs

@@ -1,9 +1,9 @@
 use crate::mem_access::MemoryAccessError;
 use crate::store::AsStoreRef;
-use std::convert::TryInto;
 use std::marker::PhantomData;
 use std::mem::MaybeUninit;
 use std::slice;
+use std::{convert::TryInto, ops::Range};
 #[cfg(feature = "tracing")]
 use tracing::warn;
 use wasm_bindgen::JsCast;
@@ -256,11 +256,18 @@ impl<'a> MemoryView<'a> {
     /// Copies the memory and returns it as a vector of bytes
     #[allow(unused)]
     pub fn copy_to_vec(&self) -> Result<Vec<u8>, MemoryAccessError> {
+        self.copy_range_to_vec(0..self.data_size())
+    }
+
+    /// Copies a range of the memory and returns it as a vector of bytes
+    #[allow(unused)]
+    pub fn copy_range_to_vec(&self, range: Range<u64>) -> Result<Vec<u8>, MemoryAccessError> {
         let mut new_memory = Vec::new();
-        let mut offset = 0;
+        let mut offset = range.start;
+        let end = range.end.min(self.data_size());
         let mut chunk = [0u8; 40960];
-        while offset < self.data_size() {
-            let remaining = self.data_size() - offset;
+        while offset < end {
+            let remaining = end - offset;
             let sublen = remaining.min(chunk.len() as u64) as usize;
             self.read(offset, &mut chunk[..sublen])?;
             new_memory.extend_from_slice(&chunk[..sublen]);

lib/api/src/jsc/externals/memory_view.rs

@@ -2,10 +2,10 @@ use crate::store::AsStoreRef;
 use crate::MemoryAccessError;
 use rusty_jsc::JSObject;
 use std::convert::TryFrom;
-use std::convert::TryInto;
 use std::marker::PhantomData;
 use std::mem::MaybeUninit;
 use std::slice;
+use std::{convert::TryInto, ops::Range};
 use wasmer_types::{Bytes, Pages};
 
 use super::memory::{Memory, MemoryBuffer};
@@ -178,11 +178,18 @@ impl<'a> MemoryView<'a> {
     #[allow(unused)]
     /// Copies the memory and returns it as a vector of bytes
     pub fn copy_to_vec(&self) -> Result<Vec<u8>, MemoryAccessError> {
+        self.copy_range_to_vec(0..self.data_size())
+    }
+
+    #[allow(unused)]
+    /// Copies a range of the memory and returns it as a vector of bytes
+    pub fn copy_range_to_vec(&self, range: Range<u64>) -> Result<Vec<u8>, MemoryAccessError> {
         let mut new_memory = Vec::new();
-        let mut offset = 0;
+        let mut offset = range.start;
+        let end = range.end.min(self.data_size());
         let mut chunk = [0u8; 40960];
-        while offset < self.data_size() {
-            let remaining = self.data_size() - offset;
+        while offset < end {
+            let remaining = end - offset;
             let sublen = remaining.min(chunk.len() as u64) as usize;
             self.read(offset, &mut chunk[..sublen])?;
             new_memory.extend_from_slice(&chunk[..sublen]);

lib/api/src/sys/externals/memory_view.rs

@@ -1,9 +1,9 @@
 use crate::store::AsStoreRef;
 use crate::MemoryAccessError;
-use std::convert::TryInto;
 use std::marker::PhantomData;
 use std::mem::MaybeUninit;
 use std::slice;
+use std::{convert::TryInto, ops::Range};
 use wasmer_types::Pages;
 use wasmer_vm::LinearMemory;
 
@@ -163,11 +163,18 @@ impl<'a> MemoryView<'a> {
     #[allow(unused)]
     /// Copies the memory and returns it as a vector of bytes
     pub fn copy_to_vec(&self) -> Result<Vec<u8>, MemoryAccessError> {
+        self.copy_range_to_vec(0..self.data_size())
+    }
+
+    #[allow(unused)]
+    /// Copies a range of the memory and returns it as a vector of bytes
+    pub fn copy_range_to_vec(&self, range: Range<u64>) -> Result<Vec<u8>, MemoryAccessError> {
         let mut new_memory = Vec::new();
-        let mut offset = 0;
+        let mut offset = range.start;
+        let end = range.end.min(self.data_size());
         let mut chunk = [0u8; 40960];
-        while offset < self.data_size() {
-            let remaining = self.data_size() - offset;
+        while offset < end {
+            let remaining = end - offset;
             let sublen = remaining.min(chunk.len() as u64) as usize;
             self.read(offset, &mut chunk[..sublen])?;
             new_memory.extend_from_slice(&chunk[..sublen]);

lib/cli/Cargo.toml

@@ -160,9 +160,11 @@ default = [
   "wat",
   "wast",
   "compiler",
+  "snapshot",
   "wasmer-artifact-create",
   "static-artifact-create",
 ]
+snapshot = ["wasmer-wasix/snapshot"]
 backend = []
 coredump = ["wasm-coredump-builder"]
 sys = ["compiler", "wasmer-vm"]

lib/cli/src/commands/run/wasi.rs

@@ -1,6 +1,7 @@
 use std::{
     collections::{BTreeSet, HashMap},
     path::{Path, PathBuf},
+    str::FromStr,
     sync::{mpsc::Sender, Arc},
     time::Duration,
 };
@@ -33,6 +34,7 @@ use wasmer_wasix::{
             VirtualTaskManagerExt,
         },
     },
+    snapshot,
     types::__WASI_STDIN_FILENO,
     wasmer_wasix_types::wasi::Errno,
     PluggableRuntime, RewindState, Runtime, WasiEnv, WasiEnvBuilder, WasiError, WasiFunctionEnv,
@@ -105,6 +107,35 @@ pub struct Wasi {
     #[clap(long = "enable-async-threads")]
     pub enable_async_threads: bool,
 
+    /// Specifies the snapshot file that Wasmer will use to store
+    /// the state of the WASM process so that it can be later restored
+    #[cfg(feature = "snapshot")]
+    #[clap(long = "snapshot-to")]
+    pub snapshot_to: Option<PathBuf>,
+
+    /// Indicates what events will cause a snapshot to be taken
+    /// and written to the snapshot file.
+    ///
+    /// If not specified, the default is to snapshot on idle plus if a
+    /// snapshot period is provided it will also default to periodic snapshots
+    /// as well.
+    #[cfg(feature = "snapshot")]
+    #[clap(long = "snapshot-on")]
+    pub snapshot_on: Vec<SnapshotTrigger>,
+
+    /// Adds a timer (measured in seconds) that takes snapshots of the process and dumps the
+    /// journal of events to the snapshot file. When specifying this parameter it implies
+    /// that `--snapshot-on timer` has also been specified.
+    #[cfg(feature = "snapshot")]
+    #[clap(long = "snapshot-timer")]
+    pub snapshot_timer: Option<u64>,
+
+    /// When specified, the runtime will restore a previous snapshot
+    /// using the supplied file.
+    #[cfg(feature = "snapshot")]
+    #[clap(long = "resume-from")]
+    pub resume_from: Option<PathBuf>,
+
     /// Allow instances to send http requests.
     ///
     /// Access to domains is granted by default.
@@ -116,6 +147,50 @@ pub struct Wasi {
     pub deny_multiple_wasi_versions: bool,
 }
 
+/// Various triggers that will cause the runtime to take snapshot
+/// of the WASM state and store it in the snapshot file.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub enum SnapshotTrigger {
+    /// Triggered when all the threads in the process goes idle
+    Idle,
+    /// Triggered when a listen syscall is invoked on a socket
+    Listen,
+    /// Triggered when the process reads stdin for the first time
+    Stdin,
+    /// Triggered periodically based on a timer (default 10 seconds) which can be specified using the `snapshot-timer` option
+    Timer,
+    /// Issued if the user sends an interrupt signal (Ctrl + C).
+    Sigint,
+    /// Alarm clock signal (used for timers)
+    Sigalrm,
+    /// The SIGTSTP signal is sent to a process by its controlling terminal to request it to stop temporarily. It is commonly initiated by the user pressing Ctrl-Z.
+    Sigtstp,
+    /// The SIGSTOP signal instructs the operating system to stop a process for later resumption.
+    Sigstop,
+    /// When a non-determinstic call is made
+    NonDeterministicCall,
+}
+
+impl FromStr for SnapshotTrigger {
+    type Err = anyhow::Error;
+
+    fn from_str(s: &str) -> std::result::Result<Self, Self::Err> {
+        let s = s.trim().to_lowercase();
+        Ok(match s.as_str() {
+            "idle" => Self::Idle,
+            "listen" => Self::Listen,
+            "stdin" => Self::Stdin,
+            "periodic" => Self::Timer,
+            "intr" | "sigint" | "ctrlc" | "ctrl-c" => Self::Sigint,
+            "alarm" | "timer" | "sigalrm" => Self::Sigalrm,
+            "sigtstp" | "ctrlz" | "ctrl-z" => Self::Sigtstp,
+            "stop" | "sigstop" => Self::Sigstop,
+            "non-deterministic-call" => Self::NonDeterministicCall,
+            a => return Err(anyhow::format_err!("invalid or unknown trigger ({a})")),
+        })
+    }
+}
+
 pub struct RunProperties {
     pub ctx: WasiFunctionEnv,
     pub path: PathBuf,
@@ -267,6 +342,11 @@ impl Wasi {
             rt.set_networking_implementation(virtual_net::UnsupportedVirtualNetworking::default());
         }
 
+        #[cfg(feature = "snapshot")]
+        if let Some(path) = &self.resume_from {
+            rt.set_snapshot_capturer(Arc::new(snapshot::LogFileSnapshotCapturer::new_std(path)?));
+        }
+
         if !self.no_tty {
             let tty = Arc::new(SysTty::default());
             tty.reset();