Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DProxy #4462

Merged
merged 76 commits into from
Mar 21, 2024
Merged

DProxy #4462

Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
dd3bdb5
First implementation of DProxy
john-sharratt Jan 3, 2024
3b54706
Merge remote-tracking branch 'origin/master' into dproxy
john-sharratt Jan 4, 2024
1c646e2
Simplifications of DProxy
john-sharratt Jan 4, 2024
09b7556
Fixed an issue where the runtime would not load properly in DProxy
john-sharratt Jan 5, 2024
d82b1dc
Simplified the loopback listen method so that it fixes a multiplexing…
john-sharratt Jan 5, 2024
6575b12
Merge remote-tracking branch 'origin/master' into dproxy
john-sharratt Jan 7, 2024
92ef764
Refactoring of the rewind handling for DProxy
john-sharratt Jan 7, 2024
4f2e4a6
Fixed an issue where the caching compiled modules were not saving pro…
john-sharratt Jan 8, 2024
fbfbcef
Some minor journal optimizations for a restored state
john-sharratt Jan 9, 2024
13203fd
Renamed InstanceSnapshot to StoreSnapshot which better reflects its p…
john-sharratt Jan 9, 2024
bb1b647
No longer partially restoring orphaned snapshots
john-sharratt Jan 10, 2024
6ea2ab2
Added the capability for the file system to index the data from a mma…
john-sharratt Jan 17, 2024
a40d69b
Added unit tests for offload file
john-sharratt Jan 21, 2024
026636d
Fixed some alignment and serialization issues on the journals
john-sharratt Jan 22, 2024
1fab2ad
Fixed an issue with a lock on the file system
john-sharratt Jan 22, 2024
88c984d
Fixes for the journal fd seed
john-sharratt Jan 22, 2024
41528eb
Fixed some compile issues
john-sharratt Jan 22, 2024
020fcac
Finally fixed the journal alignment issues, once and for all
john-sharratt Jan 22, 2024
2fe6edc
Finished a mounting feature for the journals
john-sharratt Jan 22, 2024
5f4d2db
Now using COW on aligned vectors and strings
john-sharratt Jan 22, 2024
173b07f
Significant improvement in write performance
john-sharratt Jan 23, 2024
e12e146
Finished a progress bar for the loading of a journal
john-sharratt Jan 23, 2024
8aa4169
Significant speedup of the extent table loading
john-sharratt Jan 23, 2024
ef2d7dc
Fixed a bug where file sizes could not be set
john-sharratt Jan 23, 2024
66374be
Fixed the IO operations on file system
john-sharratt Jan 25, 2024
17be193
Fixed the loading of mounted file systems with static web servers
john-sharratt Feb 5, 2024
28b9519
Fixed a bug where journals that have incomplete write operations no l…
john-sharratt Feb 5, 2024
00ac9fa
Fixed an issue with the file system transversal on mounting file systems
john-sharratt Feb 5, 2024
b3dca80
Fixed an issue where a terminating instance does not abort the connec…
john-sharratt Feb 5, 2024
45d9809
DProxy now correctly snapshots threads to the journal
john-sharratt Feb 6, 2024
a467536
The system now snapshots when a workload goes idle
john-sharratt Feb 6, 2024
71000db
Optimized the memory snapshots to reduce wastage
john-sharratt Feb 6, 2024
3b2ec3a
The memory snapshot functionality is now much faster using region com…
john-sharratt Feb 6, 2024
967b83a
Added a todo on using dirty pages to optimize the hashing time when s…
john-sharratt Feb 6, 2024
36abb81
Fixed the remote sockets
john-sharratt Feb 7, 2024
aa64c36
Fixed the writing of the memory hashes to the bitmap
john-sharratt Feb 7, 2024
d93a2fd
Added extra data for the threads so they can be more easily recreated
john-sharratt Feb 8, 2024
3eeb86f
Added the functionality that spawns threads
john-sharratt Feb 8, 2024
c187126
Now spawning the sub threads correctly on journal resume
john-sharratt Feb 8, 2024
da21864
Fixed an issue where the main thread was not rewinding properly
john-sharratt Feb 8, 2024
f2dfd5c
Fixed an issue where the threads were not properly rewinding on journals
john-sharratt Feb 8, 2024
354e38c
Failed WASM processes now properly terminate running connections
john-sharratt Feb 8, 2024
d7255c4
Fixed the thread resuming however there is now a tokio runtime problem
john-sharratt Feb 8, 2024
d034c50
Resolved merge conflicts
john-sharratt Feb 13, 2024
1fcf56b
Journal now properly cleans up when a snapshot is missing
john-sharratt Feb 20, 2024
6216634
Fixed the final pieces for a single threaded dproxy
john-sharratt Feb 22, 2024
2cbfb91
Merge remote-tracking branch 'origin/master' into dproxy
john-sharratt Feb 22, 2024
de2606d
Correctly connected up the signal process for the ctrl-c key
john-sharratt Feb 25, 2024
bb430fe
Now taking snapshots on signal events however its not waking properly…
john-sharratt Feb 25, 2024
68fedb8
Fixed an issue where ctrl-c was not triggering during deep threading …
john-sharratt Feb 25, 2024
3757754
Renaming some of the spawn functions to make them easier to understand
john-sharratt Feb 25, 2024
f68f58d
Renamed the journal syscall player
john-sharratt Feb 25, 2024
bd95ef3
Added another struct for RemoteSocket to make it simplier
john-sharratt Feb 25, 2024
9b38ee5
Added a comment on one of the apply thread methods
john-sharratt Feb 25, 2024
049f505
Added comments on VIRUTAL_FD_ROOT
john-sharratt Feb 25, 2024
2184922
Missed one
john-sharratt Feb 25, 2024
bc520d4
Added additional comments
john-sharratt Feb 25, 2024
09672e9
Fixed a whole bunch of linting errors
john-sharratt Feb 25, 2024
d064818
Fixed some compile issues for JS
john-sharratt Feb 26, 2024
192f3e8
Merge remote-tracking branch 'origin/master' into dproxy
john-sharratt Feb 26, 2024
573b342
Linting fix
john-sharratt Feb 26, 2024
502957f
More minor linting fixes
john-sharratt Feb 26, 2024
0b04236
Removed fuse as a default dependency as it does not work on all platf…
john-sharratt Feb 26, 2024
51e2e75
More linting fixes
john-sharratt Feb 26, 2024
fc85d90
More linting fixes
john-sharratt Feb 26, 2024
62b7c72
Fixed some unit tests
john-sharratt Feb 26, 2024
0a6abfd
Fixed some unit tests
john-sharratt Feb 26, 2024
329f972
Fixed one of the lints that was missed
john-sharratt Feb 26, 2024
de476a6
Linting and unit test fixes
john-sharratt Feb 26, 2024
46914b9
Fixed multithreading for journals
john-sharratt Feb 26, 2024
f0290c8
Fixed yet another linting issue
john-sharratt Feb 26, 2024
887e386
Merge branch 'master' into dproxy
john-sharratt Feb 27, 2024
1ee2392
Merge branch 'master' into dproxy
john-sharratt Feb 27, 2024
028a5a8
Exposing WASIX method so that it can be consumed by custom task managers
john-sharratt Feb 27, 2024
03678c9
Merge branch 'master' into dproxy
john-sharratt Mar 11, 2024
c09b105
Merge branch 'master' into dproxy
john-sharratt Mar 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
203 changes: 159 additions & 44 deletions Cargo.lock

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ wasmer-toml = "0.9.2"
wasmparser = { version = "0.121.0", default-features = false }
webc = { version = "5.8.0", default-features = false, features = ["package"] }
shared-buffer = "0.1.4"
rkyv = { version = "0.7.40", features = ["indexmap", "validation", "strict"] }
memmap2 = { version = "0.6.2" }

[build-dependencies]
test-generator = { path = "tests/lib/test-generator" }
Expand Down
11 changes: 11 additions & 0 deletions lib/cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,15 @@ default = [
"wast",
"compiler",
"journal",
"fuse",
"wasmer-artifact-create",
"static-artifact-create",
]

# Tun-tap client for connecting to Wasmer Edge VPNs
tun-tap = ["dep:tun-tap", "virtual-net/tokio-tungstenite", "tokio-tungstenite", "mio", "futures-util", "mac_address", "dep:interfaces"]
journal = ["wasmer-wasix/journal"]
fuse = ["dep:fuse", "dep:time01", "dep:shared-buffer", "dep:rkyv"]
backend = []
coredump = ["wasm-coredump-builder"]
sys = ["compiler", "wasmer-vm"]
Expand Down Expand Up @@ -80,6 +82,7 @@ wasmer-wasix = { version = "0.18.0", path = "../wasix", features = [
"logging",
"webc_runner_rt_wcgi",
"webc_runner_rt_dcgi",
"webc_runner_rt_dproxy",
"webc_runner_rt_emscripten",
"host-fs",
] }
Expand Down Expand Up @@ -108,6 +111,14 @@ wasmer-api = { version = "=0.0.23", path = "../backend-api" }
edge-schema = { version = "=0.0.2" }
edge-util = { version = "=0.0.1" }

# Used by the mount command

shared-buffer = { workspace = true, optional = true }
rkyv = { workspace = true, optional = true }
fuse = { version = "0.3", optional = true }
time01 = { package = "time", version = "0.1.45", optional = true }


# Third-party dependencies.

is-terminal = "0.4.7"
Expand Down
4 changes: 2 additions & 2 deletions lib/cli/src/commands/journal/import.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ use crate::commands::CliCommand;
/// Imports events into a journal file. Events are streamed as JSON
/// objects into `stdin`
#[derive(Debug, Parser)]
pub struct CmdJournaImport {
pub struct CmdJournalImport {
/// Path to the journal that will be printed
#[clap(index = 1)]
journal_path: PathBuf,
}

impl CliCommand for CmdJournaImport {
impl CliCommand for CmdJournalImport {
type Output = ();

fn run(self) -> Result<(), anyhow::Error> {
Expand Down
4 changes: 2 additions & 2 deletions lib/cli/src/commands/journal/inspect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ use crate::commands::CliCommand;

/// Prints a summarized version of contents of a journal to stdout
#[derive(Debug, Parser)]
pub struct CmdJournaInspect {
pub struct CmdJournalInspect {
/// Path to the journal that will be printed
#[clap(index = 1)]
journal_path: PathBuf,
}

impl CliCommand for CmdJournaInspect {
impl CliCommand for CmdJournalInspect {
type Output = ();

fn run(self) -> Result<(), anyhow::Error> {
Expand Down
13 changes: 11 additions & 2 deletions lib/cli/src/commands/journal/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,16 @@ mod export;
mod filter;
mod import;
mod inspect;
#[cfg(feature = "fuse")]
mod mount;

pub use compact::*;
pub use export::*;
pub use filter::*;
pub use import::*;
pub use inspect::*;
#[cfg(feature = "fuse")]
pub use mount::*;

/// Manage Journal files.
#[derive(clap::Subcommand, Debug)]
Expand All @@ -20,11 +24,14 @@ pub enum CmdJournal {
/// Exports the contents of a journal to stdout as JSON objects
Export(CmdJournalExport),
/// Imports the events into a journal as JSON objects
Import(CmdJournaImport),
Import(CmdJournalImport),
/// Inspects the contents of a journal and summarizes it to `stdout`
Inspect(CmdJournaInspect),
Inspect(CmdJournalInspect),
/// Filters out certain events from a journal
Filter(CmdJournalFilter),
/// Mounts the journal at a particular directory
#[cfg(feature = "fuse")]
Mount(CmdJournalMount),
}

impl CliCommand for CmdJournal {
Expand All @@ -37,6 +44,8 @@ impl CliCommand for CmdJournal {
Self::Export(cmd) => cmd.run(),
Self::Inspect(cmd) => cmd.run(),
Self::Filter(cmd) => cmd.run(),
#[cfg(feature = "fuse")]
Self::Mount(cmd) => cmd.run(),
}
}
}
42 changes: 42 additions & 0 deletions lib/cli/src/commands/journal/mount/cmd.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
use std::{path::PathBuf, process::Stdio};

use clap::Parser;
use wasmer_wasix::fs::WasiFdSeed;

use super::fs::JournalFileSystemBuilder;
use crate::commands::CliCommand;

/// Mounts a journal as a file system on the local machine
#[derive(Debug, Parser)]
pub struct CmdJournalMount {
/// Path to the journal that will be printed
#[clap(index = 1)]
journal_path: PathBuf,
/// Path to the directory where the file system will be mounted
#[clap(index = 2)]
mount_path: PathBuf,
}

impl CliCommand for CmdJournalMount {
type Output = ();

fn run(self) -> Result<(), anyhow::Error> {
// First we unmount any existing file system on this path
std::process::Command::new("/bin/umount")
.arg(self.mount_path.to_string_lossy().as_ref())
.stderr(Stdio::null())
.stdout(Stdio::null())
.spawn()?
.wait()
.ok();

let fs = JournalFileSystemBuilder::new(&self.journal_path)
.with_fd_seed(WasiFdSeed::default())
.with_progress_bar(false)
.build()?;

// Mounts the journal file system at a path
fuse::mount(fs, &self.mount_path, &[])?;
Ok(())
}
}
Loading
Loading