Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

epoll with fs fixes #4050

Merged
merged 149 commits into from
Jul 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
149 commits
Select commit Hold shift + click to select a range
0e0dec2
Added asynchronous IO syscalls for WASIX based on generic wakers
john-sharratt Apr 28, 2023
5b10567
Merge branch 'asynchronous-threading-phase2' into asynchronous-io
john-sharratt May 13, 2023
495d8fa
Merge branch 'asynchronous-threading-phase2' into asynchronous-io
john-sharratt May 13, 2023
32a7e5a
Merge branch 'asynchronous-threading-phase2' into asynchronous-io
john-sharratt May 13, 2023
55861a4
No longer supporting 64bit in this version - other things have to be …
john-sharratt May 13, 2023
89bef74
Merge branch 'asynchronous-threading-phase2' into asynchronous-io
john-sharratt May 13, 2023
bb4e518
Merge branch 'asynchronous-threading-phase2' into asynchronous-io
john-sharratt May 16, 2023
e27bbad
Merge branch 'master' into asynchronous-io
john-sharratt May 16, 2023
b0604fa
Merge branch 'master' into asynchronous-io
john-sharratt Jun 17, 2023
815b181
Clearup of the documentation
john-sharratt Jun 17, 2023
f29f50b
Cleanup of some of the logic, looks ready
john-sharratt Jun 17, 2023
b226ea5
Implemented the socket polling methods and fixed the wakers
john-sharratt Jun 20, 2023
ca30bf1
Fix for linting
john-sharratt Jun 20, 2023
5427c5e
Merge branch 'master' into asynchronous-io
john-sharratt Jun 20, 2023
7a54d76
Merge branch 'master' into asynchronous-io
john-sharratt Jun 20, 2023
51f66b6
Fixes for the waker implementation
john-sharratt Jun 20, 2023
97b71ce
Merge branch 'asynchronous-io' of github.com:wasmerio/wasmer into asy…
john-sharratt Jun 20, 2023
8082549
Minor refactoring
john-sharratt Jun 20, 2023
f1d451e
Fixed an issue with runtimes running in runtimes
john-sharratt Jun 20, 2023
cf3a878
Removed the tokio runtime from the Runtime trait - wasm threads now r…
john-sharratt Jun 21, 2023
cc5c83c
Merge branch 'master' into asynchronous-io
john-sharratt Jun 21, 2023
6f30820
Switched to the threading pool
john-sharratt Jun 21, 2023
1f644f8
Removed the shared tokio runtime
john-sharratt Jun 21, 2023
f842403
Added a fix for the reqwest http client
john-sharratt Jun 21, 2023
fd55348
Tokio context is only coniditonally entered thus allowing for reentrance
john-sharratt Jun 21, 2023
97d6909
chore: Remove unused name_map from WasiFs
theduke Jun 13, 2023
faf1802
wasix: Allow passing a custom root fs to WasiRunner/CommonWasiOptions
theduke Jun 13, 2023
2c15588
wasix: Switch the Console over to the WasiRunner
theduke Jun 13, 2023
0a8d91d
chore: remove redundant webc_runner[_wasi] features
theduke Jun 13, 2023
b3ba2fc
feat(wasix): Extend Runtime trait with module loading/compiling
theduke Jun 13, 2023
b59e51c
wasix: use common Runtime::laod_module helper in spawn_exec
theduke Jun 13, 2023
a9087bf
Use async load_module() to prevent tokio nested runtime panic
theduke Jun 14, 2023
6eb85aa
chore: improve logging in bin_factory
theduke Jun 21, 2023
6a0abbb
tests(wasix): Add execution test for Console
theduke Jun 21, 2023
7964c9d
wasix: Improve SpawnError::Other
theduke Jun 21, 2023
b9b896e
cli: Use Runtime::load_module_sync in runners
theduke Jun 21, 2023
96bed05
tests(wasix): Constraint Console test
theduke Jun 22, 2023
3cc1331
chore(wasix): Cleanup of various review suggestions
theduke Jun 22, 2023
f7b91c4
Implemented epoll for asynchronous IO
john-sharratt Jun 23, 2023
fa47c6a
Now passing through the epoll interest events rather than abstracting it
john-sharratt Jun 25, 2023
0d9ec43
Hiding more of the API's for epoll
john-sharratt Jun 25, 2023
59f8acf
Added waker conversion helper
john-sharratt Jun 25, 2023
057e327
Merge branch 'master' into asynchronous-io
john-sharratt Jun 25, 2023
4f4c6fe
Removed extra pending operation
john-sharratt Jun 25, 2023
1884c62
Resolved compilation regression issues
john-sharratt Jun 25, 2023
8e8d4dd
Fixed a regression issue on Stdout runtimes
john-sharratt Jun 25, 2023
6fb884b
Fixed a regression issue on preview1.sock_accept
john-sharratt Jun 25, 2023
4752cc3
Fixed some integration tests
john-sharratt Jun 25, 2023
d0e08a5
Fixed for nonblocking pipes
john-sharratt Jun 25, 2023
c4c3bca
Added more integration tests
john-sharratt Jun 25, 2023
b54275f
Fixed some linting issues
john-sharratt Jun 27, 2023
fab5c7b
Fixes for capi
john-sharratt Jun 27, 2023
a7f4460
Added fixes for the existing poll_oneoff loop
john-sharratt Jun 27, 2023
6f475ec
More linting fixes
john-sharratt Jun 27, 2023
45825d7
Fixed an issue where capi did not have a tokio runtime
john-sharratt Jun 27, 2023
1becbad
New epoll functionality is now backwards compatible with poll_oneoff
john-sharratt Jun 27, 2023
d86ab4b
Merge branch 'master' into asynchronous-io
theduke Jun 27, 2023
53d3819
Merge branch 'master' into asynchronous-io
john-sharratt Jun 28, 2023
4d707c1
Fixes for the integration and unit tests
john-sharratt Jun 28, 2023
f9e7118
Fixed unit tests for virtual-fs
john-sharratt Jun 28, 2023
bf8d2a3
Fix for linting
john-sharratt Jun 28, 2023
3056cdb
Fix for runtime guard scope
john-sharratt Jun 28, 2023
17047da
Fixes for examples
john-sharratt Jun 28, 2023
1e620ef
Fixed a compile issue with serde
john-sharratt Jun 28, 2023
3d1bccd
Fixes for runtimes in JS
john-sharratt Jun 28, 2023
21755f4
Fix for MIO
john-sharratt Jun 28, 2023
c2e2ad7
Added a runtime startup phase
john-sharratt Jun 28, 2023
3cbcb62
Added dependency needed for examples
john-sharratt Jun 28, 2023
2a0c177
Added dependency needed for examples
john-sharratt Jun 28, 2023
68284bd
Added proper tokio initializtion on wasi_wast tests
ptitSeb Jun 28, 2023
87be930
Merge branch 'master' into asynchronous-io
ptitSeb Jun 28, 2023
d05a92b
Initialze tokio runtime for env_var_errors test
ptitSeb Jun 28, 2023
a65a3a3
This should fix the C-API tests, but will not allows some wasix calls…
ptitSeb Jun 28, 2023
bfb7b81
Fixed nodejs build
ptitSeb Jun 28, 2023
d94b096
Better fix for wasi_wast and the C-API
ptitSeb Jun 28, 2023
cd5c7a1
Fixed lint
ptitSeb Jun 28, 2023
9e2cb84
Merge branch 'master' into asynchronous-io
john-sharratt Jun 29, 2023
7305924
Fixed a compile issue on wasi-web
john-sharratt Jun 29, 2023
9db874b
Merge branch 'master' into asynchronous-io
ptitSeb Jun 30, 2023
eab445c
Fixed CAPI, by transfering ownership of tokio runtime to wasi_env fro…
ptitSeb Jun 30, 2023
e1202f3
Merge branch 'asynchronous-io' into asynchronous-io-with-fs-fixes
john-sharratt Jul 1, 2023
a4e1b70
Updated dependencies
john-sharratt Jul 1, 2023
dae078b
Fixed merge conflict
john-sharratt Jul 1, 2023
a628daf
Vendor should not be in scope for GIT
john-sharratt Jul 1, 2023
b795893
Added additional handling methods for the wakers to avoid race condit…
john-sharratt Jul 1, 2023
dc58bbd
Fix for the python use case
john-sharratt Jul 2, 2023
9e6370a
Added a faster path on the epoll wait with timeout path
john-sharratt Jul 2, 2023
31bf65d
Fixed an issue where the notifications were not renewed on epoll enab…
john-sharratt Jul 2, 2023
01d2e0a
Fixed test_console_dash_tty_with_args_and_env
ptitSeb Jul 3, 2023
41ae697
Add to tests for debugging
ptitSeb Jul 4, 2023
1d4ec81
Merge branch 'master' into asynchronous-io-with-fs-fixes
ptitSeb Jul 4, 2023
d7fdbb6
params is --nocapture
ptitSeb Jul 4, 2023
0021085
Disabling most webc_on_disk test for testing purpose
ptitSeb Jul 4, 2023
6f51980
Also disabled last webc_on_disk test for testing purpose
ptitSeb Jul 5, 2023
c542457
Merge branch 'master' into asynchronous-io-with-fs-fixes
ptitSeb Jul 5, 2023
c0f5700
Disbling all run_unstable test to see if ci don't freeze
ptitSeb Jul 5, 2023
7d0ca68
[CI] Try to define WASMER_PATH for test-integration-cli-ci
ptitSeb Jul 6, 2023
f8ad030
[CI] Another way to define WASMER_PATH for test-integration-cli-ci
ptitSeb Jul 6, 2023
4f8fe7d
Disbling all test_snapshot test to see if ci don't freeze
ptitSeb Jul 6, 2023
a697351
Re-enabling 1 run_ubstable test and correctly export WASMER_PATH
ptitSeb Jul 6, 2023
be60077
Re-enabling all run_ubstable tests
ptitSeb Jul 6, 2023
83b19b0
Re-enabling a few of run_ubstable tests
ptitSeb Jul 6, 2023
b7c551c
Re-enabling a wasm_on_disk of run_ubstable tests
ptitSeb Jul 6, 2023
2277d4a
Re-enabling 1 webc_on_disk of run_ubstable tests
ptitSeb Jul 7, 2023
6382f35
Re-enabling 3 more webc_on_disk of run_ubstable tests
ptitSeb Jul 7, 2023
923c268
Re-enabling 5 more webc_on_disk of run_ubstable tests
ptitSeb Jul 7, 2023
c36c578
Removed obsolete line (was not usefull anomore)
ptitSeb Jul 7, 2023
8ff251f
All run_unstable tests are enabled now
ptitSeb Jul 7, 2023
5a85626
Fixed a bug where empty files were being returned
john-sharratt Jul 8, 2023
409b1d0
Initialize the package loader when running tests
Jul 10, 2023
b30d7ee
Merge branch 'master' into asynchronous-io-with-fs-fixes
john-sharratt Jul 10, 2023
1632db9
Re-enable snapshot test and disable webc_on_disk tests
ptitSeb Jul 10, 2023
d38f20d
Merge branch 'master' into asynchronous-io-with-fs-fixes
ptitSeb Jul 12, 2023
3594a95
Merge branch 'master' into asynchronous-io-with-fs-fixes
ptitSeb Jul 12, 2023
82034be
Fixed an issue where the socket address family was being corrupted due
john-sharratt Jul 12, 2023
c956df9
Updated cargo.lock
john-sharratt Jul 12, 2023
cab3479
Merge branch 'asynchronous-io-with-fs-fixes' of github.com:wasmerio/w…
john-sharratt Jul 12, 2023
7809b13
Fix for the host resolving not working as it does not have a runtime
john-sharratt Jul 12, 2023
655e878
Fix so that the type conversions do not trigger a panic
john-sharratt Jul 12, 2023
103759b
Fix so that connecting sockets that pass in a peer address work
john-sharratt Jul 12, 2023
b98dad8
Fix docs-rs build step
ptitSeb Jul 13, 2023
61fb88a
Fixes for outbound TCP connections using WASIX
john-sharratt Jul 14, 2023
a460586
Merge branch 'asynchronous-io-with-fs-fixes' of github.com:wasmerio/w…
john-sharratt Jul 14, 2023
0c66278
Fixed an issue where InlineWaker was not working on Javascript enviro…
john-sharratt Jul 15, 2023
ea9f7a4
Fixed an issue where static files were being copied on the web browse…
john-sharratt Jul 15, 2023
13ed3c4
Fixed an issue where the browser had no Wasmer engine
john-sharratt Jul 15, 2023
04803a2
Reduced the logging level back down again
john-sharratt Jul 15, 2023
482d783
Fixed the wasmer CLI build error
john-sharratt Jul 15, 2023
78b807f
Fixed some compile issue on the compiler tests
john-sharratt Jul 15, 2023
c7b4bc9
Fixed a race condition path on InlineWaker
john-sharratt Jul 15, 2023
7d46fad
Converted the InlineWaker to use a standard async runtime rather than…
john-sharratt Jul 15, 2023
eb436ed
Now flushing the file system writes to fix a flacky test
john-sharratt Jul 16, 2023
5b13590
This unit test should have been doing and not calls
john-sharratt Jul 16, 2023
debf855
Fixed an issue where the reqwest client was using the wrong Tokio run…
john-sharratt Jul 17, 2023
295f7e5
Fixed a warning on the unit tests
john-sharratt Jul 17, 2023
3ff7e6f
Added a watchman which will kill the subprocess of snapshot tests aft…
john-sharratt Jul 17, 2023
d22f78a
Fixed runners tests on Windows
ptitSeb Jul 17, 2023
ed2d5eb
Windows can't use the watchman
john-sharratt Jul 17, 2023
0f29d06
Re-enabled disabled tests
ptitSeb Jul 17, 2023
d0622ba
Merge branch 'asynchronous-io-with-fs-fixes' of github.com:wasmerio/w…
john-sharratt Jul 17, 2023
e6e390d
Increase number of threads in rayon thread pool.
theduke Jul 20, 2023
b4c0f40
force-trigger CI
theduke Jul 20, 2023
5ed18a6
Now passing through more log information for integration tests
john-sharratt Jul 20, 2023
e2f5b39
Adding more tracing on spawned threads
john-sharratt Jul 20, 2023
efbdebc
Added more logging on the background thread
john-sharratt Jul 20, 2023
618e6be
Merge branch 'asynchronous-io-with-fs-fixes' of github.com:wasmerio/w…
john-sharratt Jul 20, 2023
2d35ac7
Fix for the thread pool blocking issue
john-sharratt Jul 20, 2023
8e07d72
Cargo.lock updates
john-sharratt Jul 20, 2023
09ee543
Merge branch 'master' into asynchronous-io-with-fs-fixes
john-sharratt Jul 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -541,24 +541,28 @@ jobs:
build: linux-x64,
os: ubuntu-22.04,
target: x86_64-unknown-linux-gnu,
exe: '',
llvm_url: 'https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.0/clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz'
},
{
build: macos-x64,
os: macos-11,
target: x86_64-apple-darwin,
exe: '',
llvm_url: 'https://github.com/llvm/llvm-project/releases/download/llvmorg-14.0.6/clang+llvm-14.0.6-x86_64-apple-darwin.tar.xz'
},
{
build: windows-x64,
os: windows-2019,
target: x86_64-pc-windows-msvc,
exe: '.exe',
llvm_url: 'https://github.com/wasmerio/llvm-custom-builds/releases/download/14.x/llvm-windows-amd64.tar.xz'
},
{
build: linux-musl,
target: x86_64-unknown-linux-musl,
os: ubuntu-22.04,
exe: '',
container: 'alpine:latest'
}
]
Expand Down Expand Up @@ -789,7 +793,9 @@ jobs:
- name: Test integration CLI
if: matrix.build != 'macos-arm'
shell: bash
run: export WASMER_DIR=`pwd`/package && make test-integration-cli-ci
run: |
export WASMER_PATH=`pwd`/target/${{ matrix.target }}/release/wasmer${{ matrix.exe }}
export WASMER_DIR=`pwd`/package && make test-integration-cli-ci
env:
TARGET: ${{ matrix.target }}
TARGET_DIR: target/${{ matrix.target }}/release
Expand Down
24 changes: 23 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 4 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ wasmer-cache = { version = "=4.0.0", path = "lib/cache", optional = true }
wasmer-types = { version = "=4.0.0", path = "lib/types" }
wasmer-middlewares = { version = "=4.0.0", path = "lib/middlewares", optional = true }
cfg-if = "1.0"
tokio = { version = "1", features = [ "rt", "rt-multi-thread", "macros" ], optional = true }

[workspace]
members = [
Expand All @@ -49,6 +50,7 @@ members = [
"lib/registry",
"lib/sys-utils",
"lib/types",
"lib/virtual-io",
"lib/virtual-fs",
"lib/virtual-net",
"lib/vm",
Expand Down Expand Up @@ -234,12 +236,12 @@ required-features = ["backend", "wasi"]
[[example]]
name = "wasi-manual-setup"
path = "examples/wasi_manual_setup.rs"
required-features = ["backend", "wasi"]
required-features = ["tokio", "backend", "wasi"]

[[example]]
name = "wasi-pipes"
path = "examples/wasi_pipes.rs"
required-features = ["backend", "wasi"]
required-features = ["tokio", "backend", "wasi"]

[[example]]
name = "table"
Expand Down
4 changes: 1 addition & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,6 @@ endif
# If findstring is not empty, then it have found the value

exclude_tests := --exclude wasmer-c-api --exclude wasmer-cli --exclude wasmer-compiler-cli
# Is failing to compile in Linux for some reason
exclude_tests += --exclude wasmer-wasi-experimental-io-devices
# We run integration tests separately (it requires building the c-api)
exclude_tests += --exclude wasmer-integration-tests-cli
exclude_tests += --exclude wasmer-integration-tests-ios
Expand Down Expand Up @@ -626,7 +624,7 @@ test-integration-cli: build-wasmer build-capi package-capi-headless package dist
# Before running this in the CI, we need to set up link.tar.gz and /cache/wasmer-[target].tar.gz
test-integration-cli-ci:
rustup target add wasm32-wasi
$(CARGO_BINARY) test $(CARGO_TARGET_FLAG) --features webc_runner -p wasmer-integration-tests-cli -- --test-threads=1
$(CARGO_BINARY) test $(CARGO_TARGET_FLAG) --features webc_runner -p wasmer-integration-tests-cli -- --test-threads=1 --nocapture

test-integration-ios:
$(CARGO_BINARY) test $(CARGO_TARGET_FLAG) --features webc_runner -p wasmer-integration-tests-ios
Expand Down
7 changes: 7 additions & 0 deletions examples/wasi_manual_setup.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
// Let's compile the Wasm module.
let module = Module::new(&store, wasm_bytes)?;

println!("Starting `tokio` runtime...");
let runtime = tokio::runtime::Builder::new_multi_thread()
john-sharratt marked this conversation as resolved.
Show resolved Hide resolved
.enable_all()
.build()
.unwrap();
let _guard = runtime.enter();

println!("Creating `WasiEnv`...");
// First, we create the `WasiEnv`
let mut wasi_env = WasiEnv::builder("hello")
Expand Down
3 changes: 2 additions & 1 deletion lib/c-api/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ libc = { version = "^0.2", default-features = false }
thiserror = "1"
typetag = { version = "0.1", optional = true }
paste = "1.0"
tokio = { version = "1", features = [ "rt", "rt-multi-thread", "io-util", "sync", "macros"], default_features = false }

[dev-dependencies]
field-offset = "0.3.3"
Expand Down Expand Up @@ -97,7 +98,7 @@ wasmer-artifact-load = ["wasmer-compiler/wasmer-artifact-load"]
wasmer-artifact-create = ["wasmer-compiler/wasmer-artifact-create"]
static-artifact-load = ["wasmer-compiler/static-artifact-load"]
static-artifact-create = ["wasmer-compiler/static-artifact-create"]
webc_runner = ["wasmer-wasix/webc_runner", "virtual-fs", "webc"]
webc_runner = ["virtual-fs", "webc"]
# Deprecated features.
jit = ["compiler"]

Expand Down
Loading