stdio in spin

[Mossaka]

How does application wire it's own stdin/out/err path to spin such that spin pass them to WasiCtxBuilder?

I found the relevent implementation here:

spin/crates/http/src/spin.rs

Lines 217 to 235 in a9181f7

	pub fn prepare_io_redirects() -> Result<IoStreamRedirects> {
	let stdin = ReadPipe::from(vec![]);
	let stdout_buf: Vec<u8> = vec![];
	let lock = Arc::new(RwLock::new(stdout_buf));
	let stdout = WritePipe::from_shared(lock.clone());
	let stdout = OutRedirect { out: stdout, lock };
	let stderr_buf: Vec<u8> = vec![];
	let lock = Arc::new(RwLock::new(stderr_buf));
	let stderr = WritePipe::from_shared(lock.clone());
	let stderr = OutRedirect { out: stderr, lock };
	Ok(IoStreamRedirects {
	stdin,
	stdout,
	stderr,
	})
	}

[lann]

We don't currently expose stdio streams, we just save them to logs. PR #431 aims to extend this behavior to optionally forward guest stdio to host stdio. Could you expand on what you'd like to do? If you have another use case it might change the design of #431.

[radu-matei]

ref #380

[Mossaka]

Could you expand on what you'd like to do?

I am using the Http Trigger in an application, which has inherited stdio (they are path to stdio files). I want to wire them to spin's stdio, or more specifically, WasiCtx's stdio

[radu-matei]

As @lann points out, #380 and #431 address redirecting component stdio to Spin's standard output/error.
So if the goal is to capture logs, #380 is the place we are trying to figure out the user stories for that.

(see https://github.com/fermyon/spin/pull/431/files#diff-e009a66ed5d35b924eb37f0c5fc5c5abd0abf68521a94d316a4dba88c97410a9R1)

[lann]

I was mistaken - we do expose stdio, but only as buffers (Vec<u8>), so you can pass in a complete buffer to stdin and get the stdout/stderr buffers. What we don't support is a streaming interface to either of these.

For your use-case, can you clarify if you want stdio for logging or some other purpose? The Wagi HTTP executor, for example, uses stdin to provide the request to the guest, and parses stdout into the response:

spin/crates/http/src/wagi.rs

Line 54 in 0f51ad0

let iostream = Self::streams_from_body(body);

spin/crates/http/src/wagi.rs

Line 116 in 0f51ad0

wagi::handlers::compose_response(iostream.stdout.lock)

[danbugs]

@lann ~ We are looking to capture IO to memory. Actually, it seems like #431 implements pretty much all we need. We'd make use of capture_io_to_memory, and the updated prepare_component. The only thing we'd like and that is currently not provided is the ability to pass our own Box<dyn WasiFile>s to capture_io_to_memory.

[itowlson]

@danbugs If you want to pass your own Box<dyn WasiFile>, and are at the level where you could call capture_io_to_memory yourself, you could construct a ModuleIoRedirects directly. capture_io_to_memory just encapsulates a pattern for producing WasiFles backed by memory buffers with optional streaming. You can also lean on redirect_to_mem_buffer which does the "standard" behaviour for the output streams.

Would that provide the flexibility you need? This area and the current PR draft are very much up for refinement, and it would be great to have your feedback on the right structure.

[danbugs]

@itowlson ~ If I understand this correctly, redirect_to_mem_buffer just gives us a pipe (i.e., WasiFile) to build ModuleIoRedirects with, which we pass to prepare_component that hooks up the spin engine to use that for IO, right?

If that's the case, I don't think we need redirect_to_mem_buffer — we have our WasiFiles already. We just need to create a ModuleIoRedirects instance with our own WasiFiles and pass to prepare_component...

[itowlson]

That's exactly right. If you have your WasiFiles already, you can call ModuleIoRedirects::new directly, and pass that to prepare_component. (At least, you'll be able to if that PR is accepted.)

WAGI does this in a very simple form because it needs different behaviour for stdin, and a slightly different interaction with the buffer for stdout. It's really valuable to know that this is more broadly useful as a public customisation point though!

[radu-matei]

#431 (comment)

How would we feel about prepending the component name in the followed logs?
For example, following the logs of the Spin docs from this repo, it is unclear in which component a log line originated.

Not found: /favicon.ico

Cannot read file: cannot open /image/nonexistent.png

Caused by:
    No such file or directory (os error 44)

For example, Docker Compose's output (source):

#docker-compose -f docker-compose-all.yml logs -t|head
Attaching to oauth_web_1, oauth_core_1, oauth_core-ok_1, oauth_core-fb_1, oauth_oauth_1, oauth_fill-sqls-web_1, oauth_fill-sqls-oauth_1, oauth_web-redis_1, oauth_core-redis_1, oauth_core-mysql_1, oauth_oauth-mysql_1, oauth_web-mysql_1
core_1 | 2016-10-07T06:22:10.603699134Z [nodemon] 1.10.2
web_1 | 2016-10-07T06:23:26.013419543Z [06:23:26] Using gulpfile /usr/src/app/gulpfile.js

[lann]

This is addressed by #763