Stream Audio

[sigmaSd]

This is just a test for the new api,

Here is a hack https://github.com/lemunozm/termchat/compare/master...sigmaSd:audio?expand=1

Commands: ?sa send audio | ?sas stop audio sending | ?ra receive audio | ?ras stop receiving audio

Works only on linux and ?ra needs to be called before ?sa or aplay wont understand the stream

New api shortcomings:

• Stopping( or Modifying) Sending or Receiving is not possible with the new api, since once an action is created, its not possible to access it anymore

[lemunozm]

Hi @sigmaSd,

I like so much this feature!

I will work to add the possibility of modifying a current action :)

[sigmaSd]

Also current api expects a commands with ?

I think it should also account for special key press

For example I imagine sending audio like whatsaps, you hold a key like ctrl space while recording, then when you release the key it get sent.

[lemunozm]

The problem with keys is that in the terminal you have not a "key release" event. So the behavior of running an action during holding a key could be difficult to emulate :S

Regarding to dispatch an action at keypress, could be aliasing sending a "?command" when the key pressed in detected

[sigmaSd]

Sticking with ?command is fine. And since the code base is nicely organized maybe in the future you can add another frontend like gtk which will enable this and some other cool things like streaming video maybe

[lemunozm]

I like so much the idea of the video. Maybe, even parse the video into some asciI format could be really cool and something not seen :D

[sigmaSd]

Ascii video is definitly a crazy idea!
should be possible!

[sigmaSd]

https://github.com/ecumene/rust-sloth the hardest part which is rendering is possible apperently

[sigmaSd]

Something like https://github.com/mazy1998/asciiVideo integrated in termchat will defently be next level

[lemunozm]

We would need something like https://github.com/mazy1998/asciiVideo but implemented in rust.

In the future it could be the competition of zoom! 😄

[sigmaSd]

xD

[lemunozm]

Stopping( or Modifying) Sending or Receiving is not possible with the new api, since once an action is created, its not possible to access it anymore

I was thinking about this. It is reachable with the new API using the State. In my mind it works in the following way:

1. ?ra is a new command in which Action::process update the state with a variable play.
2. Some similiar with ?ras nad ?sas (also commands with actions that mutate the state (play and stop)).
3. And from ?sa you read that state. Depending on the state value (play and stop values), you can stop it by returning Processing::Complete.

Instead of modifying the acreated action, it's the action which read the state to modify itself

[sigmaSd]

I add streaming video to termchat https://github.com/lemunozm/termchat/compare/master...sigmaSd:video?expand=1

Any suggestion on how to actually use your api correctly ? I tried your idea of putting the information in state but that still wasn't enough

[sigmaSd]

It works quite well actually!
Maybe I'll open a pr if I can figure out how to adapt to the api

2 limitations:

• The windows are not closed when the stream finishes ( still trying to figure a workaround for this one) fixed
• Linux only But this seems promising https://crates.io/crates/eye it uses https://github.com/openpnp/openpnp-capture under the hood which already supports crossplatform usage

[lemunozm]

Hi @sigmaSd,

I tested your branch and looks really good!

Any suggestion on how to actually use your api correctly ? I tried your idea of putting the information in state but that still wasn't enough

Somewhere in particular?
As a little comment, I would place the windows hashmap into the state. Also, I would create small commands for ?stream and ?stopstream (in the send_stream module is ok). The idea is to leave the application module without touching as much as possible. At this moment only for processing income messages.

Additional points:

• I get a panic in the sender if close the receiver. It seems like the connection with the receiver is closed but the the sender continues sending frames.
• As an idea. It could be cool to create a UDP connection for streaming purposes. For that, each termchat instance should start listening from a UDP port. I can add this feature to allow building actions that need to send data by UDP.

[sigmaSd]

Hi @lemunozm

I applied some of your suggestions here 64ba021

The panic is problematic, I think the disconnect should be handled by RemoveEndpoint message, but there is a time before that message arrive where the stream is still sending frames

Since send doesn't return an error anymore I'm not sure how to handle this, with the error you could immediately react to the disconnection, but now you have to wait for a disconnect message while some things may happen before it arrives

[sigmaSd]

For reference to add audio to the stream, I think cpal crate should make it easy.

https://github.com/RustAudio/cpal/blob/master/examples/feedback.rs

[lemunozm]

Since send doesn't return an error anymore I'm not sure how to handle this, with the error you could immediately react to the disconnection, but now you have to wait for a disconnect message while some things may happen before it arrives

I think that sending the RemoveEndpoint event with priority in the callback located in Network::new() could solve the problem. Anyway, I will think about a more "automatic" solution in message-io.

[sigmaSd]

I tried with send_with_priority but its still paniced unfortunately.

I opened a pr here #24 we can continue the discussion there.

[sigmaSd]

@lemunozm regarding asciivideo, https://github.com/sigmaSd/term_cam/blob/master/src/main.rs

[lemunozm]

Wow!! It looks really cool :)

I can't wait to see it in termchat! 😃

I suppose that using the Buffer struct of tui-rs could help a lot in performance because it makes a lot of performance tricks. You can reach probably 30fps (no more because the terminals are usually fixed to 30 fps)

[sigmaSd]

@lemunozm Any idea about how to integrate it in termchat, like where to put the video?

What about something like:

name: \<live video feedback here\> msg

name: \<live video feedback here\> msg

name: \<live video feedback here\> msg

[sigmaSd]

Or maybe a side panel that appears when a live cam feed appears.

[lemunozm]

I opened an issue to keep the conversation there :)

[sigmaSd]

Hello @lemunozm ,

I'm trying to add streaming audio with https://github.com/libcala/wavy crate here is a poc https://gist.github.com/sigmaSd/30d0e2b1b6bf9bdf85ea60034e8823d4

The problem this requires and async executor, I'm not sure how well it integrates in termchat.

Feel free to comment here ardaku/fon#6 (reply in thread) for ideas.

[sigmaSd]

Here is another poc this time integrated into termchat https://github.com/lemunozm/termchat/compare/master...sigmaSd:audio_wavy?expand=1

The current problems:

• The pasts executor calls process::exit when its done, this can be fixed by using another async executor
• I have a bad mic in my laptop so I'm not sure if this is really working need someone else to test!

[sigmaSd]

I updated the branch to use futures::executor and it works now.
https://github.com/lemunozm/termchat/compare/master...sigmaSd:audio_wavy?expand=1

The problem is performance, it spins the cpu a a lot.