feat: support load plugins from the web

[jaeheonji]

resolve #1393

This PR is currently in very rough.
I have only implemented a few concepts and there are still more to be implemented.

What is this feature?

Zellij now adds a new format for plugin location. If the location is an external URL such as http or https, zellij will download the plugins from URL and load it automatically.

following is example:

layout {
    pane
    pane {
        plugin location="remote:https://github.com/imsnif/monocle/releases/download/0.37.2/monocle.wasm"
    }
    pane size=1 borderless=true {
        plugin location="zellij:compact-bar"
    }
}

To-Do

• loading plugins directly with the action command (currently only support layout file)
• add URL-based hash to the cache folder path where plugins are stored. (ex. ZELLIJ_CACHE_DIR/{URL_HASH}/plguins.wasm)

[imsnif]

!!!!!

I'm very happy to see this! Will go over it tonight. Any specific questions or concerns you'd like me to look into @jaeheonji ?

[jaeheonji]

@imsnif Glad to hear 😄

I just want to hear your opinion on the overall concept :)
(I think the To-Do content can be completed within 2-3 days. There is no problem 👍 )

[imsnif]

Hey @jaeheonji - great start on this!

I think my main comment for now is that as far as I understand it, the plugins are downloaded when we parse the KDL layout. I think I would rather they happen when we load the plugin, somewhere here: https://github.com/zellij-org/zellij/blob/main/zellij-server/src/plugins/wasm_bridge.rs#L144

This way we don't need to deal with concurrency (eg. bringing in tokio) because we are already doing this async, and loading plugins from the web will not block Zellij - it will just block the relevant pane and show us an indication (eg. Fetching plugin from the web: <URL>... DONE).

I think this will also make it easier to load these URLs through the CLI or from other plugins. What do you think?

[jaeheonji]

@imsnif

Wow, It makes sense. I forgot that our plugin loading system runs async. I think it would be better. I'm going to change it in this direction.

[imsnif]

Cool - just a quick tip for you (which I found while working on a different feature): reqwest will not work with async_std - which is what we use for all the plugin async stuff. I haven't tried it yet, but might be a good idea to use surf instead.

[jaeheonji]

@imsnif Hi, I just finished most of the work.

First, there are a few changes and issues.

The change is that a prefix called "remote" was created in the plugin location. The reason for creating this is to make parsing easier and uniform with other prefixes.

And, about issues. There are two issues.

1. I used the most common ring package for the hash function. However, it seems like this is causing an error due to dependency within CI.
2. I saw that you used h1-client-rustls as a client for surf. I wanted to use this setting as is, but when I sent a request for the github link to the client, an HTTP version error occurred. So, I tried setting it back to the default client. Are there any other problems by doing this?

[imsnif]

Hey @jaeheonji - great work, I'm happy to see you making such good progress with this. Let's get it over the finish line!

The change is that a prefix called "remote" was created in the plugin location. The reason for creating this is to make parsing easier and uniform with other prefixes.

Good call.

1. I used the most common ring package for the hash function. However, it seems like this is causing an error due to dependency within CI.

What did you need ring for? I'm not sure it's the same thing, but we use highway (eg. here) for hashing. Maybe it can work for your case too?

2. I saw that you used h1-client-rustls as a client for surf. I wanted to use this setting as is, but when I sent a request for the github link to the client, an HTTP version error occurred. So, I tried setting it back to the default client. Are there any other problems by doing this?

Yeah, the default curl client was failing for me when building our musl executable. We use it for the e2e tests (which is where I tested it and got the error) - but we also use it to to distribute a generic binary. So I think we have two ways forward here:

1. Use the default curl client, find out what's wrong with building the musl executable and fix it.
2. Use the rustls client and find out what's wrong with the version mismatch and how to fix it.

Makes sense?

[jaeheonji]

What did you need ring for? I'm not sure it's the same thing, but we use highway (eg. here) for hashing. Maybe it can work for your case too?

Oh cool, then I'll use it!

Yeah, the default curl client was failing for me when building our musl executable. We use it for the e2e tests (which is where I tested it and got the error) - but we also use it to to distribute a generic binary. So I think we have two ways forward here:

Ah I see, I can understand. All makes sense.

[jaeheonji]

Hey, @imsnif

I would like to share with you what I have worked on so far.

First, thank you for letting me know about highway. Using this, I was able to implement a Hash Function very simply.

And I did some research on Surf Client. The reason h1-client did not work in my environment is because the async-h1 library does not support HTTP2. After looking at Github's traffic, I found that most response were HTTP2.

Recently, many sites support HTTP2, and it is believed that the library's failure to support this will be a problem in the future. So I thought it would be appropriate to use curl-client.

The problem here, as you already mentioned, is with the musl build. The key is that it doesn't find the openssl library. Fortunately, The Rust community already had a nice solution for this. (https://github.com/alexcrichton/openssl-src-rs) And, this could be enabled as a vendored feature of openssl-sys.

Therefore, since the step of installing musl-tools already exists in our CI, the build issue has also been resolved.

[imsnif]

Great work, @jaeheonji ! So just to make sure: if we use the curl-client, then we would only be adding build-time dependencies, right? For example, we would not need openssl or curl or anything else installed if we download the pre-built musl binary from our releases?

[jaeheonji]

@imsnif Yes sure,

Just all we need is musl-tools to build e2e locally. (In other words, nothing changes in development and build.)

[imsnif]

Hey @jaeheonji - I have gone over the PR and tested it. First off: great work as always!

I made some small fixes and minor adjustments:

1. I fixed a minor issue in Cargo.toml that caused some warnings about openssl-sys
2. I removed the need for the remote: prefix in URLs. So we're still using Remote(..) internally, but we create it out of urls starting with http or https. I think this is more user friendly.
3. I added a new zellij plugin command, so that users can do zellij plugin -- https://path/to/my/plugin.wasm

If all is well I'll merge this later today. Excited for this!!