Implementation experience in Rust

[rustaceanrob]

Raising an issue here to discuss my initial pass using the IPC interface in a foreign language. I built a simple proof of concept that fetches a block template from a Rust program. I had to make a couple of modifications to the capnp files. First being to remove any reference to C headers within the capnp files. I am wondering if anything may be done here to remain language agnostic? Also, using the Proxy interface requires an mp namespace, and within the Rust project it made more sense to include this at the top level group of capnp files, since any references to C headers were removed anyway.

This was my first pass at using capnp, so I understand this issue may be vague. However I think it is useful to see how one might use IPC in a foreign language and what may be done to accommodate that use-case.

[TheCharlatan]

Are you sure those modifications are actually needed? I believe in theory the capnpc-rust compiler should just skip over them. I tested your branch by manually copying over the current capnp files from bitcoin, removing the schema/proxy.capnp file in your repository. Does my branch run for you https://github.com/TheCharlatan/bitcoin-ipc/tree/tc_mining?

[rustaceanrob]

Pulling that branch down, starting bitcoin-node and running cargo run ... gives an error:

thread 'main' panicked at build.rs:17:10:
  called `Result::unwrap()` on an `Err` value: Error { kind: Failed, extra: "Error while trying to execute `capnp compile`: Premature end of file." }

With stderr output:

capnp/common.capnp:10:22-39: error: Import failed: /mp/proxy.capnp
capnp/common.capnp:11:2-7: error: Not defined: Proxy
capnp/common.capnp:13:18-23: error: Not defined: Proxy

Which continues on for echo.capnp etc.

I notice your commit has an includeTypes that references a path starting with ipc/ in reference to the proxy. Is there a requirement on where this repository is located relative to bitcoin?

Thanks for the quick feedback

[ryanofsky]

If I am understanding the issue correctly, it does seem unfortunate that the bitcoin capnp files have using Proxy = import "/mp/proxy.capnp"; lines requiring the proxy.capnp file from libmultiprocess to be installed.

I'm not sure if there is a easy way to handle this other than by adding documentation. It seem like there a number of similar things (like the need to for clients to create threads for IPC calls to run on) that should be documented for anybody trying to use the capnp files without using the libmultiprocess c++ library and code generator.

Maybe another solution could be to make a libmultiprocess rust crate that provides the proxy.capnp file? I don't know how hard this might be, since I haven't really experimented with rust. I think if a libmultiprocess crate existed it could potentially provide other useful things like helper functions and macros to eliminate boilerplate. In theory it could even provide a code generator, though I imagine rust macros are powerful enough that a code generator wouldn't actually be helpful.

[rustaceanrob]

requiring the proxy.capnp file from libmultiprocess to be installed.

This would make a lot of sense. I am using NixOS, which has strict requirements on if/how things are installed on the system, so I've just been using ./build to run the binaries. If there is an installation requirement that would explain the issue.

I think documentation would generally solve this. There is an ecosystem of crates for capnp support, it is just a matter of copying the schemas over and the client/server code is generated automatically.

[TheCharlatan]

If I am understanding the issue correctly, it does seem unfortunate that the bitcoin capnp files have using Proxy = import "/mp/proxy.capnp"; lines requiring the proxy.capnp file from libmultiprocess to be installed.

Ah, of course, I still have those installed system-wide.

[ryanofsky]

I think in order to get this working properly with nix two things need to happen:

1 - The /mp/proxy.capnp file needs to be provided to the rust build
2 - In the rust build, the capnpc rust crate needs to call capnp compiler with an --import-path=DIR option pointing to the include directory containing the file.

I think there are two ways (1) could happen. One is that there could be a libmultiprocess nix package created which installs the file to /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-libmultiprocess-dev/include/proxy.capnp. Another way might be that a libmultiprocess rust crates provides the file.

I am less sure about (2). If there is a libmultiprocess nix package it will provide a pkgconfig .pc file that the rust build can locate and read. If there is a libmultiprocess rust crate, I'm guessing there is another mechanism rust crates might use to locate files from other crates.

In meantime it might be easiest to just copy the proxy.capnp file like you did since there isn't really an equivalent of installing things system-wide on nix (that I know of).

[TheCharlatan]

@rustaceanrob can you check my latest push? That should include it correctly, and all the files have just been copied over from either bitcoin, or this repository.

[ryanofsky]

In meantime it might be easiest to just copy the proxy.capnp file like you did since there isn't really an equivalent of installing things system-wide on nix (that I know of).

Chatgpt suggested using the import_path option to capnpc::CompilerCommand to specify a custom import path and just be able to copy the .capnp files without editing them:

your-repo/
├── capnp/
│   └── …                 # your existing .capnp files
└── vendor/
    └── mp/
        └── proxy.capnp   # copy of the file from libmultiprocess

// build.rs
capnpc::CompilerCommand::new()
    .import_path("vendor")          // <- gives the compiler vendor/mp/proxy.capnp
    .file("capnp/common.capnp")
    .run()
    .unwrap();

Maybe this could be a good approach. (Full suggestion is https://chatgpt.com/share/6824b9ce-7118-800a-885d-95e3a983ad3f)

... and it seems like the TheCharlatan posted a similar solution https://github.com/TheCharlatan/bitcoin-ipc/blob/tc_mining/build.rs

[rustaceanrob]

I will be away from my computer until Monday, but I will have feedback then

[rustaceanrob]

can you check my latest push? That should include it correctly, and all the files have just been copied over from either bitcoin, or this repository.

This commit worked locally

[ryanofsky]

This commit worked locally

Great! Can leave this issue open because I think at very least we should add some documentation about this.