Including a genfile at compile time is not possible

[SirVer]

Minimal repro case with a potential (but ugly) fix:

https://github.com/bazelbuild/rules_rust/compare/master...SirVer:00_include_genfiles?expand=1

The linked branch does the following changes:

1. Add a genrule call that trivially creates a file containing a string.
2. The goal is to include_str! this file in hello_world.rs. Code for this is here. For this to ever work, the genfile is made a data dependency of the rust_binary.
3. I could not pull this off without exposing bazel's ctx.bin_dir and ctx.genfiles_dir as environment variables. This is still ugly, since it includes "external", so the file needs to have the context that it is embedded as an external workspace.

Ideally, I would just want to include_str!("message.string"); or include_str!("../message.string");, but both do not currently work. Is there a better way or solution to this issue? Should my repro case become a patch?

[acmcarther]

I've been able to do this in an ugly way by using .../ as many times as my dir is nested into the workspace:

// experimental/renderer/planet/planet_renderer.rs
let frag_shader_module = do_or_die!(vkl::builtins::make_shader_module(
  &base_renderer.device,
  include_bytes!(
    "../../../bazel-out/k8-fastbuild/genfiles/experimental/renderer/planet/planet_frag_shader.spv"
  ),
));

# experimental/renderer/planet/BUILD
rust_library(
    name = "planet_renderer",
    data = [
        ":planet_frag_shader",
    ],
)

compile_spirv_shader(
    name = "planet_frag_shader",
    src = ":planet.frag",
)

I suspect that this is breaking the isolation provided by the sandbox, but it does work. It would probably not be a bad idea for one of us to put together a client library for genfile access (taking into account what cc_library does to support such files). If they use env_vars for it, then we should probably do the same.

[SirVer]

@acmcarther Your approach would mostly work, but has some issues:

• It only works with one buildtype, since k8-fastbuild is hardcoded in your code. If you compile using -c opt it will fail.
• It only works in the current space. If you introduce an extra workspace it will start failing. I guess this is okay though.

a client library for genfile access

I remember from my google days that there is no clear suggestion how to get to the runfiles directory at runtime. You just had to know what borg would do when it ran a binary. I do not think they use env_vars for anything. This issue seems to have bled out into bazel too: bazelbuild/bazel#2545

My issues here is not about runtime files though, it is about including a file at compile times. The ENV variables I exposed are set for the call to rustc which requires it to find the files.

[mfarrugi]

I don't see any problem with providing an environment variable, but I couldn't figure out what the equivalent workflow for cc_rules is.
The only concern is that it's not clear which paths are stable enough to expose.

Another hack is to use a genrule to copy sources to genfiles, and point rust_library to the genrule's output and the message.string file.

[acmcarther]

I took a look as well at cc_library and couldn't find anything analogous to a "compile time file include". Perhaps a genrule that generates a data-bearing crate would be "idiomatic"? Something like this:

compile_spirv_shader(
    name = "frag_shader",
    src = ":planet.frag",
)

rust_data_crate(
  name = "my_bytes",
  data = {
    "vert_shader": ":vert_shader",
    "frag_shader": ":frag_shader"
  }
)

This would generate a rust_library like this:

rust_library(
  name = "my_bytes",
  srcs = ["my_bytes_lib_rs"]
)
native.genrule(
  name = name + "_lib_rs",
  srcs = data.values()
  outs = [name + "_lib.rs"],
  cmd = synthesize_rust_data_source(data.keys())
)

With an example source

pub const vert_shader: &'static [u8; 3] = [1u8, 2u8, 3u8];
pub const frag_shader: &'static [u8; 3] = [1u8, 2u8, 3u8];

---

I understand that this is only an option if you control the source of a crate, but this does strike me as the "more correct" way to do this. I suspect the only reason include_str exists is because one couldn't expect Cargo to produce new build targets so seamlessly.

[SirVer]

@acmcarther Alex, your suggestion is not fully suitable to my needs: My use case is including the minified and packed frontend code output by a node build step. If I want to dump this into a rust string, I'd need to do proper encoding of the string - a much more complex premise.

include_str! is the idiomatic rust way of dealing with this and I find it natural to be supported in bazel as well to include data dependencies.

[acmcarther]

I can interpret your point in one of two ways, trying to address both

• if emphasis is on include_str

There's no reason why the include content has to be bytes, any kind of serialization or deserialization could be supported at compile time, no worries.

• if emphasis is on include_str

I'm on the same page as you -- we should support as many features out of the box as possible.

As I see it though, there is a major snag in how include_(byte|str|etc)! is implemented. My understanding is that it looks in the working directory (of the crate root). As I understand it, the only way this could work without impact in your rust code is if all genfiles were placed directly into the working directory at build time -- something that Bazel is unlikely to support without weirdness.

Definitely needs more thought. I really suspect something like a proc-macro "client library" might be able to replicate the include_xyz style, while being slightly smarter about the whole thing.

[acmcarther]

For example:

If you have minified frontend code (something like a single really long line of javascript), you could generate a rust crate containing that string by inlining the contents into a crate. A primitive example similar to my protobuf code would work:

# A really, really, escaped string (would work unless '######"' was in the string)
"const my_minified_js: &'static str = r######\"%s\"######".format(file_contents)

This is far from robust, and you'd probably want to follow the conventional discipline regarding string escaping. I'm not actually very familiar with the problem space, so I don't have any advice there.

---

We also could do something entirely different. Since this is a unique capability of rust, we could add a unique attr: something like compile_time_data. As a prereq in rust_library, we could copy all files into something like ./__compile_time_data, and then use it like include_str("./__compile_time_data/whatever.txt"). To me this seems like a much more complex solution, and it doesn't maintain compatibility with Cargo at all, so I'm not really a fan.

---

EDIT: And of course, we could have a compile-time ENV value, to be used like

include_str!(env!("BAZEL_COMPILE_FILES")/workspace.txt)

Though I feel like this solution is effectively the same as the second suggestion there.

[SirVer]

I now suggested my fix for merging. It is reasonable clean and a workable solution and solves the problem, so why not doing it like this.

[hlopko]

Sorry to be late to the discussion. I see more things I want to comment on so let's go in order:

1. runfiles were hard to work with, but @laszlocsomor has done a ton of work providing libraries for multiple languages that allow easy relative path -> absolute path conversion inside binaries. I'm trying to get some more documentation on it.
2. For including data in C++ we have a cc_embed_data rule internally. It takes a file, and depending on the toolchain uses an obj file writer or cc source writer. It then gets compiled and linked similarly to how cc_library is handled. And from the perspective of providers, cc_embed_data looks exactly like cc_library.
3. If you don't like 2) you can still write a custom rust rule, say rust_include_data, that will expose the data file in the rust provider (which will have to be modified to support this kind of files). The only thing thatrust_binary then has to do is make data files as inputs to the final action either at some well known location, or (if possible) tell the tool (rustc?) where it will find data files.

Does it make sense?

I think #82 is super simple, but also quite hacky and probably unstable. We are rethinking how external repositories are accessed and #82 will very likely be broken by that work. Also it's ugly that you have to hardcode knowledge about bazel into your source code. Let's see if we can make 2 and 3 work.

[SirVer]

@mhlopko Thanks for chiming in on this Marcel!

I think given rust's design, I like 3. the best. It is very similar to compile_time_data that @acmcarther was proposing above. rustc could get the information where to find the data passed in an environment variable and it could be accessed using the build in env! macro, similar to

const MSG: &str = include_str!(concat!(env!("COMPILE_TIME_DATA"), "../message.string"));

Note the relative path to the current src file. I think it is unavoidable that you need to encode some bazel knowledge (like the name of the environment variable) in your code though. This concerns me little though.

I like this solution, but I think this is above my bazel-foo to implement though :(. @acmcarther @mfarrugi What do you think?

[redshirtrob]

Is anything happening on this issue? I'm very interested as I'm running into a variation on this when trying to use rust-peg.

[acmcarther]

Short answer: No, I don't think anyone is spending any time on this right now.

It's high priority, but I haven't had any bandwidth to address issues recently. That said, can you use rust-peg in standalone mode as a genrule?

https://github.com/kevinmehall/rust-peg#as-a-standalone-code-generator

If it isn't clear how you might do that I can elaborate.