Grammar file not found on workspace

[librelois]

I use pest in a crate of a workspace, so this crate has no src folder. I was forced to create an empty src subfolder in my crate so that pest could find my file, it's not clean.

It's easy to reproduce this bug, create a new workspace project and try to import pest grammar file from a crate of workspace.

[dragostis]

The question is, how does one solve this? It makes sense for the grammar file to be in src since that will make sure it's included in the crate.

[CAD97]

I've been using pest in a workspace, and this hasn't been a problem for me.

What I might see being a problem is a binary crate using bin instead of src. But for my own workspace, I've had no issues; the src path is the src of the crate I'm working in.

Could you provide an exact reproduction steps? (Also, OS may play a role here, unfortunately.)

[dragostis]

I think he means that it will not work where no src is present within a crate?

[librelois]

I have created an example repository that reproduces the bug.
If you clone the repository it does not compile. If you create an empty folder named src in my-usb-crate then it compiles.

https://github.com/librelois/workspace_use_pest

[librelois]

The question is, how does one solve this?

My opinion is that the path should start from the root folder of the crate, and not from the sub-folder src :)

[CAD97]

Oh, that project is set up in a very unconventional manner.

Most workspaces are set up where each subcrate has a src directory containing the sources, rather than redefining where the entry point is in Cargo.toml. (They also can have other "magic" directories, such as tests and examples.)

Your problem isn't that you're in a workspace, but that you've defined a different lib.path.

A common workspace layout

/Cargo.toml
/derive/Cargo.toml
/derive/src/lib.rs
/derive/tests/test001.rs
/traits/Cargo.toml
/traits/src/lib.rs

---

Any change to where the #[grammar = ""] path starts from is technically a breaking change. The path is currently defined to be relative to the src directory, so for full strict backwards compatibility, that needs to be the default for a relative path.

One solution is to change the path to instead of being relative from src, to be relative from the directory of lib.path (bin.path?).

Another is to take a page from the JS world and use "anchored" paths: #[grammar = "@/src/pest/grammar.pest"], where @ is basically a "path macro" for %CARGO_MANIFEST_DIR%.

And another (poor?) solution is just to make the path %CARGO_MANIFEST_DIR%/src/../{} work even when src doesn't exist (and "just" collapsing isn't necessarily correct in the face of symlinks, thus why Rust's canonicalize currently uses the filesystem instead of being "simple").

[cramertj]

It's worth mentioning that #360 was necessary because not all users are using cargo (and are missing the env vars relied upon here). IMO what this really wants is the ability to use include! and have it pre-expand before proc macros run. I'd consider opening an issue on rust-lang/rust, since this seems like a pretty reasonable thing to want and AFAIK there aren't a lot of great options at the moment.

[dragostis]

@cramertj, expending include!s before procedural macros are run would probably not be a stop-gap solution for our Cargo dependency. IMO the needed API is just being able to get the path of the file where the procedural macro resides. I've thought about opening an issue on Rust a while ago, but the procedural macros API was still heavily discussed.

[cramertj]

@dragostis

IMO the needed API is just being able to get the path of the file where the procedural macro resides.

Where the macro resides? What are you referring to here? The path of file that invokes the macro? The reason I suggested include! is that I was imagining a sort of two-stage expansion where the first expands to an include! invocation that pulls in the grammar source and the second processes that grammar, generating the resulting Rust code. This would remove the need for the extra include_str! in order to hint to cargo about the dep-tracking, but it does involve extra steps. A proc_macro API for "resolve this Span containing a path to a TokenStream from the pointed-to-file" would perhaps be a more direct way of achieving this.

[dragostis]

Sorry, yes, I meant the invocation.

The API can be as simple as to only resolve the Span to string or a byte buffer.

[cramertj]

string or a byte buffer.

I'd think you'd still want a TokenStream so that you can refer to Spans in the file and do error reporting using the normal API.

[seeekr]

Just want to leave a message here: Would be really nice to be able to use a path that does not start from the src/ dir. Use case: I'm using pest from a build.rs and thus don't really want my grammar file(s) be located in src/, which should contain only runtime code.