Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental: Add Derive Proc-Macro Caching #129102

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
3db813e
deps(rustc_expand): Add `rustc_middle` as a dep
futile Aug 11, 2024
2507253
refactor(rustc_expand): Take &SyntaxExtension instead of only &*Kind
futile Aug 11, 2024
0733fbd
wip: So yeah, tests pass, but still `eval_always` (not far from disk …
futile Aug 12, 2024
a4b675c
fix: Prevent double-enter with same `&mut ExtCtxt` (which would be UB)
futile Aug 12, 2024
209fd8a
chore: Remove old `allow`
futile Aug 12, 2024
fe36059
chore: Remove unneeded code, adapt TODOs for pr-time
futile Aug 12, 2024
eb99b01
wip(test): Run with `-- --nocapture --verbose` and `invoked` should n…
futile Aug 12, 2024
fc3bc29
wip: Activate, test (and fix) on_disk caching for proc-macro expansions!
futile Aug 12, 2024
075853d
chore: Adjust timing outputs
futile Aug 12, 2024
7a1b2f5
feat: Add `-Zcache-all-derive-macros` to use/bypass cache
futile Aug 12, 2024
d781c4b
wip: try also hashing query output
futile Aug 12, 2024
a81aaed
prepare for PR (tidy should pass, caching=yes by default for rustc-perf)
futile Aug 14, 2024
91c0d1b
wip: Undo unnecessary refactoring
futile Aug 19, 2024
8521154
rebase + fix unreachable pub-items, inline some CONTEXT-things
futile Sep 9, 2024
17dcadf
chore: Run `x fmt` after rebase, should probably squash.
futile Nov 9, 2024
35beb09
fix: comment + unused warning
futile Nov 16, 2024
382c5d0
style: move import to correct place
futile Nov 16, 2024
b71c42d
chore: remove old comment
futile Nov 16, 2024
a359e3c
don't flatten proc macro output, let's see how it goes
futile Nov 16, 2024
f768697
only retrieve expn_data once
futile Nov 16, 2024
bb3bddf
fix: Ensure incremental compilation is also enabled
futile Dec 8, 2024
77944c9
refactor!: Rename `cache-all-derive-macros` option to `cache-proc-mac…
futile Dec 28, 2024
de0fad9
nit: remove comment
futile Dec 28, 2024
d6c2527
refactor: Require `ecx` less
futile Dec 28, 2024
00cff94
refactor: Only call invoc_id.expn_data() once, inline some `res`
futile Dec 28, 2024
abcc4a2
refactor: Don't enter context if not caching (don't need it then)
futile Dec 28, 2024
e93beac
refactor: Move derive_macro_expansion.rs to end of proc_macro.rs
futile Dec 28, 2024
cd82e38
fix: Make non-context expansion actually work
futile Dec 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: comment + unused warning
futile committed Dec 24, 2024
commit 35beb09850763026a6011e7a6b277cb45f709f64
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ extern crate proc_macro;
use proc_macro::TokenStream;

#[proc_macro_derive(Nothing)]
pub fn derive(input: TokenStream) -> TokenStream {
pub fn derive(_input: TokenStream) -> TokenStream {
eprintln!("invoked");

return r#"
12 changes: 7 additions & 5 deletions tests/incremental/derive_macro_expansion/proc_macro_unchanged.rs
Original file line number Diff line number Diff line change
@@ -2,13 +2,13 @@
// HOWEVER, this test can currently only be checked manually,
// by running it (through compiletest) with `-- --nocapture --verbose`.
// The proc-macro (for `Nothing`) prints a message to stderr when invoked,
// and this message should only be present during the second invocation
// (which has `cfail2` set via cfg).
// and this message should only be present during the first invocation,
// because the cached result should be used for the second invocation.
// FIXME(pr-time): Properly have the test check this, but how? UI-test that tests for `.stderr`?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why do you want to test stderr output, incremental tests have directives for testing invalidation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's what the "Properly have the test check this, but how?" was for, I simply didn't see how I can check this. There are directives, but I didn't figure out a way to make them work for this case. Because the directive needs to be attached to the proc macro output I think, but that was weird somehow. I will try again and see what exactly the problem was with that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried again, but couldn't figure out something that works, i.e., that succeeds with proc-macro caching enabled, but fails with proc-macro caching disabled.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that the #[rustc_clean] attribute needs to be used instead of #[rustc_partition_codegened] because the incremental system eventually figures out that the second run of derive(Nothing) effectively produced the same data as the first one, so the codegen doesn't need to happen again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, run-make tests (tests/run-make) allow to run the compiler twice and look at stderr.
There are even some examples of running the compiler in incremental mode there (search by "incremental").


//@ aux-build:derive_nothing.rs
//@ revisions:cfail1 cfail2
//@ compile-flags: -Z query-dep-graph
//@ compile-flags: -Z query-dep-graph -Zcache-all-derive-macros=true
//@ build-pass

#![feature(rustc_attrs)]
@@ -17,10 +17,12 @@
#![crate_type = "rlib"]

#![rustc_partition_codegened(module="proc_macro_unchanged-foo", cfg="cfail1")]
#![rustc_partition_codegened(module="proc_macro_unchanged-foo", cfg="cfail2")]
// #![rustc_partition_codegened(module="proc_macro_unchanged-foo", cfg="cfail2")]

// `foo::nothing_mod` is created by the derive macro and doesn't change
#![rustc_partition_reused(module="proc_macro_unchanged-foo", cfg="cfail2")]
// BUG: this yields the same result with `-Zcache-all-derive-macros=false` (i.e., uncached),
// not sure how to do this correctly.
#![rustc_partition_reused(module="proc_macro_unchanged-foo-nothing_mod", cfg="cfail2")]

#[macro_use]
extern crate derive_nothing;