[perf] Do not reuse internal TokenStream in proc_macro::TokenStream

[petrochenkov]

Use a separate type instead, so that we need to convert from one TokenStream to another and back on proc macro boundaries.
I want to check how much it affects performance.

This PR also implements the suggestion to censor token jointness at proc macro boundary from #114571 (since we now have such a boundary).
This PR is also somewhat related to #101419.

I don't like the result and will likely close this after the perf test.

[rustbot]

r? @compiler-errors

(rustbot has picked a reviewer for you, use r? to override)

[petrochenkov]

@bors try @rust-timer queue

[bors]

⌛ Trying commit d6b704c with merge 5a750ec...

[rust-log-analyzer]

The job x86_64-gnu-llvm-15 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

.........................................................F............................

failures:

---- parse::tests::string_to_tts_1 stdout ----
thread 'parse::tests::string_to_tts_1' panicked at compiler/rustc_expand/src/parse/tests.rs:145:9:
assertion `left == right` failed
  left: TokenStream([Token(Token { kind: Ident("fn", false), span: Span { lo: BytePos(0), hi: BytePos(2), ctxt: #0 } }, Alone), Token(Token { kind: Ident("a", false), span: Span { lo: BytePos(3), hi: BytePos(4), ctxt: #0 } }, Alone), Delimited(DelimSpan { open: Span { lo: BytePos(5), hi: BytePos(6), ctxt: #0 }, close: Span { lo: BytePos(13), hi: BytePos(14), ctxt: #0 } }, Parenthesis, TokenStream([Token(Token { kind: Ident("b", false), span: Span { lo: BytePos(6), hi: BytePos(7), ctxt: #0 } }, Alone), Token(Token { kind: Colon, span: Span { lo: BytePos(8), hi: BytePos(9), ctxt: #0 } }, Alone), Token(Token { kind: Ident("i32", false), span: Span { lo: BytePos(10), hi: BytePos(13), ctxt: #0 } }, Joint)])), Delimited(DelimSpan { open: Span { lo: BytePos(15), hi: BytePos(16), ctxt: #0 }, close: Span { lo: BytePos(20), hi: BytePos(21), ctxt: #0 } }, Brace, TokenStream([Token(Token { kind: Ident("b", false), span: Span { lo: BytePos(17), hi: BytePos(18), ctxt: #0 } }, Joint), Token(Token { kind: Semi, span: Span { lo: BytePos(18), hi: BytePos(19), ctxt: #0 } }, Alone)]))])
 right: TokenStream([Token(Token { kind: Ident("fn", false), span: Span { lo: BytePos(0), hi: BytePos(2), ctxt: #0 } }, Alone), Token(Token { kind: Ident("a", false), span: Span { lo: BytePos(3), hi: BytePos(4), ctxt: #0 } }, Alone), Delimited(DelimSpan { open: Span { lo: BytePos(5), hi: BytePos(6), ctxt: #0 }, close: Span { lo: BytePos(13), hi: BytePos(14), ctxt: #0 } }, Parenthesis, TokenStream([Token(Token { kind: Ident("b", false), span: Span { lo: BytePos(6), hi: BytePos(7), ctxt: #0 } }, Alone), Token(Token { kind: Colon, span: Span { lo: BytePos(8), hi: BytePos(9), ctxt: #0 } }, Alone), Token(Token { kind: Ident("i32", false), span: Span { lo: BytePos(10), hi: BytePos(13), ctxt: #0 } }, Alone)])), Delimited(DelimSpan { open: Span { lo: BytePos(15), hi: BytePos(16), ctxt: #0 }, close: Span { lo: BytePos(20), hi: BytePos(21), ctxt: #0 } }, Brace, TokenStream([Token(Token { kind: Ident("b", false), span: Span { lo: BytePos(17), hi: BytePos(18), ctxt: #0 } }, Joint), Token(Token { kind: Semi, span: Span { lo: BytePos(18), hi: BytePos(19), ctxt: #0 } }, Alone)]))])


failures:
failures:
error: test failed, to rerun pass `-p rustc_expand --lib`

test result: FAILED. 85 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.83ms

Build completed unsuccessfully in 0:19:53

[bors]

☀️ Try build successful - checks-actions
Build commit: 5a750ec (5a750eccd37dfc18760b39d6abcb8dabd3ea7f7b)

[rust-timer]

Finished benchmarking commit (5a750ec): comparison URL.

Overall result: ❌ regressions - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.1%	[0.2%, 1.4%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.0%	[0.6%, 3.3%]	2
Regressions ❌ (secondary)	2.8%	[2.5%, 3.2%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.6%	[-3.6%, -3.6%]	1
All ❌✅ (primary)	2.0%	[0.6%, 3.3%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 622.801s -> 623.308s (0.08%)
Artifact size: 270.63 MiB -> 270.66 MiB (0.01%)