Print token::Interpolated with token stream pretty printing.

[nnethercote]

This is a step towards removing token::Interpolated (#124141). It unavoidably changes the output of the stringify! macro, generally for the better.

r? @petrochenkov

[nnethercote]

@bors try @rust-timer queue

[bors]

⌛ Trying commit f7ce1ac with merge 53af78d...

[nnethercote]

Full details are in the individual commits.

@dtolnay has agreed to update the tests/ui/macros/stringify.rs test and move the AST pretty printing checks into a new test. That could be done before or after this PR is merged.

A crater run should be done before this merges, to make sure the stringify! changes don't cause problems for existing code.

Performance is unlikely to be affected, but let's confirm that:
@bors try @rust-timer queue

[bors]

⌛ Trying commit 9c693ad with merge e07dc21...

[bors]

☀️ Try build successful - checks-actions
Build commit: e07dc21 (e07dc21eff3c4c268d22684a47778eff58db7d17)

[dtolnay]

While I can see how this PR is a necessary step toward removing token::Interpolated, I think it could use some better explanation why removing token::Interpolated is a necessary step toward solving #67062 (it's a necessary stepping stone) before we would commit to this change.

It unavoidably changes the output of the stringify! macro, generally for the better.

This is the opposite of what I expect, and I consider stringify! changes to be a casualty of choosing this approach. We'll need to add a separate stringify_expr! macro to core. The AST pretty printer is always going to be more capable at stringifying expressions than the tokenstream pretty printer.

stringify! is used by the assert and dbg family of macros in the ecosystem (although apparently not by std's assert, which is a built-in and continues to use the AST pretty-printer). The prototypical usage is https://docs.rs/failure/0.1.8/failure/macro.ensure.html

#[macro_export]
macro_rules! ensure {
    ($cond:expr) => {
        if let false = $cond {
            return Err($crate::err_msg(format!("{}", stringify!($cond))));
        }
    };
}

Some changes picked up by https://github.com/dtolnay/anyhow's test suite from this PR:

- Error::msg::<<str as ToOwned>::Owned>.t(1) == 2
+ Error ::msg ::<< str as ToOwned >::Owned >.t (1) == 2

- f::<1>() != ()
+ f ::<1 >() != ()

- f::<-1>() != ()
+ f ::<- 1 >() != ()

- E::U::<> > E::U::<u8>
+ E ::U ::<> > E ::U ::<u8 >

- E::U::<u8> > E::U
+ E ::U ::<u8 > > E ::U

- b\"hmm\"[1] == b'c'
+ b\"hmm\" [1] == b'c'

- result? == 2
+ result ? == 2

- f as for<'a> fn() as usize * 0 != 0
+ f as for<'a > fn() as usize * 0 != 0

- &0 as &dyn EqDebug<i32, Assoc = bool> != &0
+ &0 as &dyn EqDebug <i32, Assoc = bool > != &0

- PhantomData as PhantomData<<i32 as ToOwned>::Owned> != PhantomData
+ PhantomData as PhantomData << i32 as ToOwned >::Owned > != PhantomData

- 0 as int!(...) != 0
+ 0 as int !(...) != 0

- 0 as int![...] != 0
+ 0 as int ![...] != 0

- 0 as int! { ... } != 0
+ 0 as int ! { ... } != 0

- if let -1..=1 = 0 { 0 } else { 1 } == 1
+ if let -1 ..=1 = 0 { 0 } else { 1 } == 1

[rust-timer]

Finished benchmarking commit (e07dc21): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.3%]	3
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.2%, 0.3%]	3

Max RSS (memory usage)

Results (primary 0.2%, secondary -5.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.6%	[0.6%, 0.6%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.2%	[-0.2%, -0.2%]	1
Improvements ✅ (secondary)	-5.0%	[-5.0%, -5.0%]	1
All ❌✅ (primary)	0.2%	[-0.2%, 0.6%]	2

Cycles

Results (primary -0.3%, secondary 0.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.3%	[3.3%, 3.3%]	1
Regressions ❌ (secondary)	2.4%	[2.1%, 2.6%]	3
Improvements ✅ (primary)	-1.5%	[-1.6%, -1.4%]	3
Improvements ✅ (secondary)	-1.5%	[-2.4%, -0.9%]	4
All ❌✅ (primary)	-0.3%	[-1.6%, 3.3%]	4

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 669.726s -> 668.949s (-0.12%)
Artifact size: 316.16 MiB -> 315.99 MiB (-0.06%)

[nnethercote]

The AST pretty printer is always going to be more capable at stringifying expressions than the tokenstream pretty printer.

We can discuss what "more capable" means, but there are cases visible in the stringify.rs diff where I consider the tokenstream pretty printer to do a better job.

original	AST pp	tokenstream pp	difference
f :: < u8>( )	f::<u8>()	f :: < u8>()	AST pp loses more whitespace
f(true,)	f(true)	f(true,)	AST loses the ,
for<> fn()	fn()	for<> fn()	AST loses the for<>
pub macro stringify() {}	pub macro stringify { () => {} }	pub macro stringify() {}	AST inserts () => {}

Some changes picked up by https://github.com/dtolnay/anyhow's test suite from this PR:

How do I run that suite? I tried cargo test test_ensure. I can't reproduce the failures, and the failures you listed are surprising because the tokenstream pretty printer usually produces much nicer output than that.

[dtolnay]

How do I run that suite? I tried cargo test test_ensure.

cargo test --test test_ensure. I built rustc from 9c693ad (the current state of this PR) and ran cargo +stage2 test --test test_ensure using dtolnay/anyhow@fecdbbb to expose more of the failures simultaneously.

[nnethercote]

#[macro_export]
macro_rules! ensure {
    ($cond:expr) => {
        if let false = $cond {
            return Err($crate::err_msg(format!("{}", stringify!($cond))));
        }
    };
}

AFAICT this is the cfg(doc) version of that macro. The cfg(not(doc)) version is much more complex. I don't have time to look at this today, I'll take a look next week.

[dtolnay]

Minimal repro:

macro_rules! repro {
    ($foo:ident $colons:tt $bar:ident) => {
        stringify_expr!($foo $colons $bar)
    };
}

macro_rules! stringify_expr {
    ($expr:expr) => {
        stringify!($expr)
    };
}

fn main() {
    println!("{}", repro!(foo::bar));
}

- foo::bar
+ foo ::bar

📌 Commit f7d49fd has been approved by petrochenkov

It is now in the queue for this repository.

⌛ Testing commit f7d49fd with merge d0227c6...

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing d0227c6 to master...

Finished benchmarking commit (d0227c6): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.4%]	9
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.2%, 0.4%]	9

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results (secondary 2.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.1%	[2.1%, 2.1%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 676.994s -> 678.403s (0.21%)
Artifact size: 320.03 MiB -> 320.07 MiB (0.01%)

@nnethercote @petrochenkov I didn't see any discussion of the perf regressions seen here. It seems like most libc benchmarks were negatively impacted while no other benchmarks were. I'm not quite sure why that would be.

I'm a little surprised, I wouldn't have thought pretty-printing was on the performance critical path for anything. The new code could be slightly slower. The fact that it's a single benchmark, and small regressions, makes me not too worried. And the relevant code (nonterminal_to_string) is going to be eliminated as part of #124141 anyway.

@rustbot label: +perf-regression-triaged