misc changes

ci/sembr/src/main.rs

@@ -14,6 +14,9 @@ struct Cli {
     /// Modify files that do not comply
     #[arg(long)]
     overwrite: bool,
+    /// This reflows file even when it complies (with one sentence per line)
+    #[arg(long)]
+    reflow_harder: bool,
     /// Applies to lines that are to be split
     #[arg(long, default_value_t = 100)]
     line_length_limit: usize,
@@ -24,7 +27,7 @@ static REGEX_IGNORE_END: LazyLock<Regex> =
 static REGEX_IGNORE_LINK_TARGETS: LazyLock<Regex> =
     LazyLock::new(|| Regex::new(r"^\[.+\]: ").unwrap());
 static REGEX_SPLIT: LazyLock<Regex> =
-    LazyLock::new(|| Regex::new(r"([^\.\d\-\*]\.|[^r\~]\?|!)\s").unwrap());
+    LazyLock::new(|| Regex::new(r"([^\.\d\-]\.|[^r\~]\?|!)\s").unwrap());
 // list elements, numbered (1.) or not  (- and *)
 static REGEX_LIST_ENTRY: LazyLock<Regex> =
     LazyLock::new(|| Regex::new(r"^\s*(\d\.|\-|\*|\d\))\s+").unwrap());
@@ -45,7 +48,10 @@ fn main() -> Result<()> {
             continue;
         }
         let old = fs::read_to_string(&path)?;
-        let new = comply(&old);
+        let mut new = comply(&old);
+        if cli.reflow_harder {
+            new = lengthen_lines(&new, cli.line_length_limit)
+        }
         if new == old {
             compliant.push(path.clone());
         } else if cli.overwrite {
@@ -385,7 +391,6 @@ encountering a cycle doesn't mean that we would get an infinite proof tree.
 }
 
 #[test]
-#[ignore]
 fn should_split() {
     let original = "the queries that we do, as well as the **query DAG**. The";
     let expected = "the queries that we do, as well as the **query DAG**.\nThe\n";

src/appendix/code-index.md

@@ -23,7 +23,7 @@ Item            |  Kind    | Short description           | Chapter            |
 `SourceFile` | struct | Part of the `SourceMap`. Maps AST nodes to their source code for a single source file. Was previously called FileMap | [The parser] | [compiler/rustc_span/src/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.SourceFile.html)
 `SourceMap` | struct | Maps AST nodes to their source code. It is composed of `SourceFile`s. Was previously called CodeMap | [The parser] | [compiler/rustc_span/src/source_map.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/source_map/struct.SourceMap.html)
 `Span` | struct  | A location in the user's source code, used for error reporting primarily | [Emitting Diagnostics] | [compiler/rustc_span/src/span_encoding.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html)
-`rustc_ast::token_stream::TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [compiler/rustc_ast/src/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/tokenstream/struct.TokenStream.html)
+`TokenStream` | struct | An abstract sequence of tokens, organized into `TokenTree`s | [The parser], [Macro expansion] | [compiler/rustc_ast/src/tokenstream.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/tokenstream/struct.TokenStream.html)
 `TraitDef` | struct | This struct contains a trait's definition with type information | [The `ty` modules] |  [compiler/rustc_middle/src/ty/trait_def.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/trait_def/struct.TraitDef.html)
 `TraitRef` | struct | The combination of a trait and its input types (e.g. `P0: Trait<P1...Pn>`) | [Trait Solving: Goals and Clauses]  |  [compiler/rustc_middle/src/ty/sty.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/type.TraitRef.html)
 `Ty<'tcx>` | struct | This is the internal representation of a type used for type checking | [Type checking] | [compiler/rustc_middle/src/ty/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.Ty.html)

src/building/bootstrapping/what-bootstrapping-does.md

@@ -16,12 +16,12 @@ See [bootstrap/README.md][bootstrap-internals] to read about bootstrap internals
 
 Compiling `rustc` is done in stages.
 Here's a diagram, adapted from Jynn
-Nelson's [talk on bootstrapping][rustconf22-talk] at RustConf 2022, with
-detailed explanations below.
+Nelson's [talk on bootstrapping][rustconf22-talk] at RustConf 2022,
+with detailed explanations below.
 
 The `A`, `B`, `C`, and `D` show the ordering of the stages of bootstrapping.
-<span style="background-color: lightblue; color: black">Blue</span> nodes are
-downloaded, <span style="background-color: yellow; color: black">yellow</span>
+<span style="background-color: lightblue; color: black">Blue</span> nodes are downloaded,
+<span style="background-color: yellow; color: black">yellow</span>
 nodes are built with the `stage0` compiler, and <span style="background-color:
 lightgreen; color: black">green</span> nodes are built with the `stage1` compiler.
 
@@ -54,8 +54,8 @@ The precompiled stage0 compiler is then used only to compile [`src/bootstrap`] a
 with precompiled stage0 std.
 
 Note that to build the stage1 compiler we use the precompiled stage0 compiler and std.
-Therefore, to use a compiler with a std that is freshly built from the tree, you need to
-build the stage2 compiler.
+Therefore, to use a compiler with a std that is freshly built from the tree,
+you need to build the stage2 compiler.
 
 There are two concepts at play here: a compiler (with its set of dependencies) and its
 'target' or 'object' libraries (`std` and `rustc`).
@@ -77,8 +77,8 @@ The `stage1` compiler itself was built by precompiled `stage0` compiler and std
 and hence not by the source in your working directory.
 This means that the ABI
 generated by the `stage0` compiler may not match the ABI that would have been made
-by the `stage1` compiler, which can cause problems for dynamic libraries, tests
-and tools using `rustc_private`.
+by the `stage1` compiler, which can cause problems for dynamic libraries,
+tests, and tools using `rustc_private`.
 
 Note that the `proc_macro` crate avoids this issue with a `C` FFI layer called
 `proc_macro::bridge`, allowing it to be used with `stage1`.
@@ -119,28 +119,28 @@ For more information about stages, [see below](#understanding-stages-of-bootstra
 
 ## Complications of bootstrapping
 
-Since the build system uses the current beta compiler to build a `stage1`
-bootstrapping compiler, the compiler source code can't use some features until
+Since the build system uses the current beta compiler to build a `stage1` bootstrapping compiler,
+the compiler source code can't use some features until
 they reach beta (because otherwise the beta compiler doesn't support them).
-On the other hand, for [compiler intrinsics][intrinsics] and internal features, the
-features _have_ to be used.
+On the other hand, for [compiler intrinsics] and internal features,
+the features _have_ to be used.
 Additionally, the compiler makes heavy use of `nightly` features (`#![feature(...)]`).
 How can we resolve this problem?
 
 There are two methods used:
 
-1. The build system sets `--cfg bootstrap` when building with `stage0`, so we
-   can use `cfg(not(bootstrap))` to only use features when built with `stage1`.
-   Setting `--cfg bootstrap` in this way is used for features that were just
-   stabilized, which require `#![feature(...)]` when built with `stage0`, but not for `stage1`.
+1. The build system sets `--cfg bootstrap` when building with `stage0`,
+   so we can use `cfg(not(bootstrap))` to only use features when built with `stage1`.
+   Setting `--cfg bootstrap` in this way is used for features that were just stabilized,
+   which require `#![feature(...)]` when built with `stage0`, but not for `stage1`.
 2. The build system sets `RUSTC_BOOTSTRAP=1`.
    This special variable means to
    _break the stability guarantees_ of Rust: allowing use of `#![feature(...)]`
    with a compiler that's not `nightly`.
    _Setting `RUSTC_BOOTSTRAP=1` should never be used except when bootstrapping the compiler._
 
 [boot]: https://en.wikipedia.org/wiki/Bootstrapping_(compilers)
-[intrinsics]: ../../appendix/glossary.md#intrinsic
+[compiler intrinsics]: ../../appendix/glossary.md#intrinsic
 
 ## Understanding stages of bootstrap
 
@@ -152,8 +152,8 @@ The convention `./x` uses is that:
 
 - A `--stage N` flag means to run the stage N compiler (`stageN/rustc`).
 - A "stage N artifact" is a build artifact that is _produced_ by the stage N compiler.
-- The stage N+1 compiler is assembled from stage N *artifacts*. This process is
-  called _uplifting_.
+- The stage N+1 compiler is assembled from stage N *artifacts*.
+  This process is called _uplifting_.
 
 #### Build artifacts
 
@@ -186,10 +186,10 @@ Build artifacts include, but are not limited to:
 - `./x test --stage 0 compiler/rustc` builds the compiler but runs no tests:
   it's running `cargo test -p rustc`, but `cargo` doesn't understand Rust's tests.
   You shouldn't need to use this; use `test` instead (without arguments).
-- `./x build --stage 0 compiler/rustc` builds the compiler, but does not build
-  `libstd` or even `libcore`.
-  Most of the time, you'll want `./x build library`
-  instead, which allows compiling programs without needing to define lang items.
+- `./x build --stage 0 compiler/rustc` builds the compiler,
+  but does not build `libstd` or even `libcore`.
+  Most of the time, you'll want `./x build library` instead,
+  which allows compiling programs without needing to define lang items.
 
 ### Building vs. running
 
@@ -199,8 +199,8 @@ will later be uplifted to be the stage1 compiler_.
 In each stage besides 0, two major steps are performed:
 
 1. `std` is compiled by the stage N compiler.
-2. That `std` is linked to programs built by the stage N compiler, including the
-   stage N artifacts (stage N+1 compiler).
+2. That `std` is linked to programs built by the stage N compiler,
+   including the stage N artifacts (stage N+1 compiler).
 
 This is somewhat intuitive if one thinks of the stage N artifacts as "just"
 another program we are building with the stage N compiler: `build --stage N
@@ -211,32 +211,32 @@ compiler/rustc` is linking the stage N artifacts to the `std` built by the stage
 Note that there are two `std` libraries in play here:
 
 1. The library _linked_ to `stageN/rustc`, which was built by stage N-1 (stage N-1 `std`)
-2. The library _used to compile programs_ with `stageN/rustc`, which was built
-   by stage N (stage N `std`).
+2. The library _used to compile programs_ with `stageN/rustc`,
+   which was built by stage N (stage N `std`).
 
 Stage N `std` is pretty much necessary for any useful work with the stage N compiler.
 Without it, you can only compile programs with `#![no_core]` -- not terribly useful!
 
 The reason these need to be different is because they aren't necessarily
-ABI-compatible: there could be new layout optimizations, changes to `MIR`, or
-other changes to Rust metadata on `nightly` that aren't present in beta.
+ABI-compatible: there could be new layout optimizations, changes to `MIR`,
+or other changes to Rust metadata on `nightly` that aren't present in beta.
 
 This is also where `--keep-stage 1 library/std` comes into play.
 Since most changes to the compiler don't actually change the ABI, once you've produced a
 `std` in `stage1`, you can probably just reuse it with a different compiler.
 If the ABI hasn't changed, you're good to go; no need to spend time recompiling that `std`.
 The flag `--keep-stage` simply instructs the build script to assume
-the previous compile is fine and copies those artifacts into the appropriate
-place, skipping the `cargo` invocation.
+the previous compile is fine and copies those artifacts into the appropriate place,
+skipping the `cargo` invocation.
 
 ### Cross-compiling rustc
 
 *Cross-compiling* is the process of compiling code that will run on another architecture.
 For instance, you might want to build an ARM version of rustc using an x86 machine.
 Building `stage2` `std` is different when you are cross-compiling.
 
-This is because `./x` uses the following logic: if `HOST` and `TARGET` are the
-same, it will reuse `stage1` `std` for `stage2`!
+This is because `./x` uses the following logic: if `HOST` and `TARGET` are the same,
+it will reuse `stage1` `std` for `stage2`!
 This is sound because `stage1`
 `std` was compiled with the `stage1` compiler, i.e. a compiler using the source
 code you currently have checked out.
@@ -250,23 +250,23 @@ So, the `stage2` compiler has to recompile `std` for the target.
 
 ### What is a 'sysroot'?
 
-When you build a project with `cargo`, the build artifacts for dependencies are
-normally stored in `target/debug/deps`.
-This only contains dependencies `cargo`
-knows about; in particular, it doesn't have the standard library.
+When you build a project with `cargo`,
+the build artifacts for dependencies are normally stored in `target/debug/deps`.
+This only contains dependencies `cargo` knows about; in particular,
+it doesn't have the standard library.
 Where do `std` or `proc_macro` come from?
-They come from the **sysroot**, the root of a number
-of directories where the compiler loads build artifacts at runtime.
-The `sysroot` doesn't just store the standard library, though - it includes anything
-that needs to be loaded at runtime.
+They come from the **sysroot**,
+the root of a number of directories where the compiler loads build artifacts at runtime.
+The `sysroot` doesn't just store the standard library though;
+it includes anything that needs to be loaded at runtime.
 That includes (but is not limited to):
 
 - Libraries `libstd`/`libtest`/`libproc_macro`.
 - Compiler crates themselves, when using `rustc_private`.
   In-tree, these are always present; out-of-tree, you need to install `rustc-dev` with `rustup`.
 - Shared object file `libLLVM.so` for the LLVM project.
-  In-tree, this is either built from source or downloaded from CI; out-of-tree, you need to install
-  `llvm-tools-preview` with `rustup`.
+  In-tree, this is either built from source or downloaded from CI; out-of-tree,
+  you need to install `llvm-tools-preview` with `rustup`.
 
 All the artifacts listed so far are *compiler* runtime dependencies.
 You can see them with `rustc --print sysroot`:
@@ -293,16 +293,16 @@ libcompiler_builtins-ef2408da76957905.rlib
 
 Directory `lib/rustlib/` includes libraries like `hashbrown` and `cfg_if`, which
 are not part of the public API of the standard library, but are used to implement it.
-Also,`lib/rustlib/` is part of the search path for linkers, but
-`lib` will never be part of the search path.
+Also,`lib/rustlib/` is part of the search path for linkers,
+but `lib` will never be part of the search path.
 
 #### `-Z force-unstable-if-unmarked`
 
 Since `lib/rustlib/` is part of the search path we have to be careful about
 which crates are included in it.
 In particular, all crates except for the
-standard library are built with the flag `-Z force-unstable-if-unmarked`, which
-means that you have to use `#![feature(rustc_private)]` in order to load it (as
+standard library are built with the flag `-Z force-unstable-if-unmarked`,
+which means that you have to use `#![feature(rustc_private)]` in order to load it (as
 opposed to the standard library, which is always available).
 
 The `-Z force-unstable-if-unmarked` flag has a variety of purposes to help
@@ -323,12 +323,12 @@ This flag has the following effects:
 - Marks the crate as "`unstable`" with the `rustc_private` feature if it is not
   itself marked as `stable` or `unstable`.
 - Allows these crates to access other forced-unstable crates without any need for attributes.
-  Normally, a crate would need a `#![feature(rustc_private)]`
-  attribute to use other `unstable` crates.
-  However, that would make it
-  impossible for a crate from crates.io to access its own dependencies since
-  that crate won't have a `feature(rustc_private)` attribute, but *everything*
-  is compiled with `-Z force-unstable-if-unmarked`.
+  Normally,
+  a crate would need a `#![feature(rustc_private)]` attribute to use other `unstable` crates.
+  However,
+  that would make it impossible for a crate from crates.io to access its own dependencies since
+  that crate won't have a `feature(rustc_private)` attribute,
+  but *everything* is compiled with `-Z force-unstable-if-unmarked`.
 
 Code which does not use `-Z force-unstable-if-unmarked` should include the
 `#![feature(rustc_private)]` crate attribute to access these forced-unstable crates.
@@ -348,10 +348,10 @@ You can find more discussion about sysroots in:
 Conveniently `./x` allows you to pass stage-specific flags to `rustc` and
 `cargo` when bootstrapping.
 The `RUSTFLAGS_BOOTSTRAP` environment variable is
-passed as `RUSTFLAGS` to the bootstrap stage (`stage0`), and
-`RUSTFLAGS_NOT_BOOTSTRAP` is passed when building artifacts for later stages.
-`RUSTFLAGS` will work, but also affects the build of `bootstrap` itself, so it
-will be rare to want to use it.
+passed as `RUSTFLAGS` to the bootstrap stage (`stage0`),
+and `RUSTFLAGS_NOT_BOOTSTRAP` is passed when building artifacts for later stages.
+`RUSTFLAGS` will work, but also affects the build of `bootstrap` itself,
+so it will be rare to want to use it.
 Finally, `MAGIC_EXTRA_RUSTFLAGS` bypasses the
 `cargo` cache to pass flags to rustc without recompiling all dependencies.
 
@@ -370,19 +370,19 @@ which you can use to find out the options accepted by the runner.
 ## Environment Variables
 
 During bootstrapping, there are a bunch of compiler-internal environment variables that are used.
-If you are trying to run an intermediate version of
-`rustc`, sometimes you may need to set some of these environment variables manually.
+If you are trying to run an intermediate version of `rustc`,
+sometimes you may need to set some of these environment variables manually.
 Otherwise, you get an error like the following:
 
 ```text
 thread 'main' panicked at 'RUSTC_STAGE was not set: NotPresent', library/core/src/result.rs:1165:5
 ```
 
-If `./stageN/bin/rustc` gives an error about environment variables, that usually
-means something is quite wrong -- such as you're trying to compile `rustc` or
+If `./stageN/bin/rustc` gives an error about environment variables,
+that usually means something is quite wrong -- such as you're trying to compile `rustc` or
 `std` or something which depends on environment variables.
-In the unlikely case that you actually need to invoke `rustc` in such a situation, you can tell the
-bootstrap shim to print all `env` variables by adding `-vvv` to your `x` command.
+In the unlikely case that you actually need to invoke `rustc` in such a situation,
+you can tell the bootstrap shim to print all `env` variables by adding `-vvv` to your `x` command.
 
 Finally, bootstrap makes use of the [cc-rs crate] which has [its own
 method][env-vars] of configuring `C` compilers and `C` flags via environment variables.
@@ -392,8 +392,8 @@ method][env-vars] of configuring `C` compilers and `C` flags via environment var
 
 ## Clarification of build command's `stdout`
 
-In this part, we will investigate the build command's `stdout` in an action
-(similar, but more detailed and complete documentation compare to topic above).
+In this part, we will investigate the build command's `stdout` in an action (similar,
+but more detailed and complete documentation compared to topic above).
 When you execute `x build --dry-run` command, the build output will be something like the following:
 
 ```text
@@ -410,8 +410,8 @@ Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
 
 ### Building stage0 {std,compiler} artifacts
 
-These steps use the provided (downloaded, usually) compiler to compile the local
-Rust source into libraries we can use.
+These steps use the provided (downloaded,
+usually) compiler to compile the local Rust source into libraries we can use.
 
 ### Copying stage0 \{std,rustc\}