#
cargo/core/compiler/fingerprint/
mod.rs

    1//! Tracks changes to determine if something needs to be recompiled.
    2//!
    3//! This module implements change-tracking so that Cargo can know whether or
    4//! not something needs to be recompiled. A Cargo [`Unit`] can be either "dirty"
    5//! (needs to be recompiled) or "fresh" (it does not need to be recompiled).
    6//!
    7//! ## Mechanisms affecting freshness
    8//!
    9//! There are several mechanisms that influence a Unit's freshness:
    10//!
    11//! - The [`Fingerprint`] is a hash, saved to the filesystem in the
    12//!   `.fingerprint` directory, that tracks information about the Unit. If the
    13//!   fingerprint is missing (such as the first time the unit is being
    14//!   compiled), then the unit is dirty. If any of the fingerprint fields
    15//!   change (like the name of the source file), then the Unit is considered
    16//!   dirty.
    17//!
    18//!   The `Fingerprint` also tracks the fingerprints of all its dependencies,
    19//!   so a change in a dependency will propagate the "dirty" status up.
    20//!
    21//! - Filesystem mtime tracking is also used to check if a unit is dirty.
    22//!   See the section below on "Mtime comparison" for more details. There
    23//!   are essentially two parts to mtime tracking:
    24//!
    25//!   1. The mtime of a Unit's output files is compared to the mtime of all
    26//!      its dependencies' output file mtimes (see
    27//!      [`check_filesystem`]). If any output is missing, or is
    28//!      older than a dependency's output, then the unit is dirty.
    29//!   2. The mtime of a Unit's source files is compared to the mtime of its
    30//!      dep-info file in the fingerprint directory (see [`find_stale_file`]).
    31//!      The dep-info file is used as an anchor to know when the last build of
    32//!      the unit was done. See the "dep-info files" section below for more
    33//!      details. If any input files are missing, or are newer than the
    34//!      dep-info, then the unit is dirty.
    35//!
    36//!  - Alternatively if you're using the unstable feature `checksum-freshness`
    37//!    mtimes are ignored entirely in favor of comparing first the file size, and
    38//!    then the checksum with a known prior value emitted by rustc. Only nightly
    39//!    rustc will emit the needed metadata at the time of writing. This is dependent
    40//!    on the unstable feature `-Z checksum-hash-algorithm`.
    41//!
    42//! Note: Fingerprinting is not a perfect solution. Filesystem mtime tracking
    43//! is notoriously imprecise and problematic. Only a small part of the
    44//! environment is captured. This is a balance of performance, simplicity, and
    45//! completeness. Sandboxing, hashing file contents, tracking every file
    46//! access, environment variable, and network operation would ensure more
    47//! reliable and reproducible builds at the cost of being complex, slow, and
    48//! platform-dependent.
    49//!
    50//! ## Fingerprints and [`UnitHash`]s
    51//!
    52//! [`Metadata`] tracks several [`UnitHash`]s, including
    53//! [`Metadata::unit_id`], [`Metadata::c_metadata`], and [`Metadata::c_extra_filename`].
    54//! See its documentation for more details.
    55//!
    56//! NOTE: Not all output files are isolated via filename hashes (like dylibs).
    57//! The fingerprint directory uses a hash, but sometimes units share the same
    58//! fingerprint directory (when they don't have Metadata) so care should be
    59//! taken to handle this!
    60//!
    61//! Fingerprints and [`UnitHash`]s are similar, and track some of the same things.
    62//! [`UnitHash`]s contains information that is required to keep Units separate.
    63//! The Fingerprint includes additional information that should cause a
    64//! recompile, but it is desired to reuse the same filenames. A comparison
    65//! of what is tracked:
    66//!
    67//! Value                                      | Fingerprint | `Metadata::unit_id` [^8] | `Metadata::c_metadata`
    68//! -------------------------------------------|-------------|--------------------------|-----------------------
    69//! rustc                                      | ✓           | ✓                        | ✓
    70//! [`Profile`]                                | ✓           | ✓                        | ✓
    71//! `cargo rustc` extra args                   | ✓           | ✓[^7]                    |
    72//! [`CompileMode`]                            | ✓           | ✓                        | ✓
    73//! Target Name                                | ✓           | ✓                        | ✓
    74//! `TargetKind` (bin/lib/etc.)                | ✓           | ✓                        | ✓
    75//! Enabled Features                           | ✓           | ✓                        | ✓
    76//! Declared Features                          | ✓           |                          |
    77//! Immediate dependency’s hashes              | ✓[^1]       | ✓                        | ✓
    78//! [`CompileKind`] (host/target)              | ✓           | ✓                        | ✓
    79//! `__CARGO_DEFAULT_LIB_METADATA`[^4]         |             | ✓                        | ✓
    80//! `package_id`                               |             | ✓                        | ✓
    81//! Target src path relative to ws             | ✓           |                          |
    82//! Target flags (test/bench/for_host/edition) | ✓           |                          |
    83//! -C incremental=… flag                      | ✓           |                          |
    84//! mtime of sources                           | ✓[^3]       |                          |
    85//! RUSTFLAGS/RUSTDOCFLAGS                     | ✓           | ✓[^7]                    |
    86//! [`Lto`] flags                              | ✓           | ✓                        | ✓
    87//! config settings[^5]                        | ✓           |                          |
    88//! `is_std`                                   |             | ✓                        | ✓
    89//! `[lints]` table[^6]                        | ✓           |                          |
    90//! `[lints.rust.unexpected_cfgs.check-cfg]`   | ✓           |                          |
    91//!
    92//! [^1]: Bin dependencies are not included.
    93//!
    94//! [^3]: See below for details on mtime tracking.
    95//!
    96//! [^4]: `__CARGO_DEFAULT_LIB_METADATA` is set by rustbuild to embed the
    97//!        release channel (bootstrap/stable/beta/nightly) in libstd.
    98//!
    99//! [^5]: Config settings that are not otherwise captured anywhere else.
    100//!       Currently, this is only `doc.extern-map`.
    101//!
    102//! [^6]: Via [`Manifest::lint_rustflags`][crate::core::Manifest::lint_rustflags]
    103//!
    104//! [^7]: extra-flags and RUSTFLAGS are conditionally excluded when `--remap-path-prefix` is
    105//!       present to avoid breaking build reproducibility while we wait for trim-paths
    106//!
    107//! [^8]: including `-Cextra-filename`
    108//!
    109//! When deciding what should go in the Metadata vs the Fingerprint, consider
    110//! that some files (like dylibs) do not have a hash in their filename. Thus,
    111//! if a value changes, only the fingerprint will detect the change (consider,
    112//! for example, swapping between different features). Fields that are only in
    113//! Metadata generally aren't relevant to the fingerprint because they
    114//! fundamentally change the output (like target vs host changes the directory
    115//! where it is emitted).
    116//!
    117//! ## Fingerprint files
    118//!
    119//! Fingerprint information is stored in the
    120//! `target/{debug,release}/.fingerprint/` directory. Each Unit is stored in a
    121//! separate directory. Each Unit directory contains:
    122//!
    123//! - A file with a 16 hex-digit hash. This is the Fingerprint hash, used for
    124//!   quick loading and comparison.
    125//! - A `.json` file that contains details about the Fingerprint. This is only
    126//!   used to log details about *why* a fingerprint is considered dirty.
    127//!   `CARGO_LOG=cargo::core::compiler::fingerprint=trace cargo build` can be
    128//!   used to display this log information.
    129//! - A "dep-info" file which is a translation of rustc's `*.d` dep-info files
    130//!   to a Cargo-specific format that tweaks file names and is optimized for
    131//!   reading quickly.
    132//! - An `invoked.timestamp` file whose filesystem mtime is updated every time
    133//!   the Unit is built. This is used for capturing the time when the build
    134//!   starts, to detect if files are changed in the middle of the build. See
    135//!   below for more details.
    136//!
    137//! Note that some units are a little different. A Unit for *running* a build
    138//! script or for `rustdoc` does not have a dep-info file (it's not
    139//! applicable). Build script `invoked.timestamp` files are in the build
    140//! output directory.
    141//!
    142//! ## Fingerprint calculation
    143//!
    144//! After the list of Units has been calculated, the Units are added to the
    145//! [`JobQueue`]. As each one is added, the fingerprint is calculated, and the
    146//! dirty/fresh status is recorded. A closure is used to update the fingerprint
    147//! on-disk when the Unit successfully finishes. The closure will recompute the
    148//! Fingerprint based on the updated information. If the Unit fails to compile,
    149//! the fingerprint is not updated.
    150//!
    151//! Fingerprints are cached in the [`BuildRunner`]. This makes computing
    152//! Fingerprints faster, but also is necessary for properly updating
    153//! dependency information. Since a Fingerprint includes the Fingerprints of
    154//! all dependencies, when it is updated, by using `Arc` clones, it
    155//! automatically picks up the updates to its dependencies.
    156//!
    157//! ### dep-info files
    158//!
    159//! Cargo has several kinds of "dep info" files:
    160//!
    161//! * dep-info files generated by `rustc`.
    162//! * Fingerprint dep-info files translated from the first one.
    163//! * dep-info for external build system integration.
    164//! * Unstable `-Zbinary-dep-depinfo`.
    165//!
    166//! #### `rustc` dep-info files
    167//!
    168//! Cargo passes the `--emit=dep-info` flag to `rustc` so that `rustc` will
    169//! generate a "dep info" file (with the `.d` extension). This is a
    170//! Makefile-like syntax that includes all of the source files used to build
    171//! the crate. This file is used by Cargo to know which files to check to see
    172//! if the crate will need to be rebuilt. Example:
    173//!
    174//! ```makefile
    175//! /path/to/target/debug/deps/cargo-b6219d178925203d: src/bin/main.rs src/bin/cargo/cli.rs # … etc.
    176//! ```
    177//!
    178//! #### Fingerprint dep-info files
    179//!
    180//! After `rustc` exits successfully, Cargo will read the first kind of dep
    181//! info file and translate it into a binary format that is stored in the
    182//! fingerprint directory ([`translate_dep_info`]).
    183//!
    184//! These are used to quickly scan for any changed files. The mtime of the
    185//! fingerprint dep-info file itself is used as the reference for comparing the
    186//! source files to determine if any of the source files have been modified
    187//! (see [below](#mtime-comparison) for more detail).
    188//!
    189//! Note that Cargo parses the special `# env-var:...` comments in dep-info
    190//! files to learn about environment variables that the rustc compile depends on.
    191//! Cargo then later uses this to trigger a recompile if a referenced env var
    192//! changes (even if the source didn't change).
    193//! This also includes env vars generated from Cargo metadata like `CARGO_PKG_DESCRIPTION`.
    194//! (See [`crate::core::manifest::ManifestMetadata`]
    195//!
    196//! #### dep-info files for build system integration.
    197//!
    198//! There is also a third dep-info file. Cargo will extend the file created by
    199//! rustc with some additional information and saves this into the output
    200//! directory. This is intended for build system integration. See the
    201//! [`output_depinfo`] function for more detail.
    202//!
    203//! #### -Zbinary-dep-depinfo
    204//!
    205//! `rustc` has an experimental flag `-Zbinary-dep-depinfo`. This causes
    206//! `rustc` to include binary files (like rlibs) in the dep-info file. This is
    207//! primarily to support rustc development, so that Cargo can check the
    208//! implicit dependency to the standard library (which lives in the sysroot).
    209//! We want Cargo to recompile whenever the standard library rlib/dylibs
    210//! change, and this is a generic mechanism to make that work.
    211//!
    212//! ### Mtime comparison
    213//!
    214//! The use of modification timestamps is the most common way a unit will be
    215//! determined to be dirty or fresh between builds. There are many subtle
    216//! issues and edge cases with mtime comparisons. This gives a high-level
    217//! overview, but you'll need to read the code for the gritty details. Mtime
    218//! handling is different for different unit kinds. The different styles are
    219//! driven by the [`Fingerprint::local`] field, which is set based on the unit
    220//! kind.
    221//!
    222//! The status of whether or not the mtime is "stale" or "up-to-date" is
    223//! stored in [`Fingerprint::fs_status`].
    224//!
    225//! All units will compare the mtime of its newest output file with the mtimes
    226//! of the outputs of all its dependencies. If any output file is missing,
    227//! then the unit is stale. If any dependency is newer, the unit is stale.
    228//!
    229//! #### Normal package mtime handling
    230//!
    231//! [`LocalFingerprint::CheckDepInfo`] is used for checking the mtime of
    232//! packages. It compares the mtime of the input files (the source files) to
    233//! the mtime of the dep-info file (which is written last after a build is
    234//! finished). If the dep-info is missing, the unit is stale (it has never
    235//! been built). The list of input files comes from the dep-info file. See the
    236//! section above for details on dep-info files.
    237//!
    238//! Also note that although registry and git packages use [`CheckDepInfo`], none
    239//! of their source files are included in the dep-info (see
    240//! [`translate_dep_info`]), so for those kinds no mtime checking is done
    241//! (unless `-Zbinary-dep-depinfo` is used). Repository and git packages are
    242//! static, so there is no need to check anything.
    243//!
    244//! When a build is complete, the mtime of the dep-info file in the
    245//! fingerprint directory is modified to rewind it to the time when the build
    246//! started. This is done by creating an `invoked.timestamp` file when the
    247//! build starts to capture the start time. The mtime is rewound to the start
    248//! to handle the case where the user modifies a source file while a build is
    249//! running. Cargo can't know whether or not the file was included in the
    250//! build, so it takes a conservative approach of assuming the file was *not*
    251//! included, and it should be rebuilt during the next build.
    252//!
    253//! #### Rustdoc mtime handling
    254//!
    255//! Rustdoc does not emit a dep-info file, so Cargo currently has a relatively
    256//! simple system for detecting rebuilds. [`LocalFingerprint::Precalculated`] is
    257//! used for rustdoc units. For registry packages, this is the package
    258//! version. For git packages, it is the git hash. For path packages, it is
    259//! a string of the mtime of the newest file in the package.
    260//!
    261//! There are some known bugs with how this works, so it should be improved at
    262//! some point.
    263//!
    264//! #### Build script mtime handling
    265//!
    266//! Build script mtime handling runs in different modes. There is the "old
    267//! style" where the build script does not emit any `rerun-if` directives. In
    268//! this mode, Cargo will use [`LocalFingerprint::Precalculated`]. See the
    269//! "rustdoc" section above how it works.
    270//!
    271//! In the new-style, each `rerun-if` directive is translated to the
    272//! corresponding [`LocalFingerprint`] variant. The [`RerunIfChanged`] variant
    273//! compares the mtime of the given filenames against the mtime of the
    274//! "output" file.
    275//!
    276//! Similar to normal units, the build script "output" file mtime is rewound
    277//! to the time just before the build script is executed to handle mid-build
    278//! modifications.
    279//!
    280//! ## Considerations for inclusion in a fingerprint
    281//!
    282//! Over time we've realized a few items which historically were included in
    283//! fingerprint hashings should not actually be included. Examples are:
    284//!
    285//! * Modification time values. We strive to never include a modification time
    286//!   inside a `Fingerprint` to get hashed into an actual value. While
    287//!   theoretically fine to do, in practice this causes issues with common
    288//!   applications like Docker. Docker, after a layer is built, will zero out
    289//!   the nanosecond part of all filesystem modification times. This means that
    290//!   the actual modification time is different for all build artifacts, which
    291//!   if we tracked the actual values of modification times would cause
    292//!   unnecessary recompiles. To fix this we instead only track paths which are
    293//!   relevant. These paths are checked dynamically to see if they're up to
    294//!   date, and the modification time doesn't make its way into the fingerprint
    295//!   hash.
    296//!
    297//! * Absolute path names. We strive to maintain a property where if you rename
    298//!   a project directory Cargo will continue to preserve all build artifacts
    299//!   and reuse the cache. This means that we can't ever hash an absolute path
    300//!   name. Instead we always hash relative path names and the "root" is passed
    301//!   in at runtime dynamically. Some of this is best effort, but the general
    302//!   idea is that we assume all accesses within a crate stay within that
    303//!   crate.
    304//!
    305//! These are pretty tricky to test for unfortunately, but we should have a good
    306//! test suite nowadays and lord knows Cargo gets enough testing in the wild!
    307//!
    308//! ## Build scripts
    309//!
    310//! The *running* of a build script ([`CompileMode::RunCustomBuild`]) is treated
    311//! significantly different than all other Unit kinds. It has its own function
    312//! for calculating the Fingerprint ([`calculate_run_custom_build`]) and has some
    313//! unique considerations. It does not track the same information as a normal
    314//! Unit. The information tracked depends on the `rerun-if-changed` and
    315//! `rerun-if-env-changed` statements produced by the build script. If the
    316//! script does not emit either of these statements, the Fingerprint runs in
    317//! "old style" mode where an mtime change of *any* file in the package will
    318//! cause the build script to be re-run. Otherwise, the fingerprint *only*
    319//! tracks the individual "rerun-if" items listed by the build script.
    320//!
    321//! The "rerun-if" statements from a *previous* build are stored in the build
    322//! output directory in a file called `output`. Cargo parses this file when
    323//! the Unit for that build script is prepared for the [`JobQueue`]. The
    324//! Fingerprint code can then use that information to compute the Fingerprint
    325//! and compare against the old fingerprint hash.
    326//!
    327//! Care must be taken with build script Fingerprints because the
    328//! [`Fingerprint::local`] value may be changed after the build script runs
    329//! (such as if the build script adds or removes "rerun-if" items).
    330//!
    331//! Another complication is if a build script is overridden. In that case, the
    332//! fingerprint is the hash of the output of the override.
    333//!
    334//! ## Special considerations
    335//!
    336//! Registry dependencies do not track the mtime of files. This is because
    337//! registry dependencies are not expected to change (if a new version is
    338//! used, the Package ID will change, causing a rebuild). Cargo currently
    339//! partially works with Docker caching. When a Docker image is built, it has
    340//! normal mtime information. However, when a step is cached, the nanosecond
    341//! portions of all files is zeroed out. Currently this works, but care must
    342//! be taken for situations like these.
    343//!
    344//! HFS on macOS only supports 1 second timestamps. This causes a significant
    345//! number of problems, particularly with Cargo's testsuite which does rapid
    346//! builds in succession. Other filesystems have various degrees of
    347//! resolution.
    348//!
    349//! Various weird filesystems (such as network filesystems) also can cause
    350//! complications. Network filesystems may track the time on the server
    351//! (except when the time is set manually such as with
    352//! `filetime::set_file_times`). Not all filesystems support modifying the
    353//! mtime.
    354//!
    355//! See the [`A-rebuild-detection`] label on the issue tracker for more.
    356//!
    357//! [`check_filesystem`]: Fingerprint::check_filesystem
    358//! [`Metadata`]: crate::core::compiler::Metadata
    359//! [`Metadata::unit_id`]: crate::core::compiler::Metadata::unit_id
    360//! [`Metadata::c_metadata`]: crate::core::compiler::Metadata::c_metadata
    361//! [`Metadata::c_extra_filename`]: crate::core::compiler::Metadata::c_extra_filename
    362//! [`UnitHash`]: crate::core::compiler::UnitHash
    363//! [`Profile`]: crate::core::profiles::Profile
    364//! [`CompileMode`]: crate::core::compiler::CompileMode
    365//! [`Lto`]: crate::core::compiler::Lto
    366//! [`CompileKind`]: crate::core::compiler::CompileKind
    367//! [`JobQueue`]: super::job_queue::JobQueue
    368//! [`output_depinfo`]: super::output_depinfo()
    369//! [`CheckDepInfo`]: LocalFingerprint::CheckDepInfo
    370//! [`RerunIfChanged`]: LocalFingerprint::RerunIfChanged
    371//! [`CompileMode::RunCustomBuild`]: crate::core::compiler::CompileMode::RunCustomBuild
    372//! [`A-rebuild-detection`]: https://github.com/rust-lang/cargo/issues?q=is%3Aissue+is%3Aopen+label%3AA-rebuild-detection
    373
    374mod dep_info;
    375mod dirty_reason;
    376mod rustdoc;
    377
    378use std::collections::hash_map::{Entry, HashMap};
    379use std::env;
    380use std::ffi::OsString;
    381use std::fs;
    382use std::fs::File;
    383use std::hash::{self, Hash, Hasher};
    384use std::io::{self};
    385use std::path::{Path, PathBuf};
    386use std::sync::{Arc, Mutex};
    387use std::time::SystemTime;
    388
    389use anyhow::Context as _;
    390use anyhow::format_err;
    391use cargo_util::paths;
    392use filetime::FileTime;
    393use serde::de;
    394use serde::ser;
    395use serde::{Deserialize, Serialize};
    396use tracing::{debug, info};
    397
    398use crate::core::Package;
    399use crate::core::compiler::unit_graph::UnitDep;
    400use crate::util;
    401use crate::util::errors::CargoResult;
    402use crate::util::interning::InternedString;
    403use crate::util::log_message::LogMessage;
    404use crate::util::{StableHasher, internal, path_args};
    405use crate::{CARGO_ENV, GlobalContext};
    406
    407use super::BuildContext;
    408use super::BuildRunner;
    409use super::FileFlavor;
    410use super::Job;
    411use super::Unit;
    412use super::UnitIndex;
    413use super::Work;
    414use super::custom_build::BuildDeps;
    415
    416pub use self::dep_info::Checksum;
    417pub use self::dep_info::parse_dep_info;
    418pub use self::dep_info::parse_rustc_dep_info;
    419pub use self::dep_info::translate_dep_info;
    420pub use self::dirty_reason::DirtyReason;
    421pub use self::rustdoc::RustdocFingerprint;
    422
    423/// Result of comparing fingerprints between the current and previous builds.
    424enum FingerprintComparison {
    425    /// The unit does not need rebuilding.
    426    Fresh,
    427    /// The unit needs rebuilding.
    428    Dirty {
    429        /// The reason why the unit is dirty.
    430        reason: DirtyReason,
    431    },
    432}
    433
    434/// Determines if a [`Unit`] is up-to-date, and if not prepares necessary work to
    435/// update the persisted fingerprint.
    436///
    437/// This function will inspect `Unit`, calculate a fingerprint for it, and then
    438/// return an appropriate [`Job`] to run. The returned `Job` will be a noop if
    439/// `unit` is considered "fresh", or if it was previously built and cached.
    440/// Otherwise the `Job` returned will write out the true fingerprint to the
    441/// filesystem, to be executed after the unit's work has completed.
    442///
    443/// The `force` flag is a way to force the `Job` to be "dirty", or always
    444/// update the fingerprint. **Beware using this flag** because it does not
    445/// transitively propagate throughout the dependency graph, it only forces this
    446/// one unit which is very unlikely to be what you want unless you're
    447/// exclusively talking about top-level units.
    448#[tracing::instrument(
    449    skip(build_runner, unit),
    450    fields(package_id = %unit.pkg.package_id(), target = unit.target.name())
    451)]
    452pub fn prepare_target(
    453    build_runner: &mut BuildRunner<'_, '_>,
    454    unit: &Unit,
    455    force: bool,
    456) -> CargoResult<Job> {
    457    let bcx = build_runner.bcx;
    458    let loc = build_runner.files().fingerprint_file_path(unit, "");
    459
    460    debug!("fingerprint at: {}", loc.display());
    461
    462    // Figure out if this unit is up to date. After calculating the fingerprint
    463    // compare it to an old version, if any, and attempt to print diagnostic
    464    // information about failed comparisons to aid in debugging.
    465    let fingerprint = calculate(build_runner, unit)?;
    466    let mtime_on_use = build_runner.bcx.gctx.cli_unstable().mtime_on_use;
    467    let dirty_reason = match compare_old_fingerprint(unit, &loc, &*fingerprint, mtime_on_use, force)
    468    {
    469        FingerprintComparison::Fresh => None,
    470        FingerprintComparison::Dirty { reason } => Some(reason),
    471    };
    472
    473    if let Some(logger) = bcx.logger {
    474        let index = bcx.unit_to_index[unit];
    475        let mut cause = None;
    476        let status = match dirty_reason.as_ref() {
    477            Some(reason) if reason.is_fresh_build() => util::log_message::FingerprintStatus::New,
    478            Some(reason) => {
    479                cause = Some(reason.clone());
    480                util::log_message::FingerprintStatus::Dirty
    481            }
    482            None => util::log_message::FingerprintStatus::Fresh,
    483        };
    484        logger.log(LogMessage::UnitFingerprint {
    485            index,
    486            status,
    487            cause,
    488        });
    489    }
    490
    491    let Some(dirty_reason) = dirty_reason else {
    492        return Ok(Job::new_fresh());
    493    };
    494
    495    // We're going to rebuild, so ensure the source of the crate passes all
    496    // verification checks before we build it.
    497    //
    498    // The `Source::verify` method is intended to allow sources to execute
    499    // pre-build checks to ensure that the relevant source code is all
    500    // up-to-date and as expected. This is currently used primarily for
    501    // directory sources which will use this hook to perform an integrity check
    502    // on all files in the source to ensure they haven't changed. If they have
    503    // changed then an error is issued.
    504    let source_id = unit.pkg.package_id().source_id();
    505    let sources = bcx.packages.sources();
    506    let source = sources
    507        .get(source_id)
    508        .ok_or_else(|| internal("missing package source"))?;
    509    source.verify(unit.pkg.package_id())?;
    510
    511    // Clear out the old fingerprint file if it exists. This protects when
    512    // compilation is interrupted leaving a corrupt file. For example, a
    513    // project with a lib.rs and integration test (two units):
    514    //
    515    // 1. Build the library and integration test.
    516    // 2. Make a change to lib.rs (NOT the integration test).
    517    // 3. Build the integration test, hit Ctrl-C while linking. With gcc, this
    518    //    will leave behind an incomplete executable (zero size, or partially
    519    //    written). NOTE: The library builds successfully, it is the linking
    520    //    of the integration test that we are interrupting.
    521    // 4. Build the integration test again.
    522    //
    523    // Without the following line, then step 3 will leave a valid fingerprint
    524    // on the disk. Then step 4 will think the integration test is "fresh"
    525    // because:
    526    //
    527    // - There is a valid fingerprint hash on disk (written in step 1).
    528    // - The mtime of the output file (the corrupt integration executable
    529    //   written in step 3) is newer than all of its dependencies.
    530    // - The mtime of the integration test fingerprint dep-info file (written
    531    //   in step 1) is newer than the integration test's source files, because
    532    //   we haven't modified any of its source files.
    533    //
    534    // But the executable is corrupt and needs to be rebuilt. Clearing the
    535    // fingerprint at step 3 ensures that Cargo never mistakes a partially
    536    // written output as up-to-date.
    537    if loc.exists() {
    538        // Truncate instead of delete so that compare_old_fingerprint will
    539        // still log the reason for the fingerprint failure instead of just
    540        // reporting "failed to read fingerprint" during the next build if
    541        // this build fails.
    542        paths::write(&loc, b"")?;
    543    }
    544
    545    let write_fingerprint = if unit.mode.is_run_custom_build() {
    546        // For build scripts the `local` field of the fingerprint may change
    547        // while we're executing it. For example it could be in the legacy
    548        // "consider everything a dependency mode" and then we switch to "deps
    549        // are explicitly specified" mode.
    550        //
    551        // To handle this movement we need to regenerate the `local` field of a
    552        // build script's fingerprint after it's executed. We do this by
    553        // using the `build_script_local_fingerprints` function which returns a
    554        // thunk we can invoke on a foreign thread to calculate this.
    555        let build_script_outputs = Arc::clone(&build_runner.build_script_outputs);
    556        let metadata = build_runner.get_run_build_script_metadata(unit);
    557        let (gen_local, _overridden) = build_script_local_fingerprints(build_runner, unit)?;
    558        let output_path = build_runner.build_explicit_deps[unit]
    559            .build_script_output
    560            .clone();
    561        Work::new(move |_| {
    562            let outputs = build_script_outputs.lock().unwrap();
    563            let output = outputs
    564                .get(metadata)
    565                .expect("output must exist after running");
    566            let deps = BuildDeps::new(&output_path, Some(output));
    567
    568            // FIXME: it's basically buggy that we pass `None` to `call_box`
    569            // here. See documentation on `build_script_local_fingerprints`
    570            // below for more information. Despite this just try to proceed and
    571            // hobble along if it happens to return `Some`.
    572            if let Some(new_local) = (gen_local)(&deps, None)? {
    573                *fingerprint.local.lock().unwrap() = new_local;
    574            }
    575
    576            write_fingerprint(&loc, &fingerprint)
    577        })
    578    } else {
    579        Work::new(move |_| write_fingerprint(&loc, &fingerprint))
    580    };
    581
    582    Ok(Job::new_dirty(write_fingerprint, dirty_reason))
    583}
    584
    585/// Dependency edge information for fingerprints. This is generated for each
    586/// dependency and is stored in a [`Fingerprint`].
    587#[derive(Clone)]
    588struct DepFingerprint {
    589    /// The hash of the package id that this dependency points to
    590    pkg_id: u64,
    591    /// The crate name we're using for this dependency, which if we change we'll
    592    /// need to recompile!
    593    name: InternedString,
    594    /// Whether or not this dependency is flagged as a public dependency or not.
    595    public: bool,
    596    /// Whether or not this dependency is an rmeta dependency or a "full"
    597    /// dependency. In the case of an rmeta dependency our dependency edge only
    598    /// actually requires the rmeta from what we depend on, so when checking
    599    /// mtime information all files other than the rmeta can be ignored.
    600    only_requires_rmeta: bool,
    601    /// The dependency's fingerprint we recursively point to, containing all the
    602    /// other hash information we'd otherwise need.
    603    fingerprint: Arc<Fingerprint>,
    604}
    605
    606/// A fingerprint can be considered to be a "short string" representing the
    607/// state of a world for a package.
    608///
    609/// If a fingerprint ever changes, then the package itself needs to be
    610/// recompiled. Inputs to the fingerprint include source code modifications,
    611/// compiler flags, compiler version, etc. This structure is not simply a
    612/// `String` due to the fact that some fingerprints cannot be calculated lazily.
    613///
    614/// Path sources, for example, use the mtime of the corresponding dep-info file
    615/// as a fingerprint (all source files must be modified *before* this mtime).
    616/// This dep-info file is not generated, however, until after the crate is
    617/// compiled. As a result, this structure can be thought of as a fingerprint
    618/// to-be. The actual value can be calculated via [`hash_u64()`], but the operation
    619/// may fail as some files may not have been generated.
    620///
    621/// Note that dependencies are taken into account for fingerprints because rustc
    622/// requires that whenever an upstream crate is recompiled that all downstream
    623/// dependents are also recompiled. This is typically tracked through
    624/// [`DependencyQueue`], but it also needs to be retained here because Cargo can
    625/// be interrupted while executing, losing the state of the [`DependencyQueue`]
    626/// graph.
    627///
    628/// [`hash_u64()`]: crate::core::compiler::fingerprint::Fingerprint::hash_u64
    629/// [`DependencyQueue`]: crate::util::DependencyQueue
    630#[derive(Serialize, Deserialize)]
    631pub struct Fingerprint {
    632    /// Hash of the version of `rustc` used.
    633    rustc: u64,
    634    /// Sorted list of cfg features enabled.
    635    features: String,
    636    /// Sorted list of all the declared cfg features.
    637    declared_features: String,
    638    /// Hash of the `Target` struct, including the target name,
    639    /// package-relative source path, edition, etc.
    640    target: u64,
    641    /// Hash of the [`Profile`], [`CompileMode`], and any extra flags passed via
    642    /// `cargo rustc` or `cargo rustdoc`.
    643    ///
    644    /// [`Profile`]: crate::core::profiles::Profile
    645    /// [`CompileMode`]: crate::core::compiler::CompileMode
    646    profile: u64,
    647    /// Hash of the path to the base source file. This is relative to the
    648    /// workspace root for path members, or absolute for other sources.
    649    path: u64,
    650    /// Fingerprints of dependencies.
    651    deps: Vec<DepFingerprint>,
    652    /// Information about the inputs that affect this Unit (such as source
    653    /// file mtimes or build script environment variables).
    654    local: Mutex<Vec<LocalFingerprint>>,
    655    /// Cached hash of the [`Fingerprint`] struct. Used to improve performance
    656    /// for hashing.
    657    #[serde(skip)]
    658    memoized_hash: Mutex<Option<u64>>,
    659    /// RUSTFLAGS/RUSTDOCFLAGS environment variable value (or config value).
    660    rustflags: Vec<String>,
    661    /// Hash of various config settings that change how things are compiled.
    662    config: u64,
    663    /// The rustc target. This is only relevant for `.json` files, otherwise
    664    /// the metadata hash segregates the units.
    665    compile_kind: u64,
    666    /// Unit index for this fingerprint, used for tracing cascading rebuilds.
    667    /// Not persisted to disk as indices can change between builds.
    668    #[serde(skip)]
    669    index: UnitIndex,
    670    /// Description of whether the filesystem status for this unit is up to date
    671    /// or should be considered stale.
    672    #[serde(skip)]
    673    fs_status: FsStatus,
    674    /// Files, relative to `target_root`, that are produced by the step that
    675    /// this `Fingerprint` represents. This is used to detect when the whole
    676    /// fingerprint is out of date if this is missing, or if previous
    677    /// fingerprints output files are regenerated and look newer than this one.
    678    #[serde(skip)]
    679    outputs: Vec<PathBuf>,
    680}
    681
    682/// Indication of the status on the filesystem for a particular unit.
    683#[derive(Clone, Default, Debug, Serialize, Deserialize)]
    684#[serde(tag = "fs_status", rename_all = "kebab-case")]
    685pub enum FsStatus {
    686    /// This unit is to be considered stale, even if hash information all
    687    /// matches.
    688    #[default]
    689    Stale,
    690
    691    /// File system inputs have changed (or are missing), or there were
    692    /// changes to the environment variables that affect this unit. See
    693    /// the variants of [`StaleItem`] for more information.
    694    StaleItem(StaleItem),
    695
    696    /// A dependency was stale.
    697    StaleDependency {
    698        unit: UnitIndex,
    699        #[serde(with = "serde_file_time")]
    700        dep_mtime: FileTime,
    701        #[serde(with = "serde_file_time")]
    702        max_mtime: FileTime,
    703    },
    704
    705    /// A dependency's fingerprint was stale.
    706    StaleDepFingerprint { unit: UnitIndex },
    707
    708    /// This unit is up-to-date. All outputs and their corresponding mtime are
    709    /// listed in the payload here for other dependencies to compare against.
    710    #[serde(skip)]
    711    UpToDate { mtimes: HashMap<PathBuf, FileTime> },
    712}
    713
    714impl FsStatus {
    715    fn up_to_date(&self) -> bool {
    716        match self {
    717            FsStatus::UpToDate { .. } => true,
    718            FsStatus::Stale
    719            | FsStatus::StaleItem(_)
    720            | FsStatus::StaleDependency { .. }
    721            | FsStatus::StaleDepFingerprint { .. } => false,
    722        }
    723    }
    724}
    725
    726mod serde_file_time {
    727    use filetime::FileTime;
    728    use serde::Deserialize;
    729    use serde::Serialize;
    730
    731    /// Serialize FileTime as milliseconds with nano.
    732    pub(super) fn serialize<S>(ft: &FileTime, s: S) -> Result<S::Ok, S::Error>
    733    where
    734        S: serde::Serializer,
    735    {
    736        let secs_as_millis = ft.unix_seconds() as f64 * 1000.0;
    737        let nanos_as_millis = ft.nanoseconds() as f64 / 1_000_000.0;
    738        (secs_as_millis + nanos_as_millis).serialize(s)
    739    }
    740
    741    /// Deserialize FileTime from milliseconds with nano.
    742    pub(super) fn deserialize<'de, D>(d: D) -> Result<FileTime, D::Error>
    743    where
    744        D: serde::Deserializer<'de>,
    745    {
    746        let millis = f64::deserialize(d)?;
    747        let secs = (millis / 1000.0) as i64;
    748        let nanos = ((millis % 1000.0) * 1_000_000.0) as u32;
    749        Ok(FileTime::from_unix_time(secs, nanos))
    750    }
    751}
    752
    753impl Serialize for DepFingerprint {
    754    fn serialize<S>(&self, ser: S) -> Result<S::Ok, S::Error>
    755    where
    756        S: ser::Serializer,
    757    {
    758        (
    759            &self.pkg_id,
    760            &self.name,
    761            &self.public,
    762            &self.fingerprint.hash_u64(),
    763        )
    764            .serialize(ser)
    765    }
    766}
    767
    768impl<'de> Deserialize<'de> for DepFingerprint {
    769    fn deserialize<D>(d: D) -> Result<DepFingerprint, D::Error>
    770    where
    771        D: de::Deserializer<'de>,
    772    {
    773        let (pkg_id, name, public, hash) = <(u64, String, bool, u64)>::deserialize(d)?;
    774        Ok(DepFingerprint {
    775            pkg_id,
    776            name: name.into(),
    777            public,
    778            fingerprint: Arc::new(Fingerprint {
    779                memoized_hash: Mutex::new(Some(hash)),
    780                ..Fingerprint::new()
    781            }),
    782            // This field is never read since it's only used in
    783            // `check_filesystem` which isn't used by fingerprints loaded from
    784            // disk.
    785            only_requires_rmeta: false,
    786        })
    787    }
    788}
    789
    790/// A `LocalFingerprint` represents something that we use to detect direct
    791/// changes to a `Fingerprint`.
    792///
    793/// This is where we track file information, env vars, etc. This
    794/// `LocalFingerprint` struct is hashed and if the hash changes will force a
    795/// recompile of any fingerprint it's included into. Note that the "local"
    796/// terminology comes from the fact that it only has to do with one crate, and
    797/// `Fingerprint` tracks the transitive propagation of fingerprint changes.
    798///
    799/// Note that because this is hashed its contents are carefully managed. Like
    800/// mentioned in the above module docs, we don't want to hash absolute paths or
    801/// mtime information.
    802///
    803/// Also note that a `LocalFingerprint` is used in `check_filesystem` to detect
    804/// when the filesystem contains stale information (based on mtime currently).
    805/// The paths here don't change much between compilations but they're used as
    806/// inputs when we probe the filesystem looking at information.
    807#[derive(Debug, Serialize, Deserialize, Hash)]
    808enum LocalFingerprint {
    809    /// This is a precalculated fingerprint which has an opaque string we just
    810    /// hash as usual. This variant is primarily used for rustdoc where we
    811    /// don't have a dep-info file to compare against.
    812    ///
    813    /// This is also used for build scripts with no `rerun-if-*` statements, but
    814    /// that's overall a mistake and causes bugs in Cargo. We shouldn't use this
    815    /// for build scripts.
    816    Precalculated(String),
    817
    818    /// This is used for crate compilations. The `dep_info` file is a relative
    819    /// path anchored at `target_root(...)` to the dep-info file that Cargo
    820    /// generates (which is a custom serialization after parsing rustc's own
    821    /// `dep-info` output).
    822    ///
    823    /// The `dep_info` file, when present, also lists a number of other files
    824    /// for us to look at. If any of those files are newer than this file then
    825    /// we need to recompile.
    826    ///
    827    /// If the `checksum` bool is true then the `dep_info` file is expected to
    828    /// contain file checksums instead of file mtimes.
    829    CheckDepInfo { dep_info: PathBuf, checksum: bool },
    830
    831    /// This represents a nonempty set of `rerun-if-changed` annotations printed
    832    /// out by a build script. The `output` file is a relative file anchored at
    833    /// `target_root(...)` which is the actual output of the build script. That
    834    /// output has already been parsed and the paths printed out via
    835    /// `rerun-if-changed` are listed in `paths`. The `paths` field is relative
    836    /// to `pkg.root()`
    837    ///
    838    /// This is considered up-to-date if all of the `paths` are older than
    839    /// `output`, otherwise we need to recompile.
    840    RerunIfChanged {
    841        output: PathBuf,
    842        paths: Vec<PathBuf>,
    843    },
    844
    845    /// This represents a single `rerun-if-env-changed` annotation printed by a
    846    /// build script. The exact env var and value are hashed here. There's no
    847    /// filesystem dependence here, and if the values are changed the hash will
    848    /// change forcing a recompile.
    849    RerunIfEnvChanged { var: String, val: Option<String> },
    850}
    851
    852/// See [`FsStatus::StaleItem`].
    853#[derive(Clone, Debug, Serialize, Deserialize)]
    854#[serde(tag = "stale_item", rename_all = "kebab-case")]
    855pub enum StaleItem {
    856    MissingFile {
    857        path: PathBuf,
    858    },
    859    UnableToReadFile {
    860        path: PathBuf,
    861    },
    862    FailedToReadMetadata {
    863        path: PathBuf,
    864    },
    865    FileSizeChanged {
    866        path: PathBuf,
    867        old_size: u64,
    868        new_size: u64,
    869    },
    870    ChangedFile {
    871        reference: PathBuf,
    872        #[serde(with = "serde_file_time")]
    873        reference_mtime: FileTime,
    874        stale: PathBuf,
    875        #[serde(with = "serde_file_time")]
    876        stale_mtime: FileTime,
    877    },
    878    ChangedChecksum {
    879        source: PathBuf,
    880        stored_checksum: Checksum,
    881        new_checksum: Checksum,
    882    },
    883    MissingChecksum {
    884        path: PathBuf,
    885    },
    886    ChangedEnv {
    887        var: String,
    888        previous: Option<String>,
    889        current: Option<String>,
    890    },
    891}
    892
    893impl LocalFingerprint {
    894    /// Read the environment variable of the given env `key`, and creates a new
    895    /// [`LocalFingerprint::RerunIfEnvChanged`] for it. The `env_config` is used firstly
    896    /// to check if the env var is set in the config system as some envs need to be overridden.
    897    /// If not, it will fallback to `std::env::var`.
    898    ///
    899    // TODO: `std::env::var` is allowed at this moment. Should figure out
    900    // if it makes sense if permitting to read env from the env snapshot.
    901    #[allow(clippy::disallowed_methods)]
    902    fn from_env<K: AsRef<str>>(
    903        key: K,
    904        env_config: &Arc<HashMap<String, OsString>>,
    905    ) -> LocalFingerprint {
    906        let key = key.as_ref();
    907        let var = key.to_owned();
    908        let val = if let Some(val) = env_config.get(key) {
    909            val.to_str().map(ToOwned::to_owned)
    910        } else {
    911            env::var(key).ok()
    912        };
    913        LocalFingerprint::RerunIfEnvChanged { var, val }
    914    }
    915
    916    /// Checks dynamically at runtime if this `LocalFingerprint` has a stale
    917    /// item inside of it.
    918    ///
    919    /// The main purpose of this function is to handle two different ways
    920    /// fingerprints can be invalidated:
    921    ///
    922    /// * One is a dependency listed in rustc's dep-info files is invalid. Note
    923    ///   that these could either be env vars or files. We check both here.
    924    ///
    925    /// * Another is the `rerun-if-changed` directive from build scripts. This
    926    ///   is where we'll find whether files have actually changed
    927    fn find_stale_item(
    928        &self,
    929        mtime_cache: &mut HashMap<PathBuf, FileTime>,
    930        checksum_cache: &mut HashMap<PathBuf, Checksum>,
    931        pkg: &Package,
    932        build_root: &Path,
    933        cargo_exe: &Path,
    934        gctx: &GlobalContext,
    935    ) -> CargoResult<Option<StaleItem>> {
    936        let pkg_root = pkg.root();
    937        match self {
    938            // We need to parse `dep_info`, learn about the crate's dependencies.
    939            //
    940            // For each env var we see if our current process's env var still
    941            // matches, and for each file we see if any of them are newer than
    942            // the `dep_info` file itself whose mtime represents the start of
    943            // rustc.
    944            LocalFingerprint::CheckDepInfo { dep_info, checksum } => {
    945                let dep_info = build_root.join(dep_info);
    946                let Some(info) = parse_dep_info(pkg_root, build_root, &dep_info)? else {
    947                    return Ok(Some(StaleItem::MissingFile { path: dep_info }));
    948                };
    949                for (key, previous) in info.env.iter() {
    950                    if let Some(value) = pkg.manifest().metadata().env_var(key.as_str()) {
    951                        if Some(value.as_ref()) == previous.as_deref() {
    952                            continue;
    953                        }
    954                    }
    955
    956                    let current = if key == CARGO_ENV {
    957                        Some(cargo_exe.to_str().ok_or_else(|| {
    958                            format_err!(
    959                                "cargo exe path {} must be valid UTF-8",
    960                                cargo_exe.display()
    961                            )
    962                        })?)
    963                    } else {
    964                        if let Some(value) = gctx.env_config()?.get(key) {
    965                            value.to_str()
    966                        } else {
    967                            gctx.get_env(key).ok()
    968                        }
    969                    };
    970                    if current == previous.as_deref() {
    971                        continue;
    972                    }
    973                    return Ok(Some(StaleItem::ChangedEnv {
    974                        var: key.clone(),
    975                        previous: previous.clone(),
    976                        current: current.map(Into::into),
    977                    }));
    978                }
    979                if *checksum {
    980                    Ok(find_stale_file(
    981                        mtime_cache,
    982                        checksum_cache,
    983                        &dep_info,
    984                        info.files.iter().map(|(file, checksum)| (file, *checksum)),
    985                        *checksum,
    986                    ))
    987                } else {
    988                    Ok(find_stale_file(
    989                        mtime_cache,
    990                        checksum_cache,
    991                        &dep_info,
    992                        info.files.into_keys().map(|p| (p, None)),
    993                        *checksum,
    994                    ))
    995                }
    996            }
    997
    998            // We need to verify that no paths listed in `paths` are newer than
    999            // the `output` path itself, or the last time the build script ran.
    1000            LocalFingerprint::RerunIfChanged { output, paths } => Ok(find_stale_file(
    1001                mtime_cache,
    1002                checksum_cache,
    1003                &build_root.join(output),
    1004                paths.iter().map(|p| (pkg_root.join(p), None)),
    1005                false,
    1006            )),
    1007
    1008            // These have no dependencies on the filesystem, and their values
    1009            // are included natively in the `Fingerprint` hash so nothing
    1010            // tocheck for here.
    1011            LocalFingerprint::RerunIfEnvChanged { .. } => Ok(None),
    1012            LocalFingerprint::Precalculated(..) => Ok(None),
    1013        }
    1014    }
    1015
    1016    fn kind(&self) -> &'static str {
    1017        match self {
    1018            LocalFingerprint::Precalculated(..) => "precalculated",
    1019            LocalFingerprint::CheckDepInfo { .. } => "dep-info",
    1020            LocalFingerprint::RerunIfChanged { .. } => "rerun-if-changed",
    1021            LocalFingerprint::RerunIfEnvChanged { .. } => "rerun-if-env-changed",
    1022        }
    1023    }
    1024}
    1025
    1026impl Fingerprint {
    1027    fn new() -> Fingerprint {
    1028        Fingerprint {
    1029            rustc: 0,
    1030            target: 0,
    1031            profile: 0,
    1032            path: 0,
    1033            features: String::new(),
    1034            declared_features: String::new(),
    1035            deps: Vec::new(),
    1036            local: Mutex::new(Vec::new()),
    1037            memoized_hash: Mutex::new(None),
    1038            rustflags: Vec::new(),
    1039            config: 0,
    1040            compile_kind: 0,
    1041            index: UnitIndex::default(),
    1042            fs_status: FsStatus::Stale,
    1043            outputs: Vec::new(),
    1044        }
    1045    }
    1046
    1047    /// For performance reasons fingerprints will memoize their own hash, but
    1048    /// there's also internal mutability with its `local` field which can
    1049    /// change, for example with build scripts, during a build.
    1050    ///
    1051    /// This method can be used to bust all memoized hashes just before a build
    1052    /// to ensure that after a build completes everything is up-to-date.
    1053    pub fn clear_memoized(&self) {
    1054        *self.memoized_hash.lock().unwrap() = None;
    1055    }
    1056
    1057    fn hash_u64(&self) -> u64 {
    1058        if let Some(s) = *self.memoized_hash.lock().unwrap() {
    1059            return s;
    1060        }
    1061        let ret = util::hash_u64(self);
    1062        *self.memoized_hash.lock().unwrap() = Some(ret);
    1063        ret
    1064    }
    1065
    1066    /// Compares this fingerprint with an old version which was previously
    1067    /// serialized to filesystem.
    1068    ///
    1069    /// The purpose of this is exclusively to produce a diagnostic message
    1070    /// [`DirtyReason`], indicating why we're recompiling something.
    1071    fn compare(&self, old: &Fingerprint) -> DirtyReason {
    1072        if self.rustc != old.rustc {
    1073            return DirtyReason::RustcChanged;
    1074        }
    1075        if self.features != old.features {
    1076            return DirtyReason::FeaturesChanged {
    1077                old: old.features.clone(),
    1078                new: self.features.clone(),
    1079            };
    1080        }
    1081        if self.declared_features != old.declared_features {
    1082            return DirtyReason::DeclaredFeaturesChanged {
    1083                old: old.declared_features.clone(),
    1084                new: self.declared_features.clone(),
    1085            };
    1086        }
    1087        if self.target != old.target {
    1088            return DirtyReason::TargetConfigurationChanged;
    1089        }
    1090        if self.path != old.path {
    1091            return DirtyReason::PathToSourceChanged;
    1092        }
    1093        if self.profile != old.profile {
    1094            return DirtyReason::ProfileConfigurationChanged;
    1095        }
    1096        if self.rustflags != old.rustflags {
    1097            return DirtyReason::RustflagsChanged {
    1098                old: old.rustflags.clone(),
    1099                new: self.rustflags.clone(),
    1100            };
    1101        }
    1102        if self.config != old.config {
    1103            return DirtyReason::ConfigSettingsChanged;
    1104        }
    1105        if self.compile_kind != old.compile_kind {
    1106            return DirtyReason::CompileKindChanged;
    1107        }
    1108        let my_local = self.local.lock().unwrap();
    1109        let old_local = old.local.lock().unwrap();
    1110        if my_local.len() != old_local.len() {
    1111            return DirtyReason::LocalLengthsChanged;
    1112        }
    1113        for (new, old) in my_local.iter().zip(old_local.iter()) {
    1114            match (new, old) {
    1115                (LocalFingerprint::Precalculated(a), LocalFingerprint::Precalculated(b)) => {
    1116                    if a != b {
    1117                        return DirtyReason::PrecalculatedComponentsChanged {
    1118                            old: b.to_string(),
    1119                            new: a.to_string(),
    1120                        };
    1121                    }
    1122                }
    1123                (
    1124                    LocalFingerprint::CheckDepInfo {
    1125                        dep_info: a_dep,
    1126                        checksum: checksum_a,
    1127                    },
    1128                    LocalFingerprint::CheckDepInfo {
    1129                        dep_info: b_dep,
    1130                        checksum: checksum_b,
    1131                    },
    1132                ) => {
    1133                    if a_dep != b_dep {
    1134                        return DirtyReason::DepInfoOutputChanged {
    1135                            old: b_dep.clone(),
    1136                            new: a_dep.clone(),
    1137                        };
    1138                    }
    1139                    if checksum_a != checksum_b {
    1140                        return DirtyReason::ChecksumUseChanged { old: *checksum_b };
    1141                    }
    1142                }
    1143                (
    1144                    LocalFingerprint::RerunIfChanged {
    1145                        output: a_out,
    1146                        paths: a_paths,
    1147                    },
    1148                    LocalFingerprint::RerunIfChanged {
    1149                        output: b_out,
    1150                        paths: b_paths,
    1151                    },
    1152                ) => {
    1153                    if a_out != b_out {
    1154                        return DirtyReason::RerunIfChangedOutputFileChanged {
    1155                            old: b_out.clone(),
    1156                            new: a_out.clone(),
    1157                        };
    1158                    }
    1159                    if a_paths != b_paths {
    1160                        return DirtyReason::RerunIfChangedOutputPathsChanged {
    1161                            old: b_paths.clone(),
    1162                            new: a_paths.clone(),
    1163                        };
    1164                    }
    1165                }
    1166                (
    1167                    LocalFingerprint::RerunIfEnvChanged {
    1168                        var: a_key,
    1169                        val: a_value,
    1170                    },
    1171                    LocalFingerprint::RerunIfEnvChanged {
    1172                        var: b_key,
    1173                        val: b_value,
    1174                    },
    1175                ) => {
    1176                    if *a_key != *b_key {
    1177                        return DirtyReason::EnvVarsChanged {
    1178                            old: b_key.clone(),
    1179                            new: a_key.clone(),
    1180                        };
    1181                    }
    1182                    if *a_value != *b_value {
    1183                        return DirtyReason::EnvVarChanged {
    1184                            name: a_key.clone(),
    1185                            old_value: b_value.clone(),
    1186                            new_value: a_value.clone(),
    1187                        };
    1188                    }
    1189                }
    1190                (a, b) => {
    1191                    return DirtyReason::LocalFingerprintTypeChanged {
    1192                        old: b.kind().to_owned(),
    1193                        new: a.kind().to_owned(),
    1194                    };
    1195                }
    1196            }
    1197        }
    1198
    1199        if self.deps.len() != old.deps.len() {
    1200            return DirtyReason::NumberOfDependenciesChanged {
    1201                old: old.deps.len(),
    1202                new: self.deps.len(),
    1203            };
    1204        }
    1205        for (a, b) in self.deps.iter().zip(old.deps.iter()) {
    1206            if a.name != b.name {
    1207                return DirtyReason::UnitDependencyNameChanged {
    1208                    old: b.name,
    1209                    new: a.name,
    1210                };
    1211            }
    1212
    1213            if a.fingerprint.hash_u64() != b.fingerprint.hash_u64() {
    1214                return DirtyReason::UnitDependencyInfoChanged {
    1215                    unit: a.fingerprint.index,
    1216                };
    1217            }
    1218        }
    1219
    1220        if !self.fs_status.up_to_date() {
    1221            return DirtyReason::FsStatusOutdated(self.fs_status.clone());
    1222        }
    1223
    1224        // This typically means some filesystem modifications happened or
    1225        // something transitive was odd. In general we should strive to provide
    1226        // a better error message than this, so if you see this message a lot it
    1227        // likely means this method needs to be updated!
    1228        DirtyReason::NothingObvious
    1229    }
    1230
    1231    /// Dynamically inspect the local filesystem to update the `fs_status` field
    1232    /// of this `Fingerprint`.
    1233    ///
    1234    /// This function is used just after a `Fingerprint` is constructed to check
    1235    /// the local state of the filesystem and propagate any dirtiness from
    1236    /// dependencies up to this unit as well. This function assumes that the
    1237    /// unit starts out as [`FsStatus::Stale`] and then it will optionally switch
    1238    /// it to `UpToDate` if it can.
    1239    fn check_filesystem(
    1240        &mut self,
    1241        mtime_cache: &mut HashMap<PathBuf, FileTime>,
    1242        checksum_cache: &mut HashMap<PathBuf, Checksum>,
    1243        pkg: &Package,
    1244        build_root: &Path,
    1245        cargo_exe: &Path,
    1246        gctx: &GlobalContext,
    1247    ) -> CargoResult<()> {
    1248        assert!(!self.fs_status.up_to_date());
    1249
    1250        let pkg_root = pkg.root();
    1251        let mut mtimes = HashMap::new();
    1252
    1253        // Get the `mtime` of all outputs. Optionally update their mtime
    1254        // afterwards based on the `mtime_on_use` flag. Afterwards we want the
    1255        // minimum mtime as it's the one we'll be comparing to inputs and
    1256        // dependencies.
    1257        for output in self.outputs.iter() {
    1258            let Ok(mtime) = paths::mtime(output) else {
    1259                // This path failed to report its `mtime`. It probably doesn't
    1260                // exists, so leave ourselves as stale and bail out.
    1261                let item = StaleItem::FailedToReadMetadata {
    1262                    path: output.clone(),
    1263                };
    1264                self.fs_status = FsStatus::StaleItem(item);
    1265                return Ok(());
    1266            };
    1267            assert!(mtimes.insert(output.clone(), mtime).is_none());
    1268        }
    1269
    1270        let opt_max = mtimes.iter().max_by_key(|kv| kv.1);
    1271        let Some((max_path, max_mtime)) = opt_max else {
    1272            // We had no output files. This means we're an overridden build
    1273            // script and we're just always up to date because we aren't
    1274            // watching the filesystem.
    1275            self.fs_status = FsStatus::UpToDate { mtimes };
    1276            return Ok(());
    1277        };
    1278        debug!(
    1279            "max output mtime for {:?} is {:?} {}",
    1280            pkg_root, max_path, max_mtime
    1281        );
    1282
    1283        for dep in self.deps.iter() {
    1284            let dep_mtimes = match &dep.fingerprint.fs_status {
    1285                FsStatus::UpToDate { mtimes } => mtimes,
    1286                // If our dependency is stale, so are we, so bail out.
    1287                FsStatus::Stale
    1288                | FsStatus::StaleItem(_)
    1289                | FsStatus::StaleDependency { .. }
    1290                | FsStatus::StaleDepFingerprint { .. } => {
    1291                    self.fs_status = FsStatus::StaleDepFingerprint {
    1292                        unit: dep.fingerprint.index,
    1293                    };
    1294                    return Ok(());
    1295                }
    1296            };
    1297
    1298            // If our dependency edge only requires the rmeta file to be present
    1299            // then we only need to look at that one output file, otherwise we
    1300            // need to consider all output files to see if we're out of date.
    1301            let (dep_path, dep_mtime) = if dep.only_requires_rmeta {
    1302                dep_mtimes
    1303                    .iter()
    1304                    .find(|(path, _mtime)| {
    1305                        path.extension().and_then(|s| s.to_str()) == Some("rmeta")
    1306                    })
    1307                    .expect("failed to find rmeta")
    1308            } else {
    1309                match dep_mtimes.iter().max_by_key(|kv| kv.1) {
    1310                    Some(dep_mtime) => dep_mtime,
    1311                    // If our dependencies is up to date and has no filesystem
    1312                    // interactions, then we can move on to the next dependency.
    1313                    None => continue,
    1314                }
    1315            };
    1316            debug!(
    1317                "max dep mtime for {:?} is {:?} {}",
    1318                pkg_root, dep_path, dep_mtime
    1319            );
    1320
    1321            // If the dependency is newer than our own output then it was
    1322            // recompiled previously. We transitively become stale ourselves in
    1323            // that case, so bail out.
    1324            //
    1325            // Note that this comparison should probably be `>=`, not `>`, but
    1326            // for a discussion of why it's `>` see the discussion about #5918
    1327            // below in `find_stale`.
    1328            if dep_mtime > max_mtime {
    1329                info!(
    1330                    "dependency on `{}` is newer than we are {} > {} {:?}",
    1331                    dep.name, dep_mtime, max_mtime, pkg_root
    1332                );
    1333
    1334                self.fs_status = FsStatus::StaleDependency {
    1335                    unit: dep.fingerprint.index,
    1336                    dep_mtime: *dep_mtime,
    1337                    max_mtime: *max_mtime,
    1338                };
    1339
    1340                return Ok(());
    1341            }
    1342        }
    1343
    1344        // If we reached this far then all dependencies are up to date. Check
    1345        // all our `LocalFingerprint` information to see if we have any stale
    1346        // files for this package itself. If we do find something log a helpful
    1347        // message and bail out so we stay stale.
    1348        for local in self.local.get_mut().unwrap().iter() {
    1349            if let Some(item) = local.find_stale_item(
    1350                mtime_cache,
    1351                checksum_cache,
    1352                pkg,
    1353                build_root,
    1354                cargo_exe,
    1355                gctx,
    1356            )? {
    1357                item.log();
    1358                self.fs_status = FsStatus::StaleItem(item);
    1359                return Ok(());
    1360            }
    1361        }
    1362
    1363        // Everything was up to date! Record such.
    1364        self.fs_status = FsStatus::UpToDate { mtimes };
    1365        debug!("filesystem up-to-date {:?}", pkg_root);
    1366
    1367        Ok(())
    1368    }
    1369}
    1370
    1371impl hash::Hash for Fingerprint {
    1372    fn hash<H: Hasher>(&self, h: &mut H) {
    1373        let Fingerprint {
    1374            rustc,
    1375            ref features,
    1376            ref declared_features,
    1377            target,
    1378            path,
    1379            profile,
    1380            ref deps,
    1381            ref local,
    1382            config,
    1383            compile_kind,
    1384            ref rustflags,
    1385            ..
    1386        } = *self;
    1387        let local = local.lock().unwrap();
    1388        (
    1389            rustc,
    1390            features,
    1391            declared_features,
    1392            target,
    1393            path,
    1394            profile,
    1395            &*local,
    1396            config,
    1397            compile_kind,
    1398            rustflags,
    1399        )
    1400            .hash(h);
    1401
    1402        h.write_usize(deps.len());
    1403        for DepFingerprint {
    1404            pkg_id,
    1405            name,
    1406            public,
    1407            fingerprint,
    1408            only_requires_rmeta: _, // static property, no need to hash
    1409        } in deps
    1410        {
    1411            pkg_id.hash(h);
    1412            name.hash(h);
    1413            public.hash(h);
    1414            // use memoized dep hashes to avoid exponential blowup
    1415            h.write_u64(fingerprint.hash_u64());
    1416        }
    1417    }
    1418}
    1419
    1420impl DepFingerprint {
    1421    fn new(
    1422        build_runner: &mut BuildRunner<'_, '_>,
    1423        parent: &Unit,
    1424        dep: &UnitDep,
    1425    ) -> CargoResult<DepFingerprint> {
    1426        let fingerprint = calculate(build_runner, &dep.unit)?;
    1427        // We need to be careful about what we hash here. We have a goal of
    1428        // supporting renaming a project directory and not rebuilding
    1429        // everything. To do that, however, we need to make sure that the cwd
    1430        // doesn't make its way into any hashes, and one source of that is the
    1431        // `SourceId` for `path` packages.
    1432        //
    1433        // We already have a requirement that `path` packages all have unique
    1434        // names (sort of for this same reason), so if the package source is a
    1435        // `path` then we just hash the name, but otherwise we hash the full
    1436        // id as it won't change when the directory is renamed.
    1437        let pkg_id = if dep.unit.pkg.package_id().source_id().is_path() {
    1438            util::hash_u64(dep.unit.pkg.package_id().name())
    1439        } else {
    1440            util::hash_u64(dep.unit.pkg.package_id())
    1441        };
    1442
    1443        Ok(DepFingerprint {
    1444            pkg_id,
    1445            name: dep.extern_crate_name,
    1446            public: dep.public,
    1447            fingerprint,
    1448            only_requires_rmeta: build_runner.only_requires_rmeta(parent, &dep.unit),
    1449        })
    1450    }
    1451}
    1452
    1453impl StaleItem {
    1454    /// Use the `log` crate to log a hopefully helpful message in diagnosing
    1455    /// what file is considered stale and why. This is intended to be used in
    1456    /// conjunction with `CARGO_LOG` to determine why Cargo is recompiling
    1457    /// something. Currently there's no user-facing usage of this other than
    1458    /// that.
    1459    fn log(&self) {
    1460        match self {
    1461            StaleItem::MissingFile { path } => {
    1462                info!("stale: missing {:?}", path);
    1463            }
    1464            StaleItem::UnableToReadFile { path } => {
    1465                info!("stale: unable to read {:?}", path);
    1466            }
    1467            StaleItem::FailedToReadMetadata { path } => {
    1468                info!("stale: couldn't read metadata {:?}", path);
    1469            }
    1470            StaleItem::ChangedFile {
    1471                reference,
    1472                reference_mtime,
    1473                stale,
    1474                stale_mtime,
    1475            } => {
    1476                info!("stale: changed {:?}", stale);
    1477                info!("          (vs) {:?}", reference);
    1478                info!("               {:?} < {:?}", reference_mtime, stale_mtime);
    1479            }
    1480            StaleItem::FileSizeChanged {
    1481                path,
    1482                new_size,
    1483                old_size,
    1484            } => {
    1485                info!("stale: changed {:?}", path);
    1486                info!("prior file size {old_size}");
    1487                info!("  new file size {new_size}");
    1488            }
    1489            StaleItem::ChangedChecksum {
    1490                source,
    1491                stored_checksum,
    1492                new_checksum,
    1493            } => {
    1494                info!("stale: changed {:?}", source);
    1495                info!("prior checksum {stored_checksum}");
    1496                info!("  new checksum {new_checksum}");
    1497            }
    1498            StaleItem::MissingChecksum { path } => {
    1499                info!("stale: no prior checksum {:?}", path);
    1500            }
    1501            StaleItem::ChangedEnv {
    1502                var,
    1503                previous,
    1504                current,
    1505            } => {
    1506                info!("stale: changed env {:?}", var);
    1507                info!("       {:?} != {:?}", previous, current);
    1508            }
    1509        }
    1510    }
    1511}
    1512
    1513/// Calculates the fingerprint for a [`Unit`].
    1514///
    1515/// This fingerprint is used by Cargo to learn about when information such as:
    1516///
    1517/// * A non-path package changes (changes version, changes revision, etc).
    1518/// * Any dependency changes
    1519/// * The compiler changes
    1520/// * The set of features a package is built with changes
    1521/// * The profile a target is compiled with changes (e.g., opt-level changes)
    1522/// * Any other compiler flags change that will affect the result
    1523///
    1524/// Information like file modification time is only calculated for path
    1525/// dependencies.
    1526fn calculate(build_runner: &mut BuildRunner<'_, '_>, unit: &Unit) -> CargoResult<Arc<Fingerprint>> {
    1527    // This function is slammed quite a lot, so the result is memoized.
    1528    if let Some(s) = build_runner.fingerprints.get(unit) {
    1529        return Ok(Arc::clone(s));
    1530    }
    1531    let mut fingerprint = if unit.mode.is_run_custom_build() {
    1532        calculate_run_custom_build(build_runner, unit)?
    1533    } else if unit.mode.is_doc_test() {
    1534        panic!("doc tests do not fingerprint");
    1535    } else {
    1536        calculate_normal(build_runner, unit)?
    1537    };
    1538
    1539    // After we built the initial `Fingerprint` be sure to update the
    1540    // `fs_status` field of it.
    1541    let build_root = build_root(build_runner);
    1542    let cargo_exe = build_runner.bcx.gctx.cargo_exe()?;
    1543    fingerprint.check_filesystem(
    1544        &mut build_runner.mtime_cache,
    1545        &mut build_runner.checksum_cache,
    1546        &unit.pkg,
    1547        &build_root,
    1548        cargo_exe,
    1549        build_runner.bcx.gctx,
    1550    )?;
    1551
    1552    let fingerprint = Arc::new(fingerprint);
    1553    build_runner
    1554        .fingerprints
    1555        .insert(unit.clone(), Arc::clone(&fingerprint));
    1556    Ok(fingerprint)
    1557}
    1558
    1559/// Calculate a fingerprint for a "normal" unit, or anything that's not a build
    1560/// script. This is an internal helper of [`calculate`], don't call directly.
    1561fn calculate_normal(
    1562    build_runner: &mut BuildRunner<'_, '_>,
    1563    unit: &Unit,
    1564) -> CargoResult<Fingerprint> {
    1565    let deps = {
    1566        // Recursively calculate the fingerprint for all of our dependencies.
    1567        //
    1568        // Skip fingerprints of binaries because they don't actually induce a
    1569        // recompile, they're just dependencies in the sense that they need to be
    1570        // built. The only exception here are artifact dependencies,
    1571        // which is an actual dependency that needs a recompile.
    1572        //
    1573        // Create Vec since mutable build_runner is needed in closure.
    1574        let deps = Vec::from(build_runner.unit_deps(unit));
    1575        let mut deps = deps
    1576            .into_iter()
    1577            .filter(|dep| !dep.unit.target.is_bin() || dep.unit.artifact.is_true())
    1578            .map(|dep| DepFingerprint::new(build_runner, unit, &dep))
    1579            .collect::<CargoResult<Vec<_>>>()?;
    1580        deps.sort_by(|a, b| a.pkg_id.cmp(&b.pkg_id));
    1581        deps
    1582    };
    1583
    1584    // Afterwards calculate our own fingerprint information.
    1585    let build_root = build_root(build_runner);
    1586    let is_any_doc_gen = unit.mode.is_doc() || unit.mode.is_doc_scrape();
    1587    let rustdoc_depinfo_enabled = build_runner.bcx.gctx.cli_unstable().rustdoc_depinfo;
    1588    let local = if is_any_doc_gen && !rustdoc_depinfo_enabled {
    1589        // rustdoc does not have dep-info files.
    1590        let fingerprint = pkg_fingerprint(build_runner.bcx, &unit.pkg).with_context(|| {
    1591            format!(
    1592                "failed to determine package fingerprint for documenting {}",
    1593                unit.pkg
    1594            )
    1595        })?;
    1596        vec![LocalFingerprint::Precalculated(fingerprint)]
    1597    } else {
    1598        let dep_info = dep_info_loc(build_runner, unit);
    1599        let dep_info = dep_info.strip_prefix(&build_root).unwrap().to_path_buf();
    1600        vec![LocalFingerprint::CheckDepInfo {
    1601            dep_info,
    1602            checksum: build_runner.bcx.gctx.cli_unstable().checksum_freshness,
    1603        }]
    1604    };
    1605
    1606    // Figure out what the outputs of our unit is, and we'll be storing them
    1607    // into the fingerprint as well.
    1608    let outputs = build_runner
    1609        .outputs(unit)?
    1610        .iter()
    1611        .filter(|output| {
    1612            !matches!(
    1613                output.flavor,
    1614                FileFlavor::DebugInfo | FileFlavor::Auxiliary | FileFlavor::Sbom
    1615            )
    1616        })
    1617        .map(|output| output.path.clone())
    1618        .collect();
    1619
    1620    // Fill out a bunch more information that we'll be tracking typically
    1621    // hashed to take up less space on disk as we just need to know when things
    1622    // change.
    1623    let extra_flags = if unit.mode.is_doc() || unit.mode.is_doc_scrape() {
    1624        &unit.rustdocflags
    1625    } else {
    1626        &unit.rustflags
    1627    }
    1628    .to_vec();
    1629
    1630    let profile_hash = util::hash_u64((
    1631        &unit.profile,
    1632        unit.mode,
    1633        build_runner.bcx.extra_args_for(unit),
    1634        build_runner.lto[unit],
    1635        unit.pkg.manifest().lint_rustflags(),
    1636    ));
    1637    let mut config = StableHasher::new();
    1638    if let Some(linker) = build_runner.compilation.target_linker(unit.kind) {
    1639        linker.hash(&mut config);
    1640    }
    1641    if unit.mode.is_doc() && build_runner.bcx.gctx.cli_unstable().rustdoc_map {
    1642        if let Ok(map) = build_runner.bcx.gctx.doc_extern_map() {
    1643            map.hash(&mut config);
    1644        }
    1645    }
    1646    if let Some(allow_features) = &build_runner.bcx.gctx.cli_unstable().allow_features {
    1647        allow_features.hash(&mut config);
    1648    }
    1649    // -Zno-embed-metadata changes how all units are compiled, and it also changes how we tell
    1650    // rustc to link to deps using `--extern`. If it changes, we should rebuild everything.
    1651    build_runner
    1652        .bcx
    1653        .gctx
    1654        .cli_unstable()
    1655        .no_embed_metadata
    1656        .hash(&mut config);
    1657
    1658    let compile_kind = unit.kind.fingerprint_hash();
    1659    let mut declared_features = unit.pkg.summary().features().keys().collect::<Vec<_>>();
    1660    declared_features.sort(); // to avoid useless rebuild if the user orders it's features
    1661    // differently
    1662    Ok(Fingerprint {
    1663        rustc: util::hash_u64(&build_runner.bcx.rustc().verbose_version),
    1664        target: util::hash_u64(&unit.target),
    1665        profile: profile_hash,
    1666        // Note that .0 is hashed here, not .1 which is the cwd. That doesn't
    1667        // actually affect the output artifact so there's no need to hash it.
    1668        path: util::hash_u64(path_args(build_runner.bcx.ws, unit).0),
    1669        features: format!("{:?}", unit.features),
    1670        declared_features: format!("{declared_features:?}"),
    1671        deps,
    1672        local: Mutex::new(local),
    1673        memoized_hash: Mutex::new(None),
    1674        config: Hasher::finish(&config),
    1675        compile_kind,
    1676        index: build_runner.bcx.unit_to_index[unit],
    1677        rustflags: extra_flags,
    1678        fs_status: FsStatus::Stale,
    1679        outputs,
    1680    })
    1681}
    1682
    1683/// Calculate a fingerprint for an "execute a build script" unit.  This is an
    1684/// internal helper of [`calculate`], don't call directly.
    1685fn calculate_run_custom_build(
    1686    build_runner: &mut BuildRunner<'_, '_>,
    1687    unit: &Unit,
    1688) -> CargoResult<Fingerprint> {
    1689    assert!(unit.mode.is_run_custom_build());
    1690    // Using the `BuildDeps` information we'll have previously parsed and
    1691    // inserted into `build_explicit_deps` built an initial snapshot of the
    1692    // `LocalFingerprint` list for this build script. If we previously executed
    1693    // the build script this means we'll be watching files and env vars.
    1694    // Otherwise if we haven't previously executed it we'll just start watching
    1695    // the whole crate.
    1696    let (gen_local, overridden) = build_script_local_fingerprints(build_runner, unit)?;
    1697    let deps = &build_runner.build_explicit_deps[unit];
    1698    let local = (gen_local)(
    1699        deps,
    1700        Some(&|| {
    1701            const IO_ERR_MESSAGE: &str = "\
    1702An I/O error happened. Please make sure you can access the file.
    1703
    1704By default, if your project contains a build script, cargo scans all files in
    1705it to determine whether a rebuild is needed. If you don't expect to access the
    1706file, specify `rerun-if-changed` in your build script.
    1707See https://doc.rust-lang.org/cargo/reference/build-scripts.html#rerun-if-changed for more information.";
    1708            pkg_fingerprint(build_runner.bcx, &unit.pkg).map_err(|err| {
    1709                let mut message = format!("failed to determine package fingerprint for build script for {}", unit.pkg);
    1710                if err.root_cause().is::<io::Error>() {
    1711                    message = format!("{}\n{}", message, IO_ERR_MESSAGE)
    1712                }
    1713                err.context(message)
    1714            })
    1715        }),
    1716    )?
    1717    .unwrap();
    1718    let output = deps.build_script_output.clone();
    1719
    1720    // Include any dependencies of our execution, which is typically just the
    1721    // compilation of the build script itself. (if the build script changes we
    1722    // should be rerun!). Note though that if we're an overridden build script
    1723    // we have no dependencies so no need to recurse in that case.
    1724    let deps = if overridden {
    1725        // Overridden build scripts don't need to track deps.
    1726        vec![]
    1727    } else {
    1728        // Create Vec since mutable build_runner is needed in closure.
    1729        let deps = Vec::from(build_runner.unit_deps(unit));
    1730        deps.into_iter()
    1731            .map(|dep| DepFingerprint::new(build_runner, unit, &dep))
    1732            .collect::<CargoResult<Vec<_>>>()?
    1733    };
    1734
    1735    let rustflags = unit.rustflags.to_vec();
    1736
    1737    Ok(Fingerprint {
    1738        local: Mutex::new(local),
    1739        rustc: util::hash_u64(&build_runner.bcx.rustc().verbose_version),
    1740        deps,
    1741        outputs: if overridden { Vec::new() } else { vec![output] },
    1742        rustflags,
    1743        index: build_runner.bcx.unit_to_index[unit],
    1744
    1745        // Most of the other info is blank here as we don't really include it
    1746        // in the execution of the build script, but... this may be a latent
    1747        // bug in Cargo.
    1748        ..Fingerprint::new()
    1749    })
    1750}
    1751
    1752/// Get ready to compute the [`LocalFingerprint`] values
    1753/// for a [`RunCustomBuild`] unit.
    1754///
    1755/// This function has, what's on the surface, a seriously wonky interface.
    1756/// You'll call this function and it'll return a closure and a boolean. The
    1757/// boolean is pretty simple in that it indicates whether the `unit` has been
    1758/// overridden via `.cargo/config.toml`. The closure is much more complicated.
    1759///
    1760/// This closure is intended to capture any local state necessary to compute
    1761/// the `LocalFingerprint` values for this unit. It is `Send` and `'static` to
    1762/// be sent to other threads as well (such as when we're executing build
    1763/// scripts). That deduplication is the rationale for the closure at least.
    1764///
    1765/// The arguments to the closure are a bit weirder, though, and I'll apologize
    1766/// in advance for the weirdness too. The first argument to the closure is a
    1767/// `&BuildDeps`. This is the parsed version of a build script, and when Cargo
    1768/// starts up this is cached from previous runs of a build script.  After a
    1769/// build script executes the output file is reparsed and passed in here.
    1770///
    1771/// The second argument is the weirdest, it's *optionally* a closure to
    1772/// call [`pkg_fingerprint`]. The `pkg_fingerprint` requires access to
    1773/// "source map" located in `Context`. That's very non-`'static` and
    1774/// non-`Send`, so it can't be used on other threads, such as when we invoke
    1775/// this after a build script has finished. The `Option` allows us to for sure
    1776/// calculate it on the main thread at the beginning, and then swallow the bug
    1777/// for now where a worker thread after a build script has finished doesn't
    1778/// have access. Ideally there would be no second argument or it would be more
    1779/// "first class" and not an `Option` but something that can be sent between
    1780/// threads. In any case, it's a bug for now.
    1781///
    1782/// This isn't the greatest of interfaces, and if there's suggestions to
    1783/// improve please do so!
    1784///
    1785/// FIXME(#6779) - see all the words above
    1786///
    1787/// [`RunCustomBuild`]: crate::core::compiler::CompileMode::RunCustomBuild
    1788fn build_script_local_fingerprints(
    1789    build_runner: &mut BuildRunner<'_, '_>,
    1790    unit: &Unit,
    1791) -> CargoResult<(
    1792    Box<
    1793        dyn FnOnce(
    1794                &BuildDeps,
    1795                Option<&dyn Fn() -> CargoResult<String>>,
    1796            ) -> CargoResult<Option<Vec<LocalFingerprint>>>
    1797            + Send,
    1798    >,
    1799    bool,
    1800)> {
    1801    assert!(unit.mode.is_run_custom_build());
    1802    // First up, if this build script is entirely overridden, then we just
    1803    // return the hash of what we overrode it with. This is the easy case!
    1804    if let Some(fingerprint) = build_script_override_fingerprint(build_runner, unit) {
    1805        debug!("override local fingerprints deps {}", unit.pkg);
    1806        return Ok((
    1807            Box::new(
    1808                move |_: &BuildDeps, _: Option<&dyn Fn() -> CargoResult<String>>| {
    1809                    Ok(Some(vec![fingerprint]))
    1810                },
    1811            ),
    1812            true, // this is an overridden build script
    1813        ));
    1814    }
    1815
    1816    // ... Otherwise this is a "real" build script and we need to return a real
    1817    // closure. Our returned closure classifies the build script based on
    1818    // whether it prints `rerun-if-*`. If it *doesn't* print this it's where the
    1819    // magical second argument comes into play, which fingerprints a whole
    1820    // package. Remember that the fact that this is an `Option` is a bug, but a
    1821    // longstanding bug, in Cargo. Recent refactorings just made it painfully
    1822    // obvious.
    1823    let pkg_root = unit.pkg.root().to_path_buf();
    1824    let build_dir = build_root(build_runner);
    1825    let env_config = Arc::clone(build_runner.bcx.gctx.env_config()?);
    1826    let calculate =
    1827        move |deps: &BuildDeps, pkg_fingerprint: Option<&dyn Fn() -> CargoResult<String>>| {
    1828            if deps.rerun_if_changed.is_empty() && deps.rerun_if_env_changed.is_empty() {
    1829                match pkg_fingerprint {
    1830                    // FIXME: this is somewhat buggy with respect to docker and
    1831                    // weird filesystems. The `Precalculated` variant
    1832                    // constructed below will, for `path` dependencies, contain
    1833                    // a stringified version of the mtime for the local crate.
    1834                    // This violates one of the things we describe in this
    1835                    // module's doc comment, never hashing mtimes. We should
    1836                    // figure out a better scheme where a package fingerprint
    1837                    // may be a string (like for a registry) or a list of files
    1838                    // (like for a path dependency). Those list of files would
    1839                    // be stored here rather than the mtime of them.
    1840                    Some(f) => {
    1841                        let s = f()?;
    1842                        debug!(
    1843                            "old local fingerprints deps {:?} precalculated={:?}",
    1844                            pkg_root, s
    1845                        );
    1846                        return Ok(Some(vec![LocalFingerprint::Precalculated(s)]));
    1847                    }
    1848                    None => return Ok(None),
    1849                }
    1850            }
    1851
    1852            // Ok so now we're in "new mode" where we can have files listed as
    1853            // dependencies as well as env vars listed as dependencies. Process
    1854            // them all here.
    1855            Ok(Some(local_fingerprints_deps(
    1856                deps,
    1857                &build_dir,
    1858                &pkg_root,
    1859                &env_config,
    1860            )))
    1861        };
    1862
    1863    // Note that `false` == "not overridden"
    1864    Ok((Box::new(calculate), false))
    1865}
    1866
    1867/// Create a [`LocalFingerprint`] for an overridden build script.
    1868/// Returns None if it is not overridden.
    1869fn build_script_override_fingerprint(
    1870    build_runner: &mut BuildRunner<'_, '_>,
    1871    unit: &Unit,
    1872) -> Option<LocalFingerprint> {
    1873    // Build script output is only populated at this stage when it is
    1874    // overridden.
    1875    let build_script_outputs = build_runner.build_script_outputs.lock().unwrap();
    1876    let metadata = build_runner.get_run_build_script_metadata(unit);
    1877    // Returns None if it is not overridden.
    1878    let output = build_script_outputs.get(metadata)?;
    1879    let s = format!(
    1880        "overridden build state with hash: {}",
    1881        util::hash_u64(output)
    1882    );
    1883    Some(LocalFingerprint::Precalculated(s))
    1884}
    1885
    1886/// Compute the [`LocalFingerprint`] values for a [`RunCustomBuild`] unit for
    1887/// non-overridden new-style build scripts only. This is only used when `deps`
    1888/// is already known to have a nonempty `rerun-if-*` somewhere.
    1889///
    1890/// [`RunCustomBuild`]: crate::core::compiler::CompileMode::RunCustomBuild
    1891fn local_fingerprints_deps(
    1892    deps: &BuildDeps,
    1893    build_root: &Path,
    1894    pkg_root: &Path,
    1895    env_config: &Arc<HashMap<String, OsString>>,
    1896) -> Vec<LocalFingerprint> {
    1897    debug!("new local fingerprints deps {:?}", pkg_root);
    1898    let mut local = Vec::new();
    1899
    1900    if !deps.rerun_if_changed.is_empty() {
    1901        // Note that like the module comment above says we are careful to never
    1902        // store an absolute path in `LocalFingerprint`, so ensure that we strip
    1903        // absolute prefixes from them.
    1904        let output = deps
    1905            .build_script_output
    1906            .strip_prefix(build_root)
    1907            .unwrap()
    1908            .to_path_buf();
    1909        let paths = deps
    1910            .rerun_if_changed
    1911            .iter()
    1912            .map(|p| p.strip_prefix(pkg_root).unwrap_or(p).to_path_buf())
    1913            .collect();
    1914        local.push(LocalFingerprint::RerunIfChanged { output, paths });
    1915    }
    1916
    1917    local.extend(
    1918        deps.rerun_if_env_changed
    1919            .iter()
    1920            .map(|s| LocalFingerprint::from_env(s, env_config)),
    1921    );
    1922
    1923    local
    1924}
    1925
    1926/// Writes the short fingerprint hash value to `<loc>`
    1927/// and logs detailed JSON information to `<loc>.json`.
    1928fn write_fingerprint(loc: &Path, fingerprint: &Fingerprint) -> CargoResult<()> {
    1929    debug_assert_ne!(fingerprint.rustc, 0);
    1930    // fingerprint::new().rustc == 0, make sure it doesn't make it to the file system.
    1931    // This is mostly so outside tools can reliably find out what rust version this file is for,
    1932    // as we can use the full hash.
    1933    let hash = fingerprint.hash_u64();
    1934    debug!("write fingerprint ({:x}) : {}", hash, loc.display());
    1935    paths::write(loc, util::to_hex(hash).as_bytes())?;
    1936
    1937    let json = serde_json::to_string(fingerprint).unwrap();
    1938    if cfg!(debug_assertions) {
    1939        let f: Fingerprint = serde_json::from_str(&json).unwrap();
    1940        assert_eq!(f.hash_u64(), hash);
    1941    }
    1942    paths::write(&loc.with_extension("json"), json.as_bytes())?;
    1943    Ok(())
    1944}
    1945
    1946/// Prepare for work when a package starts to build
    1947pub fn prepare_init(build_runner: &mut BuildRunner<'_, '_>, unit: &Unit) -> CargoResult<()> {
    1948    let new1 = build_runner.files().fingerprint_dir(unit);
    1949
    1950    // Doc tests have no output, thus no fingerprint.
    1951    if !new1.exists() && !unit.mode.is_doc_test() {
    1952        paths::create_dir_all(&new1)?;
    1953    }
    1954
    1955    Ok(())
    1956}
    1957
    1958/// Returns the location that the dep-info file will show up at
    1959/// for the [`Unit`] specified.
    1960pub fn dep_info_loc(build_runner: &mut BuildRunner<'_, '_>, unit: &Unit) -> PathBuf {
    1961    build_runner.files().fingerprint_file_path(unit, "dep-")
    1962}
    1963
    1964/// Returns an absolute path that build directory.
    1965/// All paths are rewritten to be relative to this.
    1966fn build_root(build_runner: &BuildRunner<'_, '_>) -> PathBuf {
    1967    build_runner.bcx.ws.build_dir().into_path_unlocked()
    1968}
    1969
    1970/// Reads the value from the old fingerprint hash file and compare.
    1971///
    1972/// If dirty, it then restores the detailed information
    1973/// from the fingerprint JSON file, and provides an rich dirty reason.
    1974fn compare_old_fingerprint(
    1975    unit: &Unit,
    1976    old_hash_path: &Path,
    1977    new_fingerprint: &Fingerprint,
    1978    mtime_on_use: bool,
    1979    forced: bool,
    1980) -> FingerprintComparison {
    1981    if mtime_on_use {
    1982        // update the mtime so other cleaners know we used it
    1983        let t = FileTime::from_system_time(SystemTime::now());
    1984        debug!("mtime-on-use forcing {:?} to {}", old_hash_path, t);
    1985        paths::set_file_time_no_err(old_hash_path, t);
    1986    }
    1987
    1988    let compare = _compare_old_fingerprint(old_hash_path, new_fingerprint);
    1989
    1990    match compare.as_ref() {
    1991        Ok(FingerprintComparison::Fresh) => {}
    1992        Ok(FingerprintComparison::Dirty { reason }) => {
    1993            info!(
    1994                "fingerprint dirty for {}/{:?}/{:?}",
    1995                unit.pkg, unit.mode, unit.target,
    1996            );
    1997            info!("    dirty: {reason:?}");
    1998        }
    1999        Err(e) => {
    2000            info!(
    2001                "fingerprint error for {}/{:?}/{:?}",
    2002                unit.pkg, unit.mode, unit.target,
    2003            );
    2004            info!("    err: {e:?}");
    2005        }
    2006    }
    2007
    2008    match compare {
    2009        Ok(FingerprintComparison::Fresh) if forced => FingerprintComparison::Dirty {
    2010            reason: DirtyReason::Forced,
    2011        },
    2012        Ok(cmp) => cmp,
    2013        Err(_) => FingerprintComparison::Dirty {
    2014            reason: DirtyReason::FreshBuild,
    2015        },
    2016    }
    2017}
    2018
    2019fn _compare_old_fingerprint(
    2020    old_hash_path: &Path,
    2021    new_fingerprint: &Fingerprint,
    2022) -> CargoResult<FingerprintComparison> {
    2023    let old_fingerprint_short = paths::read(old_hash_path)?;
    2024
    2025    let new_hash = new_fingerprint.hash_u64();
    2026
    2027    if util::to_hex(new_hash) == old_fingerprint_short && new_fingerprint.fs_status.up_to_date() {
    2028        return Ok(FingerprintComparison::Fresh);
    2029    }
    2030
    2031    let old_fingerprint_json = paths::read(&old_hash_path.with_extension("json"))?;
    2032    let old_fingerprint: Fingerprint = serde_json::from_str(&old_fingerprint_json)
    2033        .with_context(|| internal("failed to deserialize json"))?;
    2034    // Fingerprint can be empty after a failed rebuild (see comment in prepare_target).
    2035    if !old_fingerprint_short.is_empty() {
    2036        debug_assert_eq!(
    2037            util::to_hex(old_fingerprint.hash_u64()),
    2038            old_fingerprint_short
    2039        );
    2040    }
    2041
    2042    let reason = new_fingerprint.compare(&old_fingerprint);
    2043    Ok(FingerprintComparison::Dirty { reason })
    2044}
    2045
    2046/// Calculates the fingerprint of a unit thats contains no dep-info files.
    2047fn pkg_fingerprint(bcx: &BuildContext<'_, '_>, pkg: &Package) -> CargoResult<String> {
    2048    let source_id = pkg.package_id().source_id();
    2049    let sources = bcx.packages.sources();
    2050
    2051    let source = sources
    2052        .get(source_id)
    2053        .ok_or_else(|| internal("missing package source"))?;
    2054    source.fingerprint(pkg)
    2055}
    2056
    2057/// The `reference` file is considered as "stale" if any file from `paths` has a newer mtime.
    2058fn find_stale_file<I, P>(
    2059    mtime_cache: &mut HashMap<PathBuf, FileTime>,
    2060    checksum_cache: &mut HashMap<PathBuf, Checksum>,
    2061    reference: &Path,
    2062    paths: I,
    2063    use_checksums: bool,
    2064) -> Option<StaleItem>
    2065where
    2066    I: IntoIterator<Item = (P, Option<(u64, Checksum)>)>,
    2067    P: AsRef<Path>,
    2068{
    2069    let reference_mtime = match paths::mtime(reference) {
    2070        Ok(mtime) => mtime,
    2071        Err(..) => {
    2072            return Some(StaleItem::MissingFile {
    2073                path: reference.to_path_buf(),
    2074            });
    2075        }
    2076    };
    2077
    2078    let skippable_dirs = if let Ok(cargo_home) = home::cargo_home() {
    2079        let skippable_dirs: Vec<_> = ["git", "registry"]
    2080            .into_iter()
    2081            .map(|subfolder| cargo_home.join(subfolder))
    2082            .collect();
    2083        Some(skippable_dirs)
    2084    } else {
    2085        None
    2086    };
    2087    for (path, prior_checksum) in paths {
    2088        let path = path.as_ref();
    2089
    2090        // Assuming anything in cargo_home/{git, registry} is immutable
    2091        // (see also #9455 about marking the src directory readonly) which avoids rebuilds when CI
    2092        // caches $CARGO_HOME/registry/{index, cache} and $CARGO_HOME/git/db across runs, keeping
    2093        // the content the same but changing the mtime.
    2094        if let Some(ref skippable_dirs) = skippable_dirs {
    2095            if skippable_dirs.iter().any(|dir| path.starts_with(dir)) {
    2096                continue;
    2097            }
    2098        }
    2099        if use_checksums {
    2100            let Some((file_len, prior_checksum)) = prior_checksum else {
    2101                return Some(StaleItem::MissingChecksum {
    2102                    path: path.to_path_buf(),
    2103                });
    2104            };
    2105            let path_buf = path.to_path_buf();
    2106
    2107            let path_checksum = match checksum_cache.entry(path_buf) {
    2108                Entry::Occupied(o) => *o.get(),
    2109                Entry::Vacant(v) => {
    2110                    let Ok(current_file_len) = fs::metadata(&path).map(|m| m.len()) else {
    2111                        return Some(StaleItem::FailedToReadMetadata {
    2112                            path: path.to_path_buf(),
    2113                        });
    2114                    };
    2115                    if current_file_len != file_len {
    2116                        return Some(StaleItem::FileSizeChanged {
    2117                            path: path.to_path_buf(),
    2118                            new_size: current_file_len,
    2119                            old_size: file_len,
    2120                        });
    2121                    }
    2122                    let Ok(file) = File::open(path) else {
    2123                        return Some(StaleItem::MissingFile {
    2124                            path: path.to_path_buf(),
    2125                        });
    2126                    };
    2127                    let Ok(checksum) = Checksum::compute(prior_checksum.algo(), file) else {
    2128                        return Some(StaleItem::UnableToReadFile {
    2129                            path: path.to_path_buf(),
    2130                        });
    2131                    };
    2132                    *v.insert(checksum)
    2133                }
    2134            };
    2135            if path_checksum == prior_checksum {
    2136                continue;
    2137            }
    2138            return Some(StaleItem::ChangedChecksum {
    2139                source: path.to_path_buf(),
    2140                stored_checksum: prior_checksum,
    2141                new_checksum: path_checksum,
    2142            });
    2143        } else {
    2144            let path_mtime = match mtime_cache.entry(path.to_path_buf()) {
    2145                Entry::Occupied(o) => *o.get(),
    2146                Entry::Vacant(v) => {
    2147                    let Ok(mtime) = paths::mtime_recursive(path) else {
    2148                        return Some(StaleItem::MissingFile {
    2149                            path: path.to_path_buf(),
    2150                        });
    2151                    };
    2152                    *v.insert(mtime)
    2153                }
    2154            };
    2155
    2156            // TODO: fix #5918.
    2157            // Note that equal mtimes should be considered "stale". For filesystems with
    2158            // not much timestamp precision like 1s this is would be a conservative approximation
    2159            // to handle the case where a file is modified within the same second after
    2160            // a build starts. We want to make sure that incremental rebuilds pick that up!
    2161            //
    2162            // For filesystems with nanosecond precision it's been seen in the wild that
    2163            // its "nanosecond precision" isn't really nanosecond-accurate. It turns out that
    2164            // kernels may cache the current time so files created at different times actually
    2165            // list the same nanosecond precision. Some digging on #5919 picked up that the
    2166            // kernel caches the current time between timer ticks, which could mean that if
    2167            // a file is updated at most 10ms after a build starts then Cargo may not
    2168            // pick up the build changes.
    2169            //
    2170            // All in all, an equality check here would be a conservative assumption that,
    2171            // if equal, files were changed just after a previous build finished.
    2172            // Unfortunately this became problematic when (in #6484) cargo switch to more accurately
    2173            // measuring the start time of builds.
    2174            if path_mtime <= reference_mtime {
    2175                continue;
    2176            }
    2177
    2178            return Some(StaleItem::ChangedFile {
    2179                reference: reference.to_path_buf(),
    2180                reference_mtime,
    2181                stale: path.to_path_buf(),
    2182                stale_mtime: path_mtime,
    2183            });
    2184        }
    2185    }
    2186
    2187    debug!(
    2188        "all paths up-to-date relative to {:?} mtime={}",
    2189        reference, reference_mtime
    2190    );
    2191    None
    2192}