libnix rust v4 release

motivation

this is a short status update on cargo (libnix) covering what changed since the libnix-1.87.0+v3 release. with libnix-1.87.0+v4 the nix backend is no longer limited to cargo build: tests, cargo run and cargo install work as well, so a normal day of rust development can happen in nix mode.

the code can be found here:

https://github.com/nixcloud/cargo/blob/master-libnix/README.md

improvements

cargo test

cargo test works in nix mode: unit, integration and doc tests, with the same output as vanilla cargo.

the test harnesses and examples are built in nix like every other unit, from target/<profile>/nix-test with GC roots of their own, so cargo build and cargo test don’t delete each other’s store paths. the tests themselves run outside of nix, like with vanilla cargo:

• OUT_DIR and rustc-env of the build scripts are read from the store
• CARGO_BIN_EXE_<name> points at the binary in the store, and integration tests find the binaries of their package next to themselves (snapbox, assert_cmd)
• doc tests use the rustdoc and linker the units were built with, so the host needs no toolchain
• env!("CARGO_MANIFEST_DIR") points at the sources in the store, which are read-only: fixtures can be read, not written

cargo run

cargo run runs binaries and examples with the arguments, exit code, working directory, OUT_DIR and rustc-env of vanilla cargo. target/debug/<bin> is a symlink into the store, so std::env::current_exe() is the store path.

cargo install and cargo uninstall

installed binaries stay in the nix store, since they link against store paths (glibc, the libraries from Cargo.dependencies.nix). each one is protected by an indirect GC root and ~/.cargo/bin links to it through that root:

~/.cargo/
├── .crates.toml / .crates2.json          unchanged, cargo install --list works
├── nix/install/
│   └── sd-1.0.0-<hash>/
│       └── sd  ->  /nix/store/…-sd-1_0_0-bin-…
└── bin/
    └── sd  ->  ../nix/install/sd-1.0.0-<hash>/sd/sd-<hash>

crates.io packages are fetched into the store like any other dependency. upgrading to another version replaces the root, and cargo uninstall removes the links and deletes the binary from the store, unless something else still uses it:

    Removing /home/joachim/.cargo/bin/sd
     Removed 3 store path(s), 2.4MiB

crate sources and crate builds of an install are left in the store for nix-collect-garbage, like ~/.cargo/registry they are shared with other builds.

    Finished 16 built, 0 fetched, 0 fresh, 0 failed, 0 skipped in 80.6s
     Removed 15 unused store path(s), 431.9MiB

build failures at the end

failures used to be printed as they happened and got interleaved with the output of other builds. now they are collected and printed after all builds are done, each under a header naming the crate and its derivation:

──── error 1/1: sd (bin) (sd-1_0_0-bin-a45bb84d6cc05f0a) ───────────────────────
       error failed to build `sd (bin)` in phase `buildPhase`: PermanentFailure

git dependencies

two fixes, both found with RustPython:

• git dependencies outside ~/.cargo/git/checkouts (vendored sources, source replacement) no longer crash the generator
• git crates are built in a writable copy of their repository instead of the read-only store, which also fixes crates in a subdirectory of their repository

regression tests

test.nix builds the projects of #18 at pinned revisions with the cargo (libnix) of the checkout: it generates each build system in nix (IFD), builds it with rustc/cargo 1.87 and runs every binary with --version. tests-libnix/run.sh builds them one by one with --keep-going and writes a RESULTS.md.

nix-build test.nix -A ripgrep.smoke
tests-libnix/run.sh

23 of 31 projects build and run, among them nushell, rust-analyzer, RustPython, ripgrep, bat, atuin, vaultwarden, tokio and axum. the other 8 (codex, leptos, mdBook, sniffnet, starship, typst, union, wasmtime) need a rustc newer than 1.87 at those revisions, that’s a toolchain limitation and not a libnix one.

smaller things

• flake.nix: cargo-libnix-git-head-via-ifd (the working tree) and cargo-libnix-git-revision-via-ifd (a pinned revision) build cargo (libnix) from source via IFD
• the dev shell uses the libnix-1.87.0+v4 release as its default cargo
• the README got sections on supported subcommands and features, how it works, requirements, a copy-pasteable shell.nix and how to export the build system for your own project

known issue

the generated nix code still contains one absolute path: the path of the .nix file of each build script run ends up in an error message. so a different output directory rebuilds the build script runs and everything depending on them. you notice it with cargo install, which builds in a new temporary directory every time, and with the IFD builds, which rebuild after every change to cargo. a fix is planned.

outlook

the next big topic is faster iteration (#3): computing a minimal file list per unit from rustc’s dep-info, so nix only rebuilds a root crate when one of its source files changed, and not when someone edits the README.md.

summary

with v4 cargo (libnix) covers build, test, run, install and clean, keeps its store paths alive as long as they are needed and cleans up after itself. it is still slower than vanilla cargo, but it is now usable for a normal day of rust development.