libnix cargo-nix-backend

motivation

this post explores integrating nix directly into cargo as a continuation of the earlier rust abstractions post.

unlike previous attempts like cargo2nix, crate2nix, crane, and naersk to adapt cargo builds to nix tooling, this libnix initiative seeks to enhance cargo with an inherent nix backend.

so, what is cargo exactly?

cargo encompasses the following features:

• a utility to interpret the crates.io index and resolve dependencies using semver defined in Cargo.toml.
• a dependency retrieval tool supporting http(s), git, and ssh protocols.
• a translator: converting a dependency graph into a series of build steps, akin to generating Makefiles with dependencies.
• a parallel job scheduler for executing the build process.
• an integrated toolchain: cargo includes rustc and may need external tools like ld/lld for linking or compilers such as gcc for -sys crates.
• a manager for testing, generating documentation, and publishing crate updates to crates.io.

current status

cargo with libnix insight

curious about the source code? explore the README.md for a starting point.

cargo architecture overview

this map is the result of weeks of reading the cargo source code and figuring out where to place the nix build backend.

cargo libnix advantages

• merges development and deployment, uses same artifacts (new)
• sandbox builds (new), no forgotten dependencies
• sharing crate sources and related builds between different rust projects (new)
• sharing crate builds via binary substitutes: crate compilation gets more fine grained, so CI can also serve intermediate artifacts (new)
• more fine grained build dependencies via nix, i.e. only the crates with -sys need to have custom libraries and compilers exposed to them (new)
• better GC: no more artifacts in rm ./target/ as it is moved into /nix/store and nix-collect-garbage is used for cleanups (new)
• uses nix build scheduler instead of the one inside cargo (new)
• massive logging DX improvement over naersk, cargo2nix and other nix tools (new)
• supports all features cargo (no reimplementation like cargo2nix & similar)

cargo libnix downsides

• no .fingerprint support, lots of rebuilds when using lots of workspace-crates in project
• ideally the build system can be created soly from Cargo.toml/Cargo.lock in a reproducible way but that would require a nix implementation
• running cargo build with the nix backend is not reproducible
• nix backend works only on nix platforms (obvious)
• cargo.io index still lives in ~/.cargo/registry

diagrams

both imags below illustrate the new locations for files / artifacts used to build the cargo project using libnix.

allCollectedInputs = a: lib.unique (
  builtins.foldl' (acc: el: acc ++ [el] ++ (allCollectedInputs el.rust_crate_libraries)) [] a
);
rustc_linker_arguments = rust_crate_libraries: builtins.concatStringsSep " " (map (lib: "-L ${lib}") (allCollectedInputs rust_crate_libraries));
rustc_arguments = rust_crate_parent: assert lib.assertMsg (builtins.length rust_crate_parent <= 1) "passthru.rust_crate_parent must have 0 or 1 element at maxium" ;
  lib.replaceStrings ["\n"] [""] (builtins.concatStringsSep " " (map (crate:
    if builtins.pathExists "${crate}/nix/rustc-arguments" then
      builtins.readFile "${crate}/nix/rustc-arguments"
    else
      ""
    ) rust_crate_parent));
rustc_propagated_arguments = rust_crate_libraries: lib.replaceStrings ["\n"] [""] (builtins.concatStringsSep " " (map (crate:
  if builtins.pathExists "${crate}/nix/rustc-propagated-arguments" then
    builtins.readFile "${crate}/nix/rustc-propagated-arguments"
  else
    ""
  ) (allCollectedInputs rust_crate_libraries)));
environment_variables = rust_script_build_run: builtins.concatStringsSep " " (map (d: "${d}/nix/environment-variables") rust_script_build_run);
get_rust_crate_parent = parent_list:
  assert lib.assertMsg (builtins.length parent_list == 1) "item is supposed to have exactly one parent, while it has 0 or more than 1";
  builtins.head parent_list;

logone

logone is a custom logger crate library and standalone program written in rust, inspired by https://github.com/maralorn/nix-output-monitor at the high cost of reverse-engineering of the internal-json log format in nix.

logone became the successor of a proposed concept from 2014 it seems!

this is how logone displays the @nix / @cargo output to the user:

it converts this verbose nix trash outputs:

• screencast of nix build ... -v (i like this, except i never get warnings)
• screencast of nix build ... -L (way too verbose)

into something one actualy can consume as humans (it is very similar to cargo’s output):

• screencast of cargo build with libnix backend
• screencast of cargo build with libnix backend (showing an error)

it is very hard to use the current state of the @nix protocol and create a good DX with it. therefore i’ve created a few tickets which help to solve this:

• https://github.com/nixos/nix/issues/13958
• https://github.com/nixos/nix/issues/13935
• https://github.com/nixos/nix/issues/13910
• https://github.com/nixos/nix/issues/13909

logone @cargo protocol

in the libnix extension for cargo i’ve created the @cargo protocol, which is piggybacked onto @nix messages and extracted. this new @cargo protocol pretty much solves all the issues i addressed in the tickets above for the @nix protocol.

logone supports --level verbose mode, there i tried to map strings to id in order to map error message(s) to respective mkDerivation(s), this is very desperate and often fails. this needs to be fixed in nix.

below is the nix code which generates the @cargo error messages:

# generated from rustc-call.nix.handlebars using cargo (manual edits won't be persistent)
{ pkgs, fn, cargo, rustc, deps, build_parser, cargo-0_88_0-script_build-cfc654fccb259515 }: with deps;
  pkgs.stdenv.mkDerivation rec {
    name = "cargo-0_88_0-script_build_run-f5d51778f22880c0";
    meta.cargo_crate_info = {
      name = "cargo";
      version = "0.88.0";
      crate_hash = "f5d51778f22880c0";
    };
    buildInputs = [] ++ fn.inject meta.cargo_crate_info;
    passthru.rust_crate_libraries = [];
    passthru.rust_crate_parent = [cargo-0_88_0-script_build-cfc654fccb259515];
    passthru.rust_script_build_run = [curl-sys-0_4_80_plus_curl-8_12_1-db5fbe1d9680c71f libgit2-sys-0_18_0_plus_1_9_0-86c4b3f8f5bf526a];
    phases = "unpackPhase buildPhase";

    src = builtins.filterSource
      (path: type:
        let base = baseNameOf path;
        in !(base == "target" || base == "result" || builtins.match "result-*" base != null)
      ) /home/nixos/cargo;
    unpackPhase = "";

    RUSTC = "${rustc}/bin/rustc";
    CARGO = "${cargo}/bin/cargo";

    CARGO_CFG_FEATURE = "";
    CARGO_CFG_PANIC = "unwind";
    CARGO_CFG_TARGET_ABI = "";
    CARGO_CFG_TARGET_ARCH = "x86_64";
    CARGO_CFG_TARGET_ENDIAN = "little";
    CARGO_CFG_TARGET_ENV = "gnu";
    CARGO_CFG_TARGET_FAMILY = "unix";
    CARGO_CFG_TARGET_FEATURE = "fxsr,sse,sse2";
    CARGO_CFG_TARGET_HAS_ATOMIC = "16,32,64,8,ptr";
    CARGO_CFG_TARGET_OS = "linux";
    CARGO_CFG_TARGET_POINTER_WIDTH = "64";
    CARGO_CFG_TARGET_VENDOR = "unknown";
    CARGO_CFG_UNIX = "";
    CARGO_ENCODED_RUSTFLAGS = "";
    CARGO_MANIFEST_DIR = "./";
    CARGO_MANIFEST_PATH = "./Cargo.toml";
    CARGO_PKG_AUTHORS = "";
    CARGO_PKG_DESCRIPTION = "Cargo, a package manager for Rust.";
    CARGO_PKG_HOMEPAGE = "https://doc.rust-lang.org/cargo/index.html";
    CARGO_PKG_LICENSE = "MIT OR Apache-2.0";
    CARGO_PKG_LICENSE_FILE = "";
    CARGO_PKG_NAME = "cargo";
    CARGO_PKG_README = "README.md";
    CARGO_PKG_REPOSITORY = "https://github.com/rust-lang/cargo";
    CARGO_PKG_RUST_VERSION = "1.85";
    CARGO_PKG_VERSION = "0.88.0";
    CARGO_PKG_VERSION_MAJOR = "0";
    CARGO_PKG_VERSION_MINOR = "88";
    CARGO_PKG_VERSION_PATCH = "0";
    CARGO_PKG_VERSION_PRE = "";
    DEBUG = "true";
    HOST = "x86_64-unknown-linux-gnu";
    NUM_JOBS = "16";
    OPT_LEVEL = "0";
    PROFILE = "debug";
    RUSTC_WORKSPACE_WRAPPER = "";
    RUSTC_WRAPPER = "";
    RUSTDOC = "rustdoc";
    RUSTFLAGS = "";
    TARGET = "x86_64-unknown-linux-gnu";

    buildPhase = ''
      export CARGO_MANIFEST_DIR=$(realpath $PWD/$CARGO_MANIFEST_DIR)
      export CARGO_MANIFEST_PATH=$(realpath $PWD/$CARGO_MANIFEST_PATH)

      mkdir -p $out/nix
      export OUT_DIR=$out
      export INC_DIR=$(${pkgs.mktemp}/bin/mktemp -d)

      echo -e "\e[92mCompiling\e[0m cargo-0_88_0-script_build_run-f5d51778f22880c0"
      echo "@cargo { \"type\":0, \"crate_name\":\"cargo\", \"id\":\"cargo-0_88_0-script_build_run-f5d51778f22880c0\" }"
      for file in ${fn.environment_variables passthru.rust_script_build_run}; do
        if [ -f $file ]; then
          set -a
            while read -r line; do
              echo -e "\033[38;5;208m$line\033[0m"
            done < "$file"
            source $file
            set +a
        fi
      done
      build_script_build_output_lines=$(${pkgs.mktemp}/bin/mktemp)
      set -x +e
      ${ cargo-0_88_0-script_build-cfc654fccb259515 }/build_script_build > $OUT_DIR/nix/build_script_build.out 2> $build_script_build_output_lines
      build_script_build_exit_value=$?
      set +x -e
      if [ "$build_script_build_exit_value" -ne 0 ]; then
          output=$(${pkgs.jq}/bin/jq -c -n \
              --arg crate_name "cargo" \
              --arg notice "There was an error executing build_script_build in file: '/home/nixos/cargo/target/debug/nix/derivations/cargo-0.88.0-script_build_run-f5d51778f22880c0.nix':" \
              --arg exit_code "$build_script_build_exit_value" \
              --rawfile msg $build_script_build_output_lines \
              '{type: 3, crate_name: $crate_name, exit_code: ($exit_code|tonumber), messages: [ $notice, $msg ] }')
          printf '@cargo %s\n' "$output"
          cat $build_script_build_output_lines
          exit $build_script_build_exit_value
      fi

      build_parser_output_lines=$(${pkgs.mktemp}/bin/mktemp)
      set -x +e
      ${build_parser}/bin/cargo-build_script_build-parser $OUT_DIR/nix/build_script_build.out --out-path $out/nix write-results 2> $build_parser_output_lines
      build_parser_exit_value=$?
      set +x -e
      if [ "$build_parser_exit_value" -ne 0 ]; then
          output=$(${pkgs.jq}/bin/jq -c -n \
              --arg crate_name "cargo" \
              --arg notice "There was an error executing build_parser in file: '/home/nixos/cargo/target/debug/nix/derivations/cargo-0.88.0-script_build_run-f5d51778f22880c0.nix':" \
              --arg exit_code "$build_parser_exit_value" \
              --rawfile msg $build_parser_output_lines \
              '{type: 3, crate_name: $crate_name, exit_code: ($exit_code|tonumber), messages: [ $notice, $msg ] }')
          printf '@cargo %s\n' "$output"
          cat $build_parser_output_lines
          exit $build_parser_exit_value
      fi
      echo "@cargo {\"type\": 3, \"crate_name\": \"cargo\", \"exit_code\": 0, \"messages\": []}"
    '';
}

logging principles fundamentals summarized

traditionally a nix packager assumes that the software in question compiles and works and warnings during nix build are ignored it being c++ warnings, rustc warnings or other. as a result this leaves two types of error messages a nix packager is interested in:

1. one or more mkDerivation(s) failed:

   what caused the problem? most often, missing system library or wrong version of the toolchains used.

   a nix stack trace is only of little help.

2. nix programming error

   we are interested in a nix stack trace.

nix logging seen from the cargo build perspective:

with the advent of the libnix concept, shown in this cargo extension, we must adapt the logging:

generated build system

running this command below will create a build system in target/debug/nix, call nix build internally and symlink the results at the end

CARGO_BACKEND=nix cargo build -v

for comparison: the build system to compile cargo itself, created by cargo build with the nix backend, is 2 Mib uncompressed.

once the build system is in place, after running cargo build, one can evaluate it manually in order to experiment with the implementation and to debug code:

nix build target --file /home/nixos/cargo/target/debug/nix/cargo_build_caller.nix --out-link /home/nixos/cargo/target/debug/nix/gc/result --json

this shows the root of the nix build system, the gc folder holds the symlink of the build so the result won’t be garbage-collected.

[nixos@nixos:~/cargo]$ ls -la target/debug/nix
total 20
drwxr-xr-x 4 nixos users 4096 Sep 20 01:16 .
drwxr-xr-x 8 nixos users 4096 Sep 22 08:33 ..
-rw-r--r-- 1 nixos users 1624 Sep 22 08:29 cargo_build_caller.nix
drwxr-xr-x 3 nixos users 4096 Sep 20 01:13 derivations
drwxr-xr-x 2 nixos users 4096 Sep 22 08:33 gc

derivations contains all the bins/libs which are no dependencies like those added with cargo add serde.

[nixos@nixos:~/cargo]$ ls -la target/debug/nix/derivations/
total 200
drwxr-xr-x 3 nixos users  4096 Sep 20 01:13 .
drwxr-xr-x 4 nixos users  4096 Sep 20 01:16 ..
-rw-r--r-- 1 nixos users 15626 Sep 22 08:29 cargo-0.88.0-27e7993d9cf32df7.nix
-rw-r--r-- 1 nixos users 15478 Sep 22 08:29 cargo-0.88.0-bin-85e09d7d8299b1ad.nix
-rw-r--r-- 1 nixos users  4706 Sep 22 08:29 cargo-0.88.0-script_build-cfc654fccb259515.nix
-rw-r--r-- 1 nixos users  5373 Sep 22 08:29 cargo-0.88.0-script_build_run-f5d51778f22880c0.nix
-rw-r--r-- 1 nixos users  5209 Sep 22 08:29 cargo-credential-0.4.8-a5adc6ab9fe103b0.nix
-rw-r--r-- 1 nixos users  5049 Sep 22 08:29 cargo-credential-libsecret-0.4.13-4e698a0b35f72d06.nix
-rw-r--r-- 1 nixos users  4496 Sep 22 08:29 cargo-platform-0.2.0-c5f768769f22a333.nix
-rw-r--r-- 1 nixos users  5918 Sep 22 08:29 cargo-util-0.2.20-7087e4a73afc7b23.nix
-rw-r--r-- 1 nixos users  5520 Sep 22 08:29 cargo-util-schemas-0.8.1-bce7b79eff35b46a.nix
-rw-r--r-- 1 nixos users  5132 Sep 22 08:29 crates-io-0.40.10-cb0425982b906266.nix
-rw-r--r-- 1 nixos users 47411 Sep 22 08:29 default.nix
drwxr-xr-x 2 nixos users 36864 Sep 20 01:13 deps
-rw-r--r-- 1 nixos users  4923 Sep 22 08:29 rustfix-0.9.0-9f1c66820d29e14a.nix
-rw-r--r-- 1 nixos users   276 Sep 22 08:29 target.nix

this folder contains all the crate dependencies considered read only.

[nixos@nixos:~/cargo]$ ls -la target/debug/nix/derivations/deps
total 3028
drwxr-xr-x 2 nixos users 36864 Sep 20 01:13 .
drwxr-xr-x 3 nixos users  4096 Sep 20 01:13 ..
-rw-r--r-- 1 nixos users  4010 Sep 22 08:29 adler2-2.0.0-115180b36279fc7c.nix
-rw-r--r-- 1 nixos users  5172 Sep 22 08:29 ahash-0.8.11-8f60023f209663c7.nix
-rw-r--r-- 1 nixos users  4313 Sep 22 08:29 ahash-0.8.11-script_build-dfb7cac56dd48573.nix
-rw-r--r-- 1 nixos users  5300 Sep 22 08:29 ahash-0.8.11-script_build_run-e66d9dc18aec0370.nix
-rw-r--r-- 1 nixos users  4283 Sep 22 08:29 aho-corasick-1.1.3-4faf1ab2f37c32c1.nix
-rw-r--r-- 1 nixos users  4208 Sep 22 08:29 allocator-api2-0.2.21-bd3713078dfee01f.nix
-rw-r--r-- 1 nixos users  7585 Sep 22 08:29 annotate-snippets-0.11.5-4d47bac9cbcd3256.nix
-rw-r--r-- 1 nixos users  8187 Sep 22 08:29 anstream-0.6.18-3af54164fe68ad61.nix
-rw-r--r-- 1 nixos users  7141 Sep 22 08:29 anstyle-1.0.10-bf6d032cb7d79be1.nix
-rw-r--r-- 1 nixos users  7360 Sep 22 08:29 anstyle-parse-0.2.6-7e3167a48452c319.nix
-rw-r--r-- 1 nixos users  7092 Sep 22 08:29 anstyle-query-1.1.2-df1354162236cfa0.nix
-rw-r--r-- 1 nixos users  4728 Sep 22 08:29 anyhow-1.0.96-139173be5e005a44.nix
....

[nixos@nixos:~/cargo]$ cat target/debug/nix/cargo_build_caller.nix
# generated from cargo_build_caller.nix.handlebars using cargo (manual edits won't be persistent)
{ system ? builtins.currentSystem }:
let
  nixpkgsSrc = builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/25.05.tar.gz";
    sha256 = "sha256:1915r28xc4znrh2vf4rrjnxldw2imysz819gzhk9qlrkqanmfsxd";
  };
  pkgs = import (nixpkgsSrc + "/pkgs/top-level/default.nix") {
    localSystem = { inherit system; };
  };
  # nix-prefetch-git --url https://github.com/nixcloud/cargo-build_script_build-parser --branch-name main
  build_parser_sources = pkgs.fetchFromGitHub {
    owner = "nixcloud";
    repo = "cargo-build_script_build-parser";
    rev = "788a202b8646aae41761a70c8658878bdb8f0017";
    sha256 = "1qi32hnkas9xfvwzjhy95vm7b0h8aj1fx1i9367qv1j8ng8hzcyn";
  };
  build_parser = pkgs.callPackage build_parser_sources {};
  fenixSrc = pkgs.fetchFromGitHub {
    owner = "nix-community";
    repo = "fenix";
    rev = "6ed03ef4c8ec36d193c18e06b9ecddde78fb7e42";
    sha256 = "sha256-tl/0cnsqB/Yt7DbaGMel2RLa7QG5elA8lkaOXli6VdY=";
  };
  fenix = import fenixSrc {};
  external_crate_dependencies =
    (if builtins.pathExists ../../../Cargo.dependencies.nix
    then builtins.trace "Using Cargo.dependencies.nix"
         import ../../../Cargo.dependencies.nix { inherit pkgs; }
    else builtins.trace "No Cargo.dependencies.nix found"
         { deps = {}; });
  toolchain = fenix.stable.toolchain;
  cargoPackages = import ./derivations/default.nix {
    inherit pkgs external_crate_dependencies build_parser;
    rustc = toolchain;
    cargo = toolchain;
    project_root = ./.;
  };
in
cargoPackages

{ pkgs }:
with pkgs;
{
    deps = {
        "markup5ever_rcdom" = {
            "0.3.0" =
                [ pkg-config openssl ];
        };
        "unicode-ident" = [ pkg-config curl ];
        "xml5ever" = {
            "0.20.0" = [];
        };
    };
}

comparing to other implementation

cargo2nix / crate2nix

• both use a standalone program written in rust while using parts of the cargo source code as a library.
• cargo2nix creates a Crates.nix while crate2nix creates a Cargo.nix.
• both are using parts from cargo and does not attempt to reimplement it, which is good.

crane + naersk

• reimplementations of cargo’s build system in nix which is
• exectly the oposite of what cargo2nix, crate2nix and my cargo + nix backend want to be.

cargo (with nix backend) determinnism

question: is calling CARGO_BACKEND=nix cargo build deterministic?

short answer: no, as it is written in rust and not nix! thus we have IFD problems when using it ‘on the fly’. however we could avoid IFD by shipping the generated nix toolchain, at the cost of lots of additional files (127kb bz2 compressed archive for cargo itself). this is similar to what we do for c/cpp projects in nixpkgs. the main difference between the openssl expression in nixpkgs and the crates here is that stuff in nixpkgs is handcrafted and this is automated.

cargo rewrites like crane/naersk solve this at the cost of being detached from the upsteam cargo implementation often leading to unsupported edge-cases.

long answer:

ideally the build system inputs are soly:

1. a projects src
2. Cargo.toml/Cargo.lock
3. Cargo.Dependencies.nix

and the build system is implemented as nix code base and synthesized deterministically on the fly:

that would be the ideal goal! but this paradigm shift should be attempted after integrating nix into cargo and NOT BEFORE.

time statistics

it is very hard to compare the two build systems:

time cargo build
real 1m15.856s

time CARGO_BACKEND=nix ../cargo/target/debug/cargo build 
real 2m31.875s

note:

• the normal cargo build was a cold start.
• the nix backend build requires the build-parser to be built and a nixpkgs download on top. a nix-collect-garbage -d removes part of the toolchain.

a key takeaway is, if you are a rust developer you are not interested in absolute build times but more in iterative build times and the current implementaion looses when there are many project crates as in the cargo example, as all of them have to be compiled each iteration hence no .fingerprint support and also the copying of the source code.

outlook

next steps involve:

• get more rust projects to compile
• integrate backend into cargo install
• create upstream PR and collect cargo developer feedback
• create a fenix/oxaclica like overlay so that developers can easily use this work
• support .fingerprint concept
• use bwrap so no source code copying is needed

thanks

i would like to express my heartfelt thanks to:

• nlnet for the confidence in me with sponsoring this effort
• https://x.com/bereknyei huge shoutout to thomas bereknyei, who inspired me with the @nix protocol and --log-format internal-json and other concepts

and to these useful services (no sponsors):

• https://vscodium.com/ + https://github.com/jeanp413/open-remote-ssh + https://github.com/direnv/direnv-vscode.git
• https://chatgpt.com was a huge benefit in building the concept and also in creating the actual code
• https://replit.com/ helped to create the build.rs parser and the logone concept
• https://excalidraw.com/ which was used to create all graphs
• https://grok.com/ for occational uses

the libnix project was sponsored by the EU (and me).

summary

this project presents a novel approach with the nix backend integrated into cargo to enhance the rust workflow. it offers a more native developer experience and leverages the strengths of nix. however, there are some challenges, such as the absence of .fingerprint support, leading to increased compilation time due to the necessity of copying store paths.

in the future, when nix becomes available on windows, we could make the code generator reproducible by utilizing a nix code base on all platforms, eliminating the need to distribute the toolchain for each project.