10 may 2024
status of native windows nix using MinGW from my series libnix.
last post we were looking into nix on windows from the nix master branch which uses MinGW.
in this post we’ll focus on volth’s work running nix on windows back in the year 2020.
volth ported nix to windows in ~2020.
using nix-build
he was able to build boost
1.74 using a msvc 2019 toolchain:
-keep-failed --option system x86_64-windows -o x86_64-boost -E
\bin\nix-build.exe -"with (import <nixpkgs> { }); pkgsMsvc2019.boost174.override{ staticRuntime=true; static=true; }"
sadly it wasn’t merged into upstream nix and i (qknight) was never able to run/build his code. at least we got the code, so let’s what he did!
branch diff (all commits combined):
70 commits, started around ~2019 and ended late 2020. +14664 additions and -1348 deletions.
branch diff (all commits combined):
270 commits, started late 2018. 133 changed files, showing +41167 additions and -1071 deletions.
build nix 2.1.3 from native windows using mingw+meson based toolchain
a single user setup -> no nix-daemon implementation
probably using visual studio for development
probably used cmd (build-meson-32-mt.cmd) because for powershell this would be called (build-meson-32-mt.ps)
featured msvc and mingw compilers
john ericson added meson build system support for nix (replacing autotools), let’s have a closer look at this:
there have been proposals for other build systems over the years:
in upstream nix
autotools
is a requirement. autotools
can’t be
used on native windows! i’ve always seen autotools
as
ancient technology, acting as a vendor lock-in for
gnu/unix systems.
the MinGW setup, which is discussed in the libnix series, has a clever workaround to this problem: a cross-compiler setup which comes with the downside, that it only works from linux/mac. WSL2/nixos-wsl makes this downside weight a little less, still it hurts!
either Meson
or CMake
as
replacement for autotools
build system are mandatory for
the windows nix port in order to compile nix from windows
directly.
a clear case for this quote:
spock: “the needs of the many outweigh the needs of the few, …, or the one”
in the nix programming language, builtins
are functions
which can be called, but which are not written in the nix language
themselves.
runProgram()
was replaced by
runProgramGetStdout()
which could call git
,
hg
, tar
, unzip
, xz
,
gzip
, tr
builtins.fetchGit
->
src/libexpr/primops/fetchGit.ccbuiltins.fetchMercurial
->
src/libexpr/primops/fetchMercurial.ccbuiltins.fetchurl
/ nix-prefetch-url
->
src/nix-prefetch-url/nix-prefetch-url.cc - calls unzip
/
tar
git
, hg
,
tar
, unzip
is unclear: i guess some are from
cygwin
while others are form mingw64
installation.looking at the build system, checking the the source files for patches i conclude:
working:
not working:
working nix
subcommands:
not working nix
subcommands
note: this list is incomplete but still gives a
broad overview. for instance, it seems that nix build
was
not supported, but the program nix-build
was.
the stdenv
is a build-environment, which sets
environment variables as PATH to build software. a minimalistic
installation of a compiler basically.
volth maintained these envs:
those are used in the file build-meson-32-mt.cmd and supposedly buildable:
however: using these stdenvs one could attempt a build of any software in nixpkgs so we can’t know what builds without trying. it is really worth pointing out that volth’s boost’s volths generic.nix looks nothing like the original upstream generic.nix.
looking at volths generic.nix:
''
installPhase = print("EXEC: b2 ${b2Args} install\n");
die $! if system("b2 ${b2Args} install");
renameL("$ENV{out}/include/boost-".('${version}' =~ s/^(\d+)\.(\d+).*/$1_$2/r)."/boost", "$ENV{out}/include/boost") or die $!;
rmdirL("$ENV{out}/include/boost-".('${version}' =~ s/^(\d+)\.(\d+).*/$1_$2/r)) or die $!;
'' + stdenv.lib.optionalString (!static) ''
mkdirL("$ENV{out}/bin") or die $!;
for my $dll (glob("$ENV{out}/lib/*.dll")) {
renameL($dll, "$ENV{out}/bin/".basename($dll)) or die $!;
}
'';
using the perl renameL function volth copied all dll files into the bin directory. there is no rpath concept on windows.
the main difference between these two toolchains on windows are performance, developer experience (debugging), API availability and licensing.
i never worked with visual studio / msvc much but for the purpose of libnix, i think MinGW is a good choice for porting nix and create a custom MinGW toolchain inside nix should be sufficient.
config.nix shows a variety of tools/locations of his bootstrapping environment. i personaly would like to see a “c:\” instead of unix paths “/”.
let
fromEnv = var: def:
let val = builtins.getEnv var; in
if val != "" then val else def;
in rec {
shell = "/usr/bin/bash";
coreutils = "/usr/bin";
bzip2 = "/mingw64/bin/bzip2";
gzip = "/usr/bin/gzip";
xz = "/mingw64/bin/xz";
tar = "/usr/bin/tar";
tarFlags = "--warning=no-timestamp";
tr = "/usr/bin/tr";
nixBinDir = fromEnv "NIX_BIN_DIR" "/usr/bin";
nixPrefix = "/usr";
nixLibexecDir = fromEnv "NIX_LIBEXEC_DIR" "/usr/libexec";
nixLocalstateDir = "/nix/var";
nixSysconfDir = "/usr/etc";
nixStoreDir = fromEnv "NIX_STORE_DIR" "/nix/store";
# If Nix is installed in the Nix store, then automatically add it as
# a dependency to the core packages. This ensures that they work
# properly in a chroot.
chrootDeps =
if dirOf nixPrefix == builtins.storeDir then
[ (builtins.storePath nixPrefix) ]
else
[ ];
}
nix.exe
follows the unix principle of ‘putting all the
functionality into one binary’.
volth put some effort into this by getting some subcommands for
nix.exe
working.
volth put some effort into
run(ref<Store> store) override
function for getting
it to work on windows
void run(ref<Store> store) override
{
auto outPaths = toStorePaths(store, Build, installables);
...
#ifdef _WIN32
if (boost::algorithm::iends_with(programName, ".exe")) {
= programName.substr(0, programName.size()-4);
programName }
#endif
void listText(ref<FSAccessor> accessor)
{
std::function<void(const FSAccessor::Stat &, const Path &, const std::string &, bool)> doPath;
auto showFile = [&](const Path & curPath, const std::string & relPath) {
if (verbose) {
auto st = accessor->stat1(curPath);
std::string tp =
.type == FSAccessor::Type::tRegular ? (
st#ifndef _WIN32
.isExecutable ? "-r-xr-xr-x" :
st#endif
"-r--r--r--") :
chatGPT: libexpr
typically refer to the
library that handles nix expressions. nix expressions are a
domain-specific language (DSL) used in nix and nixos for defining
packages, configurations, and system setups.
very few changes were required here!
Path EvalState::coerceToPath(const Pos & pos, Value & v, PathSet & context)
::coerceToPath(const Pos & pos, Value & v, PathSet & context)
Path EvalState{
= coerceToString(pos, v, context, false, false);
string path #ifdef _WIN32
if (path.length() >= 7 && path[0] == '\\' && path[1] == '\\' && (path[2] == '.' || path[2] == '?') && path[3] == '\\' &&
('A' <= path[4] && path[4] <= 'Z') && path[5] == ':' && isslash(path[6])) {
return path;
}
if (path.length() >= 3 && (('A' <= path[0] && path[0] <= 'Z') || ('a' <= path[0] && path[0] <= 'z')) && path[1] == ':' && isslash(path[2])) {
return path;
}
("string '%1%' doesn't represent an absolute path, at %2%", path, pos);
throwEvalError#endif
std::pair<string, string> decodeContext(const string & s)
chatGPT: libutil
is platform
abstraction: these libraries may abstract away platform-specific details
and provide a consistent interface for interacting with the underlying
operating system. this can help ensure portability and make it easier to
maintain and support Nix across different platforms.
all changes are for platform/path handling.
src/libutil/util.cc, namely:
std::string to_bytes(const std::wstring & path) {
std::wstring from_bytes(const std::string & s) {
optional<std::wstring> maybePathW(const Path & path) {
std::wstring pathW(const Path & path) {
std::wstring handleToFileName(HANDLE handle) {
Path handleToPath(HANDLE handle) {
std::string WinError::addLastError(const std::string & s)
std::wstring getCwdW()
std::wstring getArgv0W()
std::wstring getEnvW(const std::wstring & key, const std::wstring & def)
string getEnv(const string & key, const string & def)
std::map<std::wstring, std::wstring, no_case_compare> getEntireEnvW()
Path absPath(Path path, Path dir)
Path canonPath(const Path & path, bool resolveSymlinks) {
Path canonNarPath(const Path & path)
chatGPT: libstore
is a library that
provides functionality related to managing software on the system. in
nix this does the following things:
note: C:\nix\store was the store root and referenced by /nix/store in the code.
see also https://nixos.org/manual/nix/stable/store/types/local-store for different store types.
void DerivationGoal::startBuilder()
doing
tmpDir = tmpDirOrig = createTempDir("", "nix-build-" + drvName, false, false);
prepare buildvoid DerivationGoal::startBuilder()
->
env variables
get exposed to the builder -> uenv ->
uenvline -> CreateProcessW(..)
void DerivationGoal::deleteTmpDir(bool force)
void DerivationGoal::handleChildOutput(HANDLE handle, const string & data)
libcurl
work with sleep
(vs. callback)tar
with MSYS filesystem compatibilitystatic void makeSymlink(const Path & link, const Path & target)
void LocalStore::removeUnusedLinks(const GCState & state)
void canonicaliseTimestampAndPermissions(const Path & path)
static void canonicalisePathMetaData_(const std::wstring & wpath, const WIN32_FIND_DATAW * wpathFD /* attributes might be known */, InodesSeen & inodesSeen)
void canonicalisePathMetaData(const Path & path, InodesSeen & inodesSeen)
i->isExecutable
CreateHardLinkW
/ CreateFileW
file
deduplicationLocalStore::InodeHash LocalStore::loadInodeHash()
Strings LocalStore::readDirectoryIgnoringInodes(const Path & path, const InodeHash & inodeHash)
void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats, const Path & path, InodeHash & inodeHash)
AutoCloseWindowsHandle openLockFile(const Path & path, bool create)
void deleteLockFile(const Path & path)
bool lockFile(HANDLE handle, LockType lockType, bool wait)
bool PathLocks::lockPaths(const PathSet & _paths, const string & waitMsg, bool wait)
tests/local.mk contains some interesting mentions:
# these do not pass (yet) on MINGW:
# gc-concurrent.sh <-- fails trying to delete open .lock-file, need runtimeRoots
# gc-runtime.sh <-- runtimeRoots not implemented yet
# user-envs.sh <-- nix-env is not implemented yet
# remote-store.sh <-- not implemented yet
# secure-drv-outputs.sh <-- needs nix-daemin which is not ported yet
# nix-channel.sh <-- nix-channel is not implemented yet
# nix-profile.sh <-- not implemented yet
# case-hack.sh <-- not implemented yet (it might have Windows-specific)
# nix-shell.sh <-- not implemented yet
# linux-sandbox.sh <-- not implemented (Docker can be use on Windows for sandboxing)
# plugins.sh <-- not implemented yet
# nix-copy-ssh.sh <-- not implemented yet
# build-remote.sh <-- not implemented yet
# binary-cache.sh <-- \* does not work in MSYS Bash (https://superuser.com/questions/897599/escaping-asterisk-in-bash-on-windows)
# nar-access.sh <-- not possible to have args '/foo/data' (paths inside nar) without magic msys path translation (maybe `bash -c '...'` will work?)
we need to create a cross-platform stdenv
and from this
discussion https://discourse.nixos.org/t/nix-on-windows/1113 i
extracted this:
ruby
nushell
see https://determinate.systems/posts/nuenv/ from luc
perkins (see also the nixcon23 talk on
YT)i find nushell
the most interesting as it contains many
helper programs like cp
and also has an interesting
programming language.
volth’s work has been kept alive, thanks john ericson! furthermore, john ericson even managed to already integrate parts of volth’s work into nix!
volth’s amazing prototyping work shows that it is feasible to use nix on windows. thanks volth! the code base is a huge inspiration and shows how much is already possible and where we need to focus work!
some platform limitations volth faced, and in parts even fixed, are now overcome by microsoft on windows 10 and onwards, namely:
however, there is still some major issue to be dealt with:
let’s realize the potential of nix by integrating volth’s findings into nix!