lastlog.de/blog
  • timeline
  • about
  • draft
  • roadmap
  • websocket
prev. article next article
nix-language-atlas

nix-language-atlas javascript

21 feb 2018

nixosjavascriptyarnnpmnode2nixnpm2nixyarn2nix

motivation

in the nix-language-atlas series on lastlog.de/blog i want to discuss how well programming languages, for which i’m familiar with, integrate with nix. let’s revisit javascript, as there had been major improvements since i wrote about it last time. we won’t look into the emscripten toolchain.

DSL-PM in general

are ‘domain specific language package manager(s)’ (DSL-PM) as npm or yarn a good thing?

  • pro(s):
    • npm/yarn is a DSL
    • npm/yarn works on linux/mac/windows (in contrast nix 1.x only works with linux/mac due to the shell/unix requirements but nix 2.x on windows 10 tells another story)
    • DSL-PM often use semantic versioning npm, golang dep, rust cargo while nix intentionally has no semantic versioning. to be fair, go isn’t the best example for semantic versioning and rust does pretty well without it as nixcrates proved.
  • con(s):
    • DSL-PMs like npm often lack support for inter domain dependency management, interestingly something which was better with autotools based build system where one was forced to pass arguments to not only header and libraries but also to binaries. this is where node2nix shines compared to yarn2nix
    • bundling/vendoring of dependencies: outside DSL-PM as bundling openssl c-code but often also inside of the DSL-PM most often seen with golang
    • deployment not reproducible: npm 5.x that supports lock files pinpointing the exact versions used of all dependencies and transitive dependencies, and a content addressable cache.

integrate DSL-PM(s) into nix/nixpkgs

DSL-PM properties nix requires:

  • reproducible dependency calculations/downloads
    • npm: only since npm 5.x, see also package-lock.json
    • yarn: yarn.lock made it easy, before that it required much effort
  • reproducible configuration, build & installation into the store of each dependency
  • reproducible configuration, build & installation into the store of main target

DSL-PM evolution (javascript)

  • npm -> npm2nix -> node2nix
  • yarn -> yarn2nix
  • bower -> bower2nix

note: read this if you are interested in npm/yarn differences. it seems that yarn forced many of its concepts into npm.

yarn2nix workflow

let’s see a simple example how to integrate a yarn based project into your nix codebase:

  1. we clone an example:

    git clone https://github.com/yarnpkg/example-yarn-package
cd example-yarn-package
mkdir bin
touch bin/myapp
chmod u+x bin/myapp

  2. let’s add a default.nix

    { pkgs ? import <nixpkgs> {} }:
let
  yarn2nixSrc = pkgs.fetchFromGitHub {
    owner = "moretea";
    repo = "yarn2nix";
    rev = "0472167f2fa329ee4673cedec79a659d23b02e06";
    sha256 = "10gmyrz07y4baq1a1rkip6h2k4fy9is6sjv487fndbc931lwmdaf";
  };
  yarn2nixRepo = pkgs.callPackage yarn2nixSrc {};
  inherit (yarn2nixRepo) mkYarnPackage;
in
  mkYarnPackage {
    src = ./.;
  }

    and add contents to bin/myapp

    #!/usr/bin/env node

'use strict';

// For package depenency demonstration purposes only
var multiply = require('lodash/multiply');

console.log("h" + multiply(2,0.5) + ", it works!")

    add a “bin” to package.json, see this patch:

    diff --git a/package.json b/package.json
index 36952fd..a38ceaf 100644
--- a/package.json
+++ b/package.json
@@ -18,6 +18,7 @@
   "devDependencies": {
     "jest-cli": "15.1.1"
   },
+  "bin" : { "myapp" : "bin/myapp" },
   "jest": {
     "testEnvironment": "node"
   }

  3. now build the source

    nix-build -Q
building path(s) ‘/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline’
building path(s) ‘/nix/store/b6kiqlcaacxrlq4nnjgk69mwjnrlvkpv-yarn2nix-modules-0.1.0’
building
yarn config v1.3.2
success Set "yarn-offline-mirror" to "/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline".
Done in 0.10s.
yarn install v1.3.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
warning Ignored scripts due to flag.
Done in 4.99s.
...
<skipped many lines>>
...
building path(s) ‘/nix/store/7yg1mah963w35dhxjcylw5q0r62bkk83-yarn.nix’
these derivations will be built:
  /nix/store/07xy1c1yp4ada1sjiz7m416ic2i1b908-webidl-conversions-3.0.1.tgz.drv
  /nix/store/0c2xgds6pzsnsp1hmxdgaqh1n28d68k5-lodash.isarray-3.0.4.tgz.drv
  /nix/store/0dj6mwhpmzvnxzz2122isdbw85mjg0wv-regex-cache-0.4.3.tgz.drv
  /nix/store/0in2pcwsc3ln6glqkab5fsx98m28vq8g-lodash._baseassign-3.2.0.tgz.drv
  /nix/store/1rhagnq3gz68x5lbhca6z4rsx9jnabgh-is-dotfile-1.0.2.tgz.drv
  /nix/store/1vq9rlkn3qdy2acndj2q47ryswiv5yir-lodash.assign-4.2.0.tgz.drv
  /nix/store/2a4kqk4p54nvm31cyy2nnl2vzr4klyb8-babel-generator-6.14.0.tgz.drv
  /nix/store/2mf2dr0gx3bqxiqbpqdhhp8bhz95d34h-arr-diff-2.0.0.tgz.drv
  /nix/store/2pbiyjyqg57dnn0apj24lrvqp1cbix9f-expand-brackets-0.1.5.tgz.drv
  /nix/store/31h1f7rair2xq0di3b3agr7g2xfgncyz-jest-matcher-utils-15.1.0.tgz.drv
  /nix/store/3l6amqm7wdx1c45mypbgh8xgsl92zw76-exec-sh-0.2.0.tgz.drv
  /nix/store/3w6wxplb811vzfky5sgvbifm6j5p6fac-babel-core-6.14.0.tgz.drv
  /nix/store/48d8jwrpgg2pxm8mkyqvavrv4k5i2nab-lodash.keys-3.1.2.tgz.drv
  /nix/store/4l0s8plnxw2nzszbjjdicrr5np9cc2ni-jest-resolve-15.0.1.tgz.drv
  /nix/store/4mn98xigyzybcnsb71731pw8biirzzax-micromatch-2.3.11.tgz.drv
...
<skipped many lines>>
...
building path(s) ‘/nix/store/chsx89ifr7dk7mc00d6789indh0rn41x-to-fast-properties-1.0.2.tgz’
building path(s) ‘/nix/store/sqpsd9y77kkq1vvfpfcfzk9q347dwg14-walker-1.0.7.tgz’
building path(s) ‘/nix/store/j7fp6iz0wj6dcp8ibsqyph2vy4p007wk-watch-0.10.0.tgz’
building path(s) ‘/nix/store/nv5p8ijp15mc2kbpp26mvd8pn7vqlwyy-webidl-conversions-3.0.1.tgz’
building path(s) ‘/nix/store/a3amdrmhzrvd3v1lsy1ih5rfpdamyj46-whatwg-url-3.0.0.tgz’
building path(s) ‘/nix/store/vhwym48w1pvgzli9i57fnzqw0pjp3a83-window-size-0.2.0.tgz’
building path(s) ‘/nix/store/2dg59ikzfrdwbcpkwg2xwjxs9rhpilb8-worker-farm-1.3.1.tgz’
building path(s) ‘/nix/store/497p7kg9iyb09ah0vvqx88dpq3bn843p-yargs-5.0.0.tgz’
building path(s) ‘/nix/store/w3lmwwq4cac653nhajnvyl1k2vhivaws-yargs-parser-3.2.0.tgz’
building path(s) ‘/nix/store/ywn6ysmnvrkyzjwpdiwr4iivk9z9kx2y-offline’
building path(s) ‘/nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0’
building path(s) ‘/nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0’
/nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0

    note: the last output line of nix-build is what we are interested in!

  4. checking the result

    ls -lathr result/node_modules/

...
<skipped many lines>>
...
lrwxrwxrwx 1 root root  102  1. Jan 1970  align-text -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/align-text
lrwxrwxrwx 1 root root  105  1. Jan 1970  acorn-globals -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn-globals
lrwxrwxrwx 1 root root   97  1. Jan 1970  acorn -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn
lrwxrwxrwx 1 root root   98  1. Jan 1970  abbrev -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abbrev
lrwxrwxrwx 1 root root   96  1. Jan 1970  abab -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abab

  5. execute the binary

    /nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0/bin/myapp
h1, it works!

using nix-build we declaratively built the node package!

yarn2nix internals

yarn (imperative)

  1. yarn -> read yarn.lock

  2. make / build all the downloads

  3. come up with node_packages folder

  4. yarn install

    note: in general we skip step (4.)

yarn (declarative)

  1. yarn2nix -> read yarn.lock

  2. translate each dependency into a mkDerivation

    each is a mkDerivation residing in /nix/store/…

  3. evaluate each mkDerivation, gain store path

  4. call yarn with the list of all mkDerivation(s), yarn comes up with node_modules

    a single mkDerivation also residing in /nix/store/… but encapsulating other /nix/store entries

  5. final mkDerivation uses this node_modules and creates the store path we are interested in

conclusion

  • take notice, and i can’t stress this enough, we map yarn.lock entries into single pkgs.stdenv.mkDerivation (by calling mkYarnPackage) but later pass all these into yarn which creates the node_modules contents from this. this is exactly how a DSL-PM must be designed to be easily integrated
  • yarn.lock obsoletes the requirement of manually creating a dependency file per project:
    • shown in the example above where i only added a default.nix
    • i’d wish dep’s Gopkg.lock used with go would also contain a hash already but they are using GIT and we have to call nix-prefetch-git to generate a sha256 hash manually
  • yarn2nix might not be as advanced as node2nix but we at nixcloud.io use it for all projects. yarn/yarn2nix is really fast, in dependency calculation and deployment, compared what we had before
  • i’d wish that npm’s sha1 would be replaced by something more recent but in comparison to dep they at least have a hash of some sort
  • yarn2nix is not a part of nixpkgs yet, see https://github.com/moretea/yarn2nix/issues/5
article source