nix-language-atlas javascript

21 feb 2018

motivation

in the nix-language-atlas series on lastlog.de/blog i want to discuss how well programming languages, for which i’m familiar with, integrate with nix. let’s revisit javascript, as there had been major improvements since i wrote about it last time. we won’t look into the emscripten toolchain.

DSL-PM in general

are ‘domain specific language package manager(s)’ (DSL-PM) as npm or yarn a good thing?

integrate DSL-PM(s) into nix/nixpkgs

DSL-PM properties nix requires:

  • reproducible dependency calculations/downloads
  • reproducible configuration, build & installation into the store of each dependency
  • reproducible configuration, build & installation into the store of main target

DSL-PM evolution (javascript)

note: read this if you are interested in npm/yarn differences. it seems that yarn forced many of its concepts into npm.

yarn2nix workflow

let’s see a simple example how to integrate a yarn based project into your nix codebase:

  1. we clone an example:

    git clone https://github.com/yarnpkg/example-yarn-package
cd example-yarn-package
mkdir bin
touch bin/myapp
chmod u+x bin/myapp

  2. let’s add a default.nix

    { pkgs ? import <nixpkgs> {} }:
let
  yarn2nixSrc = pkgs.fetchFromGitHub {
    owner = "moretea";
    repo = "yarn2nix";
    rev = "0472167f2fa329ee4673cedec79a659d23b02e06";
    sha256 = "10gmyrz07y4baq1a1rkip6h2k4fy9is6sjv487fndbc931lwmdaf";
  };
  yarn2nixRepo = pkgs.callPackage yarn2nixSrc {};
  inherit (yarn2nixRepo) mkYarnPackage;
in
  mkYarnPackage {
    src = ./.;
  }

    and add contents to bin/myapp

    #!/usr/bin/env node

'use strict';

// For package depenency demonstration purposes only
var multiply = require('lodash/multiply');

console.log("h" + multiply(2,0.5) + ", it works!")

    add a “bin” to package.json, see this patch:

    diff --git a/package.json b/package.json
index 36952fd..a38ceaf 100644
--- a/package.json
+++ b/package.json
@@ -18,6 +18,7 @@
   "devDependencies": {
     "jest-cli": "15.1.1"
   },
+  "bin" : { "myapp" : "bin/myapp" },
   "jest": {
     "testEnvironment": "node"
   }

  3. now build the source

    nix-build -Q
building path(s) ‘/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline’
building path(s) ‘/nix/store/b6kiqlcaacxrlq4nnjgk69mwjnrlvkpv-yarn2nix-modules-0.1.0’
building
yarn config v1.3.2
success Set "yarn-offline-mirror" to "/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline".
Done in 0.10s.
yarn install v1.3.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
warning Ignored scripts due to flag.
Done in 4.99s.
...
<skipped many lines>>
...
building path(s) ‘/nix/store/7yg1mah963w35dhxjcylw5q0r62bkk83-yarn.nix’
these derivations will be built:
  /nix/store/07xy1c1yp4ada1sjiz7m416ic2i1b908-webidl-conversions-3.0.1.tgz.drv
  /nix/store/0c2xgds6pzsnsp1hmxdgaqh1n28d68k5-lodash.isarray-3.0.4.tgz.drv
  /nix/store/0dj6mwhpmzvnxzz2122isdbw85mjg0wv-regex-cache-0.4.3.tgz.drv
  /nix/store/0in2pcwsc3ln6glqkab5fsx98m28vq8g-lodash._baseassign-3.2.0.tgz.drv
  /nix/store/1rhagnq3gz68x5lbhca6z4rsx9jnabgh-is-dotfile-1.0.2.tgz.drv
  /nix/store/1vq9rlkn3qdy2acndj2q47ryswiv5yir-lodash.assign-4.2.0.tgz.drv
  /nix/store/2a4kqk4p54nvm31cyy2nnl2vzr4klyb8-babel-generator-6.14.0.tgz.drv
  /nix/store/2mf2dr0gx3bqxiqbpqdhhp8bhz95d34h-arr-diff-2.0.0.tgz.drv
  /nix/store/2pbiyjyqg57dnn0apj24lrvqp1cbix9f-expand-brackets-0.1.5.tgz.drv
  /nix/store/31h1f7rair2xq0di3b3agr7g2xfgncyz-jest-matcher-utils-15.1.0.tgz.drv
  /nix/store/3l6amqm7wdx1c45mypbgh8xgsl92zw76-exec-sh-0.2.0.tgz.drv
  /nix/store/3w6wxplb811vzfky5sgvbifm6j5p6fac-babel-core-6.14.0.tgz.drv
  /nix/store/48d8jwrpgg2pxm8mkyqvavrv4k5i2nab-lodash.keys-3.1.2.tgz.drv
  /nix/store/4l0s8plnxw2nzszbjjdicrr5np9cc2ni-jest-resolve-15.0.1.tgz.drv
  /nix/store/4mn98xigyzybcnsb71731pw8biirzzax-micromatch-2.3.11.tgz.drv
...
<skipped many lines>>
...
building path(s) ‘/nix/store/chsx89ifr7dk7mc00d6789indh0rn41x-to-fast-properties-1.0.2.tgz’
building path(s) ‘/nix/store/sqpsd9y77kkq1vvfpfcfzk9q347dwg14-walker-1.0.7.tgz’
building path(s) ‘/nix/store/j7fp6iz0wj6dcp8ibsqyph2vy4p007wk-watch-0.10.0.tgz’
building path(s) ‘/nix/store/nv5p8ijp15mc2kbpp26mvd8pn7vqlwyy-webidl-conversions-3.0.1.tgz’
building path(s) ‘/nix/store/a3amdrmhzrvd3v1lsy1ih5rfpdamyj46-whatwg-url-3.0.0.tgz’
building path(s) ‘/nix/store/vhwym48w1pvgzli9i57fnzqw0pjp3a83-window-size-0.2.0.tgz’
building path(s) ‘/nix/store/2dg59ikzfrdwbcpkwg2xwjxs9rhpilb8-worker-farm-1.3.1.tgz’
building path(s) ‘/nix/store/497p7kg9iyb09ah0vvqx88dpq3bn843p-yargs-5.0.0.tgz’
building path(s) ‘/nix/store/w3lmwwq4cac653nhajnvyl1k2vhivaws-yargs-parser-3.2.0.tgz’
building path(s) ‘/nix/store/ywn6ysmnvrkyzjwpdiwr4iivk9z9kx2y-offline’
building path(s) ‘/nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0’
building path(s) ‘/nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0’
/nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0

    note: the last output line of nix-build is what we are interested in!

  4. checking the result

    ls -lathr result/node_modules/

...
<skipped many lines>>
...
lrwxrwxrwx 1 root root  102  1. Jan 1970  align-text -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/align-text
lrwxrwxrwx 1 root root  105  1. Jan 1970  acorn-globals -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn-globals
lrwxrwxrwx 1 root root   97  1. Jan 1970  acorn -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn
lrwxrwxrwx 1 root root   98  1. Jan 1970  abbrev -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abbrev
lrwxrwxrwx 1 root root   96  1. Jan 1970  abab -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abab

  5. execute the binary

    /nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0/bin/myapp
h1, it works!

using nix-build we declaratively built the node package!

yarn2nix internals

yarn (imperative)

  1. yarn -> read yarn.lock

  2. make / build all the downloads

  3. come up with node_packages folder

  4. yarn install

    note: in general we skip step (4.)

yarn (declarative)

  1. yarn2nix -> read yarn.lock

  2. translate each dependency into a mkDerivation

    each is a mkDerivation residing in /nix/store/…

  3. evaluate each mkDerivation, gain store path

  4. call yarn with the list of all mkDerivation(s), yarn comes up with node_modules

    a single mkDerivation also residing in /nix/store/… but encapsulating other /nix/store entries

  5. final mkDerivation uses this node_modules and creates the store path we are interested in

conclusion

  • take notice, and i can’t stress this enough, we map yarn.lock entries into single pkgs.stdenv.mkDerivation (by calling mkYarnPackage) but later pass all these into yarn which creates the node_modules contents from this. this is exactly how a DSL-PM must be designed to be easily integrated
  • yarn.lock obsoletes the requirement of manually creating a dependency file per project:
    • shown in the example above where i only added a default.nix
    • i’d wish dep’s Gopkg.lock used with go would also contain a hash already but they are using GIT and we have to call nix-prefetch-git to generate a sha256 hash manually
  • yarn2nix might not be as advanced as node2nix but we at nixcloud.io use it for all projects. yarn/yarn2nix is really fast, in dependency calculation and deployment, compared what we had before
  • i’d wish that npm’s sha1 would be replaced by something more recent but in comparison to dep they at least have a hash of some sort
  • yarn2nix is not a part of nixpkgs yet, see https://github.com/moretea/yarn2nix/issues/5
article source