[[!meta date="2018-02-21 12:35"]] [[!tag nixos javascript yarn npm node2nix npm2nix yarn2nix]] [[!summary how yarn changed javascript deployment using nix]] [[!series nix-language-atlas]] # motivation in the nix-language-atlas series on lastlog.de/blog i want to discuss how well programming languages, for which i'm familiar with, integrate with nix. let's revisit javascript, as there had been major improvements since [i wrote about it last time](https://lastlog.de/blog/posts/nodejs_on_nixos_status.html). we won't look into the [emscripten toolchain](https://lastlog.de/blog/posts/xmlmirror.html#emscripten). # DSL-PM in general are 'domain specific language package manager(s)' (DSL-PM) as npm or yarn a good thing? * pro(s): * npm/yarn is a [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) * npm/yarn works on linux/mac/windows (in contrast nix 1.x only works with linux/mac due to the shell/unix requirements but [nix 2.x on windows 10 tells another story](https://www.reddit.com/r/NixOS/comments/64xyd7/nix_package_manager_works_flawlessly_in_windows/)) * DSL-PM often use semantic versioning [npm](https://www.npmjs.com/), [golang dep](https://github.com/golang/dep), [rust cargo](https://crates.io/) while [nix intentionally has no semantic versioning](https://matrix.ai/2016/04/04/content-addressed-dependencies-vs-semantic-versioning/). to be fair, go isn't the best example for semantic versioning and rust does pretty well without it as [nixcrates](https://github.com/nixclohttps://github.com/nixcloud/nixcratesud/nixcrates) proved. * con(s): * DSL-PMs like npm often lack support for [inter domain dependency management](https://github.com/svanderburg/node2nix#adding-unspecified-dependencies), interestingly something which was better with autotools based build system where one was forced to pass arguments to not only header and libraries but also to binaries. this is where node2nix shines compared to yarn2nix * bundling/vendoring of dependencies: outside DSL-PM as bundling openssl c-code but often also inside of the DSL-PM most often seen with [golang](https://golang.org/cmd/go/#hdr-Vendor_Directories) * deployment not reproducible: [npm 5.x that supports lock files pinpointing the exact versions used of all dependencies and transitive dependencies, and a content addressable cache.](https://github.com/svanderburg/node2nix#generating-packages-for-nodejs-8x) ## integrate DSL-PM(s) into nix/nixpkgs DSL-PM properties nix requires: * reproducible dependency calculations/downloads * npm: only since [npm 5.x](https://github.com/svanderburg/node2nix#generating-packages-for-nodejs-8x), see also [package-lock.json](https://medium.com/@Quigley_Ja/everything-you-wanted-to-know-about-package-lock-json-b81911aa8ab8) * yarn: [yarn.lock](https://yarnpkg.com/lang/en/docs/yarn-lock/) made it easy, before that it required much effort * reproducible configuration, build & installation into the store of each dependency * reproducible configuration, build & installation into the store of main target # DSL-PM evolution (javascript) * npm -> [npm2nix](https://github.com/NixOS/npm2nix) -> [node2nix](https://github.com/svanderburg/node2nix) * yarn -> [yarn2nix](https://github.com/moretea/yarn2nix) * bower -> [bower2nix](https://github.com/rvl/bower2nix) note: [read this if you are interested in npm/yarn differences](https://www.alexkras.com/understanding-differences-between-npm-yarn-and-pnpm/). it seems that yarn forced many of its concepts into npm. # yarn2nix workflow let's see a simple example how to integrate a `yarn` based project into your `nix` codebase: 1. we clone an example: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ git clone https://github.com/yarnpkg/example-yarn-package cd example-yarn-package mkdir bin touch bin/myapp chmod u+x bin/myapp ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2. let's add a default.nix ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ { pkgs ? import {} }: let yarn2nixSrc = pkgs.fetchFromGitHub { owner = "moretea"; repo = "yarn2nix"; rev = "0472167f2fa329ee4673cedec79a659d23b02e06"; sha256 = "10gmyrz07y4baq1a1rkip6h2k4fy9is6sjv487fndbc931lwmdaf"; }; yarn2nixRepo = pkgs.callPackage yarn2nixSrc {}; inherit (yarn2nixRepo) mkYarnPackage; in mkYarnPackage { src = ./.; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ and add contents to `bin/myapp` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/usr/bin/env node 'use strict'; // For package depenency demonstration purposes only var multiply = require('lodash/multiply'); console.log("h" + multiply(2,0.5) + ", it works!") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ add a "bin" to `package.json`, see this patch: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ {.Diff} diff --git a/package.json b/package.json index 36952fd..a38ceaf 100644 --- a/package.json +++ b/package.json @@ -18,6 +18,7 @@ "devDependencies": { "jest-cli": "15.1.1" }, + "bin" : { "myapp" : "bin/myapp" }, "jest": { "testEnvironment": "node" } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3. now build the source ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ nix-build -Q building path(s) ‘/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline’ building path(s) ‘/nix/store/b6kiqlcaacxrlq4nnjgk69mwjnrlvkpv-yarn2nix-modules-0.1.0’ building yarn config v1.3.2 success Set "yarn-offline-mirror" to "/nix/store/gkdb8hsykw3idp4xpv6by618wad5s052-offline". Done in 0.10s. yarn install v1.3.2 [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... warning Ignored scripts due to flag. Done in 4.99s. ... > ... building path(s) ‘/nix/store/7yg1mah963w35dhxjcylw5q0r62bkk83-yarn.nix’ these derivations will be built: /nix/store/07xy1c1yp4ada1sjiz7m416ic2i1b908-webidl-conversions-3.0.1.tgz.drv /nix/store/0c2xgds6pzsnsp1hmxdgaqh1n28d68k5-lodash.isarray-3.0.4.tgz.drv /nix/store/0dj6mwhpmzvnxzz2122isdbw85mjg0wv-regex-cache-0.4.3.tgz.drv /nix/store/0in2pcwsc3ln6glqkab5fsx98m28vq8g-lodash._baseassign-3.2.0.tgz.drv /nix/store/1rhagnq3gz68x5lbhca6z4rsx9jnabgh-is-dotfile-1.0.2.tgz.drv /nix/store/1vq9rlkn3qdy2acndj2q47ryswiv5yir-lodash.assign-4.2.0.tgz.drv /nix/store/2a4kqk4p54nvm31cyy2nnl2vzr4klyb8-babel-generator-6.14.0.tgz.drv /nix/store/2mf2dr0gx3bqxiqbpqdhhp8bhz95d34h-arr-diff-2.0.0.tgz.drv /nix/store/2pbiyjyqg57dnn0apj24lrvqp1cbix9f-expand-brackets-0.1.5.tgz.drv /nix/store/31h1f7rair2xq0di3b3agr7g2xfgncyz-jest-matcher-utils-15.1.0.tgz.drv /nix/store/3l6amqm7wdx1c45mypbgh8xgsl92zw76-exec-sh-0.2.0.tgz.drv /nix/store/3w6wxplb811vzfky5sgvbifm6j5p6fac-babel-core-6.14.0.tgz.drv /nix/store/48d8jwrpgg2pxm8mkyqvavrv4k5i2nab-lodash.keys-3.1.2.tgz.drv /nix/store/4l0s8plnxw2nzszbjjdicrr5np9cc2ni-jest-resolve-15.0.1.tgz.drv /nix/store/4mn98xigyzybcnsb71731pw8biirzzax-micromatch-2.3.11.tgz.drv ... > ... building path(s) ‘/nix/store/chsx89ifr7dk7mc00d6789indh0rn41x-to-fast-properties-1.0.2.tgz’ building path(s) ‘/nix/store/sqpsd9y77kkq1vvfpfcfzk9q347dwg14-walker-1.0.7.tgz’ building path(s) ‘/nix/store/j7fp6iz0wj6dcp8ibsqyph2vy4p007wk-watch-0.10.0.tgz’ building path(s) ‘/nix/store/nv5p8ijp15mc2kbpp26mvd8pn7vqlwyy-webidl-conversions-3.0.1.tgz’ building path(s) ‘/nix/store/a3amdrmhzrvd3v1lsy1ih5rfpdamyj46-whatwg-url-3.0.0.tgz’ building path(s) ‘/nix/store/vhwym48w1pvgzli9i57fnzqw0pjp3a83-window-size-0.2.0.tgz’ building path(s) ‘/nix/store/2dg59ikzfrdwbcpkwg2xwjxs9rhpilb8-worker-farm-1.3.1.tgz’ building path(s) ‘/nix/store/497p7kg9iyb09ah0vvqx88dpq3bn843p-yargs-5.0.0.tgz’ building path(s) ‘/nix/store/w3lmwwq4cac653nhajnvyl1k2vhivaws-yargs-parser-3.2.0.tgz’ building path(s) ‘/nix/store/ywn6ysmnvrkyzjwpdiwr4iivk9z9kx2y-offline’ building path(s) ‘/nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0’ building path(s) ‘/nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0’ /nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ note: the last output line of nix-build is what we are interested in! 4. checking the result ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ls -lathr result/node_modules/ ... > ... lrwxrwxrwx 1 root root 102 1. Jan 1970 align-text -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/align-text lrwxrwxrwx 1 root root 105 1. Jan 1970 acorn-globals -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn-globals lrwxrwxrwx 1 root root 97 1. Jan 1970 acorn -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/acorn lrwxrwxrwx 1 root root 98 1. Jan 1970 abbrev -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abbrev lrwxrwxrwx 1 root root 96 1. Jan 1970 abab -> /nix/store/pnbb86igrhhfw3bdv9kcqnyz4hxgfkmd-example-yarn-package-modules-1.0.0/node_modules/abab ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5. execute the binary ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /nix/store/r9s6cmq5ik47c0s1hd48xz3r6pixmm1r-example-yarn-package-1.0.0/bin/myapp h1, it works! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ using `nix-build` we declaratively built the node package! # yarn2nix internals
yarn (imperative)
1. `yarn` -> read `yarn.lock` 2. make / build all the downloads 3. come up with `node_packages` folder 4. `yarn install` note: in general we skip step (4.)
yarn (declarative)
1. `yarn2nix` -> read `yarn.lock` 2. translate each dependency into a `mkDerivation` each is a `mkDerivation` residing in /nix/store/... [[!img media/nix-language-atlas-javascript-mkderivation1.png size=30x class="noFancy" ]] 3. evaluate each `mkDerivation`, gain store path 4. call `yarn` with the list of all `mkDerivation(s)`, `yarn` comes up with `node_modules` a single `mkDerivation` also residing in /nix/store/... but encapsulating other /nix/store entries [[!img media/nix-language-atlas-javascript-mkderivation2.png size=30x class="noFancy" ]] 5. final `mkDerivation` uses this `node_modules` and creates the store path we are interested in
# conclusion * take notice, and i can't stress this enough, we map `yarn.lock` entries into single `pkgs.stdenv.mkDerivation` (by calling `mkYarnPackage`) but later pass all these into `yarn` which creates the `node_modules` contents from this. this is exactly how a DSL-PM must be designed to be easily integrated * `yarn.lock` obsoletes the requirement of manually creating a dependency file per project: * shown in the example above where i only added a `default.nix` * i'd wish [dep's](https://github.com/golang/dep) `Gopkg.lock` used with `go` would also contain a hash already but they are using GIT and we have to call `nix-prefetch-git` to generate a sha256 hash manually * `yarn2nix` might not be as advanced as `node2nix` but we at nixcloud.io use it for all projects. `yarn`/`yarn2nix` is really fast, in dependency calculation and deployment, compared what we had before * i'd wish that npm's sha1 would be replaced by something more recent but in comparison to `dep` they at least have a hash of some sort * `yarn2nix` is not a part of `nixpkgs` yet, see https://github.com/moretea/yarn2nix/issues/5