3 aug 2014
i want to give an overview on the status of node.js in nixos. in the nixpkgs documentation [1] we have a great gap regarding the ‘support for specific programming languages’.
currently there are two ways to deploy node applications on nixos:
we will look into both here!
npm is a third party tool, and does not integrate in the nixos world. npm is a classical example for a language level package management (LL-PM) system, while nix-env (one of the nix tools) is a classic example for distribution level package management (DL-PM). so we need to answer this question:
Q: why integrate language level package management (LL-PM) into nixpkgs?
A: you can then:
<pick your language>
and never end up in ‘’dependency hell’’ while, again, having complete
dependencies!so what other advantages does it bring? more interesting:
summary: it makes operating the applications much easier.
a typical node applicaton developer uses npm to handle the dependencies (npm lists 80k node packaged modules on jul 2014). exploringdata.github.io [4] has a excellent visualization of some of these.
even though npm could be used to deploy the whole application it is usually not AFAIK. i didn’t find any numbers about this but:
i want to use etherpad lite 1.4.0 [14] as example. it is deployed on nixos 14.10pre46084.0a750e0 (Caterpillar) using npm in a nix-shell, thus statefully as it would be done on all other linux distributions:
ether-etherpad-lite-c3a6a23/src/packages.json contains the required libraries (documentation [7]), see lines 13 onwards below:
# cat -n ether-etherpad-lite-c3a6a23/src/packages.json
1 {
2 "name" : "ep_etherpad-lite",
3 "description" : "A Etherpad based on node.js",
4 "homepage" : "https://github.com/ether/etherpad-lite",
5 "keywords" : ["etherpad", "realtime", "collaborative", "editor"],
6 "author" : "Peter 'Pita' Martischka <petermartischka@googlemail.com> - Primary Technology Ltd",
7 "contributors" : [
8 { "name": "John McLear" },
9 { "name": "Hans Pinckaers" },
10 { "name": "Robin Buse" },
11 { "name": "Marcel Klehr" }
12 ],
13 "dependencies" : {
14 "yajsml" : "1.1.6",
15 "request" : "2.9.100",
16 "require-kernel" : "1.0.5",
17 "resolve" : "0.2.x",
18 "socket.io" : "0.9.x",
19 "ueberDB" : "0.2.x",
20 "express" : "3.1.0",
21 "async" : "0.1.x",
22 "connect" : "2.7.x",
23 "clean-css" : "0.3.2",
24 "uglify-js" : "1.2.5",
25 "formidable" : "1.0.9",
26 "log4js" : "0.6.6",
27 "nodemailer" : "0.3.x",
28 "jsdom-nocontextifiy" : "0.2.10",
29 "async-stacktrace" : "0.0.2",
30 "npm" : "1.4.x",
31 "ejs" : "0.6.1",
32 "graceful-fs" : "1.1.5",
33 "slide" : "1.1.3",
34 "semver" : "1.0.13",
35 "security" : "1.0.0",
36 "tinycon" : "0.0.1",
37 "underscore" : "1.3.1",
38 "unorm" : "1.0.0",
39 "languages4translatewiki" : "0.1.3",
40 "swagger-node-express" : "1.2.3",
41 "channels" : "0.0.x",
42 "jsonminify" : "0.2.2",
43 "measured" : "0.1.3"
44 },
45 "bin": { "etherpad-lite": "./node/server.js" },
46 "devDependencies": {
47 "wd" : "0.0.31"
48 },
49 "engines" : { "node" : ">=0.6.3",
50 "npm" : ">=1.0"
51 },
52 "repository" : { "type" : "git",
53 "url" : "http://github.com/ether/etherpad-lite.git"
54 },
55 "version" : "1.4.0"
56 }
one could do:
nix-shell -p nodejs
cd ether-etherpad-lite-c3a6a23/src
npm install -d
this would download all the dependencies but the etherpad lite devs wrote their “own installer”, so running:
cd ether-etherpad-lite-c3a6a23
bin/run.sh
would then download the dependencies (as done manually above), install the etherpad lite app using a symlink and finally run:
cd ether-etherpad-lite-c3a6a23
node $SCRIPTPATH/node_modules/ep_etherpad-lite/node/server.js $*
note: it is important that ether-etherpad-lite-c3a6a23 is the directory the interpreter is in before running node with the server.js in order to make the execution work. also the “bin” in package.json isn’t used.
i was wondering what ‘best practice’ the node running ops teams came up with and was surprised [14] how hard it can be to run a node application as a service:
so after we installed etherpad lite node dependencies using ‘npm install -d’, you now should have:
ls src/node_modules/
async connect graceful-fs log4js request
async-stacktrace ejs jsdom-nocontextifiy measured require-kernel
channels express jsonminify nodemailer resolve
clean-css formidable languages4translatewiki npm security
semver tinycon unorm
slide ueberDB wd
socket.io uglify-js yajsml
swagger-node-express underscore
afterwards. as well as:
ls ~/.npm/
active-x-obfuscator cookie-signature fast-list inherits
addressparser core-util-is follow-redirects isarray
async cssom formidable jsdom-nocontextifiy
async-stacktrace debug fresh jsonminify
base64id deprecate generic-pool languages4translatewiki
buffer-crc32 dequeue graceful-fs log4js
bytes dirty hashish mailcomposer
channels dkim-signer he measured
clean-css docco helenus methods
commander ejs helenus-thrift mime
connect encoding htmlparser mimelib
cookie express iconv-lite minimist
mkdirp policyfile semver uglify-js
ms punycode send underscore
mysql q simplesmtp unorm
nan qs slide vargs
nodemailer rai socket.io wd
node-redis range-parser socket.io-client wordwrap
node-uuid readable-stream string_decoder ws
npm redis swagger-node-express xmlhttprequest
optimist request tinycolor xoauth2
options require-kernel tinycon yajsml
pause resolve traverse zeparser
pg security ueberDB
and
ls ~/.node-gyp/
0.10.28
on my system that is:
[nix-shell:~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ du -sh ~/.node-gyp/
20M /home/joachim/.node-gyp/
[nix-shell:~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ du -sh ~/.npm
65M /home/joachim/.npm
and interestingly:
[nix-shell:~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ du -sh node_modules/
284K node_modules/
note: npm basically writes random crap into $HOME/.npm and $HOME/.node-gyp and pollutes the workspace with about 85M!
let’s analyze what ‘npm install -d’ in the etherpad lite deployment did:
‘npm install -d’ queries http://registry.npmjs.org/ (see [16])
~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ rm -Rf ~/.npm ~/.node-gyp/ src/node_modules/
~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ nix-shell -p nodejs -p python
[nix-shell:~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ npm install -d
npm info it worked if it ends with ok
npm info using npm@1.4.9
npm info using node@v0.10.28
npm info preinstall ep_etherpad-lite@1.4.0
npm info trying registry request attempt 1 at 11:45:08
npm http GET https://registry.npmjs.org/require-kernel
npm info trying registry request attempt 1 at 11:45:08
npm http GET https://registry.npmjs.org/ueberDB
npm info trying registry request attempt 1 at 11:45:08
npm http GET https://registry.npmjs.org/log4js
npm info trying registry request attempt 1 at 11:45:08
npm http GET https://registry.npmjs.org/formidable
...
skipped lots of lines
...
/bin/sh: pg_config: command not found
gyp: Call to 'pg_config --libdir' returned exit status 127. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (/nix/store/rbhmm2sk7267bl4c4hzqszcnhnbiawgw-nodejs-0.10.28/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:340:16)
gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:807:12)
gyp ERR! System Linux 3.12.18
gyp ERR! command "node" "/nix/store/rbhmm2sk7267bl4c4hzqszcnhnbiawgw-nodejs-0.10.28/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/joachim/Desktop/etherpad/etherpad/ether-etherpad-lite-c3a6a23/src/node_modules/ueberDB/node_modules/pg
gyp ERR! node -v v0.10.28
gyp ERR! node-gyp -v v0.13.0
gyp ERR! not ok
...
skpped lots of lines
...
express@3.1.0 node_modules/express
├── methods@0.0.1
├── fresh@0.1.0
├── range-parser@0.0.4
├── cookie-signature@0.0.1
├── buffer-crc32@0.1.1
├── cookie@0.0.5
├── commander@0.6.1
├── mkdirp@0.3.3
├── debug@1.0.4 (ms@0.6.2)
├── connect@2.7.2 (pause@0.0.1, bytes@0.1.0, qs@0.5.1, formidable@1.0.11)
└── send@0.1.0 (mime@1.2.6)
nodemailer@0.3.44 node_modules/nodemailer
├── simplesmtp@0.3.32 (rai@0.1.11, xoauth2@0.1.8)
├── optimist@0.6.1 (wordwrap@0.0.2, minimist@0.0.10)
└── mailcomposer@0.2.12 (follow-redirects@0.0.3, mime@1.2.11, he@0.3.6, dkim-signer@0.1.2, mimelib@0.2.16)
socket.io@0.9.17 node_modules/socket.io
├── base64id@0.1.0
├── policyfile@0.0.4
├── redis@0.7.3
└── socket.io-client@0.9.16 (xmlhttprequest@1.4.2, ws@0.4.31, active-x-obfuscator@0.0.1)
npm info ok
npm install -d 10.96s user 2.02s system 20% cpu 1:04.05 total
note: pg_config wasn’t found so compiling ws failed, but ‘npm install -d’ terminated with 0 exit code anway which seems wrong. googling for pg_config points to postgresql, so let’s do a nix-shell with it:
~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ nix-shell -p nodejs -p postgresql -p python
[nix-shell:~/foo/etherpad/ether-etherpad-lite-c3a6a23]$ npm install -d
npm info install ws@0.4.31
> ws@0.4.31 install /home/joachim/Desktop/etherpad/etherpad/ether-etherpad-lite-c3a6a23/src/node_modules/socket.io/node_modules/socket.io-client/node_modules/ws
> (node-gyp rebuild 2> builderror.log) || (exit 0)
make: Entering directory `/home/joachim/Desktop/etherpad/etherpad/ether-etherpad-lite-c3a6a23/src/node_modules/socket.io/node_modules/socket.io-client/node_modules/ws/build'
CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
SOLINK_MODULE(target) Release/obj.target/bufferutil.node
SOLINK_MODULE(target) Release/obj.target/bufferutil.node: Finished
COPY Release/bufferutil.node
CXX(target) Release/obj.target/validation/src/validation.o
SOLINK_MODULE(target) Release/obj.target/validation.node
SOLINK_MODULE(target) Release/obj.target/validation.node: Finished
COPY Release/validation.node
make: Leaving directory `/home/joachim/Desktop/etherpad/etherpad/ether-etherpad-lite-c3a6a23/src/node_modules/socket.io/node_modules/socket.io-client/node_modules/ws/build'
so what programs were compiled?
find . -executable -type f -exec file -i '{}' \; | grep 'x-sharedlib'
./ueberDB/node_modules/pg/build/Release/obj.target/binding.node: application/x-sharedlib; charset=binary
./ueberDB/node_modules/pg/build/Release/binding.node: application/x-sharedlib; charset=binary
./socket.io/node_modules/socket.io-client/node_modules/ws/build/Release/obj.target/bufferutil.node: application/x-sharedlib; charset=binary
./socket.io/node_modules/socket.io-client/node_modules/ws/build/Release/obj.target/validation.node: application/x-sharedlib; charset=binary
./socket.io/node_modules/socket.io-client/node_modules/ws/build/Release/bufferutil.node: application/x-sharedlib; charset=binary
./socket.io/node_modules/socket.io-client/node_modules/ws/build/Release/validation.node: application/x-sharedlib; charset=binary
it turns out that node-gyp compiles at least:
and that node-gyp requires python, thus ‘nix-shell -p nodejs -p python’
summary: node package modules sometimes depend on python, might compile binaries like ueberDB or socket.io and depend on system tools like pg_config (from postgresql). currently nixpkgs does not have a way to reflect such dependencies.
a very nice tool to visualized dependencies interactively is colony [17] (i was using it on NixOS using nix-shell):
node package module can be used in a cyclic way (see [6], package ‘d’), in short:
summary deploying cyclic dependencies:
inspired by nodechecker [9] i digged into npm tests and was disappointed as most node package modules shipped via npmjs.org (read: via npm) are not including unit tests, when downloaded using npm. the only solution was to use npm info express | grep git and afterwards clone the git repo and run the tests from the clone.
interestingly they test about 71k npm(s):
the source code for nodechecker can be found at [11]. it would be awesome if we had that same setup but with hydra and nixos, instead of docker.
so let’s see how the unit test of express works on NixOS using npm in a nix-shell:
nix-shell -p nodejs
npm info express | grep git
repository: { type: 'git', url: 'git://github.com/visionmedia/express' },
homepage: 'https://github.com/visionmedia/express',
[ 'Aaron Heckmann <aaron.heckmann+github@gmail.com>',
bugs: { url: 'https://github.com/visionmedia/express/issues' },
nix-shell -p nodejs -p git cd $(mktemp -d) git clone git://github.com/visionmedia/express cd express npm install -d
finally run the unit test:
[nix-shell:/tmp/tmp.Px73DkAeW4/express]$ npm test
> express@4.4.5 test /tmp/aaaa/express
> mocha --require test/support/env --reporter dot --check-leaks test/ test/acceptance/
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
603 passing (2s)
summary: using nix-shell (stateful
deployment) i was able to execute the express unit test
successfully. the question weather this can be automated and
how to package this on nixos. at least for express there was no further
dependency introduced, like a third party testing framework written in
<pick your language>
, which looks promising.
in essence npm2nix wraps npm by reimplementing its npm deployment concept. however, only for npm’s coming from npmjs.org
currently npm2nix can be used for:
so here is the steps i was (ab)using npm2nix for etherpad anyway:
if you want to use it to deploy etherpad lite, as i did, you have to make some hacks to make it work.
append (or modify) the node-packages.json based on package.json from etherpad lite
generate the node-packages-generated.nix from the adapted package.json
pkgs/top-level/node-packages.json pkgs/top-level/node-packages-generated.nix
copy code from pkgs/development/web/nodejs/build-node-package.nix
# this was copied from pkgs/development/web/nodejs/build-node-package.nix (thanks to Shea Levy)
# if etherpad-lite was in npm one could possibliy use build-node-packge.nix instead of copying it here
mkdir -p .bin
${concatStrings (map (dep: ''
test -d ${dep}/bin && (for b in $(ls ${dep}/bin); do
ln -sv -t .bin ${dep}/bin/$b
done)
'') nodeDeps)}
${concatStrings (concatMap (dep: map (name: ''
ln -sv ${dep}/lib/node_modules/${name} .
'') dep.names) nodeDeps)}
popd
append the deps to the etherpad-lite/default.nix
let nodeDeps = with nodePackages; [
# hack: "socket.io" <- quotes are needed since socket.io would be interpreted otherwise
nodePackages."socket.io"
async
async-stacktrace
channels
clean-css
connect
ejs
# how to depend on express-3.1.0 here?
express
formidable
graceful-fs
jsdom-nocontextifiy
jsonminify
languages4translatewiki
log4js
measured
nodemailer
npm
request
require-kernel
resolve
security
semver
slide
swagger-node-express
tinycon
ueberDB
uglify-js
underscore
unorm
wd
yajsml
];
note: i wonder how to select a nodePackage with a special version with the nix syntax from above (since in node it happens quite frequently that a npm called foo is used in many different versions at the same time).
since i wanted to be able to have more than one etherpad instance i wondered if there is something more advanced than using a ‘reverse proxy’ in combination to one nixos-container per etherpad instance. it turns out that we have that already and it is called types.submodule, see nixos/modules/services/networking/spiped.nix for inspiration.
this new types.submodule let’s you basically use the
services.etherpad.
# exceprt from /etc/nixos/configuration.nix
services.etherpad.foo = {
ip="127.0.0.1";
port=9001;
sessionKey="amks3l7ayr0ywcv9gwr42";
mysqlPassword="";
};
services.etherpad.pad2 = {
enable = true;
ip="127.0.0.1";
port=9002;
sessionKey="ks3l7ayr0ywcv9gwr42am";
mysqlPassword="";
};
and it also guaranties you that the used ports inside this record are unique:
port = mkOption {
default = 9001;
type = types.uniq types.int;
description = "Port used by node instance.";
};
finally each configuration gets its own individual configuration file, like /etc/etherpad/settings-foo.json, using:
# Setup etherpad config files (minimal config, copied from spiped)
environment.etc = mapAttrs' (name: cfg: nameValuePair "etherpad/settings-${name}.json"
{ text = ''
/*
This file must be valid JSON. But comments are allowed
{
...
note: we should apply this pattern to all system services because with reverse proxies or port-forwardings this opens new ways to compose system services side by side without having to use nixos-containers.
summary: see [11] for the etherpad lite nix expressions. i’ve packaged it as a system service as well but since the expression is still kind of broken i don’t want to upload it to nixpkgs just yet.
a discussion about npm2nix, node.js and npm issues which still need to be addressed by the nixos cummunity.
shlevy designed npm2nix to wrap npm using nix (solely for node packaged modules from npmjs.org (read dependency management within the domain of npm)) but not for dependency management of applications like etherpad lite for instance.
call for action:
using shrinkwrap means that we have to update the
npm-shrinkwrap.json
individually for each node application,
from time to time, but it is the only option to get reproducibility as
there is no way to set the node packaged module (npmjs.org, using npm)
database to a certain state in time. in contrast to what we do with
c/c++ applications in nixpkgs which do depend on librarys, which are
also defined in nixpkgs.
IMHO we should:
npm shrinkwrap
generated
npm-shrinkwrap.json
into nixpkgs, along
with the etherpad-lite/default.nix
top-level/node-shrinkwraps.nix
where all the
npm-shirinkwrap.json
files are aggregated (similar to
top-level/all-packages.nix
). generate
node-packages-generated.nix
from
top-level/node-shrinkwraps.nix
and commit it to
nixpkgstop-level/node-packages.json
is not needed anymore,
therefore remove itsummary: this is a much cleaner approach, compared
to editing the node-packages.json
as we used to, and by
using shrinkwrap we also get reproducibility!
node packaged module’s usually don’t use software from the host system. exceptions to the rule are ueberDB/socket.io (what was shown above):
yet, if an npm wants to use a system software, we don’t have any way of describing such a dependency per wrapped npm.
note: for node applications like etherpad lite, which use npm to manage libraries, this is a completely different story as we can depend on certain pkgs easily:
pkgs/webapp/etherpad-lite/default.nix View
{ stdenv, fetchurl, unzip, nodejs, bash, curl, postgresql, python, utillinux}:
summary: in contrast, for python libraries, we created a nix expression for each package and can add such pkgs dependencies individually. in the future we might have to do that for npm’s as well.
i would like to avoid nix-shell usage for developing node.js apps but i don’t have a clue just yet how to do this. here is a nice example how it is done for haskell: see [18].
the good news is, that you can do development of node.js apps on nixos already using nix-shell (this is the ‘devs’ part). nixos does not yet give you an environment as good other linux distributions (or hosting services, like heroku) and this needs fixing (the ‘ops’ part).