emscripten-1.36.4 on nixos

25 may 2016

motivation

emscripten on nixos, a status update.

1.36.4

since the project i’m hacking on requires me to develop on nixos, which is awsome btw, i had to figure how to compile xml.js on nixos. one requirement was the revision bump from 1.36.4 to 1.36.4.

compiling xml.js

since xml.js wouldn’t compile on nixos caused by a missing library: cannot find -lgcc. see the bug-report for details.

in a nut-shell: using 1.36.4 emscripten xml.js compiles/works with ubuntu but fails on nixos.

./script/libxml2   

configure.ac:52: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated.
aclocal.m4:9742: AM_INIT_AUTOMAKE is expanded from...
configure.ac:52: the top level
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:52: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated.
aclocal.m4:713: AM_INIT_AUTOMAKE is expanded from...
configure.ac:52: the top level
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/\${ <-- HERE ([^ \t=:+{}]+)}/ at /nix/store/klmb2g54ini7y5hcf5sfx92agsifgazi-automake-1.15/bin/automake line 3936.
configure.ac:52: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated.  For more info, see:
configure.ac:52: http://www.gnu.org/software/automake/manual/automake.html#Modernize-AM_005fINIT_005fAUTOMAKE-invocation
configure.ac:60: installing './compile'
configure.ac:52: installing './missing'
/nix/store/klmb2g54ini7y5hcf5sfx92agsifgazi-automake-1.15/share/automake-1.15/am/ltlibrary.am: warning: 'libxml2.la': linking libtool libraries using a non-POSIX
/nix/store/klmb2g54ini7y5hcf5sfx92agsifgazi-automake-1.15/share/automake-1.15/am/ltlibrary.am: archiver requires 'AM_PROG_AR' in 'configure.ac'
Makefile.am:22:   while processing Libtool library 'libxml2.la'
/nix/store/klmb2g54ini7y5hcf5sfx92agsifgazi-automake-1.15/share/automake-1.15/am/ltlibrary.am: warning: 'testdso.la': linking libtool libraries using a non-POSIX
/nix/store/klmb2g54ini7y5hcf5sfx92agsifgazi-automake-1.15/share/automake-1.15/am/ltlibrary.am: archiver requires 'AM_PROG_AR' in 'configure.ac'
Makefile.am:173:   while processing Libtool library 'testdso.la'
Makefile.am: installing './depcomp'
doc/Makefile.am:21: warning: wildcard tutorial/*.html: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard tutorial/*.c: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard tutorial/*.pdf: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard tutorial/images/*.png: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard tutorial/images/callouts/*.png: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard API*.html: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard *.1: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard *.xsl: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard *.html: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard *.gif: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard html/*.html: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:21: warning: wildcard html/*.png: non-POSIX variable name
doc/Makefile.am:21: (probably a GNU make extension)
doc/Makefile.am:301: warning: filter-out %/xmlversion.h, $(wildcard $(top_srcdir: non-POSIX variable name
doc/Makefile.am:301: (probably a GNU make extension)
doc/Makefile.am:301: warning: wildcard $(top_srcdir: non-POSIX variable name
doc/Makefile.am:301: (probably a GNU make extension)
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /nix/store/n4zpfi4zzw10s2g91v1swkwq41v1afzz-coreutils-8.25/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/n4zpfi4zzw10s2g91v1swkwq41v1afzz-coreutils-8.25/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for gcc... /nix/store/0llznkhcyyndfhwqah01i1zcdhqxrx0d-emscripten-1.36.4/share/emscripten/emcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... configure: error: in `/home/joachim/Desktop/projects/nlnet/xml.js/build':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
ERROR:root:Configure step failed with non-zero return code 1! Command line: ['../libxml2/configure', '--with-http=no', '--with-ftp=no', '--with-python=no', '--with-threads=no', '--with-debug'] at /home/joachim/Desktop/projects/nlnet/xml.js/build
make: *** Es wurden keine Ziele angegeben und keine „make“-Steuerdatei gefunden.  Schluss.
% cat build/config.log

...
configure:3602: checking whether the C compiler works
configure:3624: /nix/store/0llznkhcyyndfhwqah01i1zcdhqxrx0d-emscripten-1.36.4/share/emscripten/emcc    conftest.c  >&5
configure:3628: $? = 0
configure:3676: result: yes
configure:3679: checking for C compiler default output file name
configure:3681: result: a.out
configure:3687: checking for suffix of executables
configure:3694: /nix/store/0llznkhcyyndfhwqah01i1zcdhqxrx0d-emscripten-1.36.4/share/emscripten/emcc -o conftest    conftest.c  >&5
configure:3698: $? = 0
configure:3720: result:
configure:3742: checking whether we are cross compiling
configure:3750: /nix/store/0llznkhcyyndfhwqah01i1zcdhqxrx0d-emscripten-1.36.4/share/emscripten/emcc -o conftest    conftest.c  >&5
/nix/store/60a1aqx579bjff3c90cw0l1zwyj2qxhg-binutils-2.26-dev/bin/ld: cannot find crt1.o: No such file or directory
/nix/store/60a1aqx579bjff3c90cw0l1zwyj2qxhg-binutils-2.26-dev/bin/ld: cannot find crti.o: No such file or directory
/nix/store/60a1aqx579bjff3c90cw0l1zwyj2qxhg-binutils-2.26-dev/bin/ld: cannot find crtbegin.o: No such file or directory
/nix/store/60a1aqx579bjff3c90cw0l1zwyj2qxhg-binutils-2.26-dev/bin/ld: cannot find -lgcc
clang: error: linker command failed with exit code 1 (use -v to see invocation)
configure:3754: $? = 1
configure:3761: ./conftest
../libxml2/configure: line 3763: ./conftest: No such file or directory
configure:3765: $? = 127
configure:3772: error: in `/home/joachim/Desktop/projects/nlnet/xml.js/build':
configure:3774: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

the interesting bit is that the crt1.o, crti.o and crtbegin.o as well as libgcc.so / libgcc_s.so are coming from different packages like gcc and glibc IIRC.

attempted fix

i’ve spent a while trying to fix these requirements with ./gen.sh a little scripte which creates fake libraries.


% cat /home/joachim/Desktop/projects/nlnet/xml.js/lib/gen.sh

#!/usr/bin/env bash 
set -v

rm *.o
rm *.so
echo "crtbegin"
echo "int main() {}" > foo.cpp
g++ -c -Wall -Werror -lc --entry main foo.cpp -o foo.o
mv foo.o crtbegin.o

##### crt1 with _start symbol
echo "crt1"
echo "void _start() {}" > foo.c
gcc -c -Wall -Werror -fpic foo.c -o crt1.o

for i in crtbegin crti crtend crtn; do
  rm -f foo.o foo.c
  echo "void foo123_$i(void) {}" > foo.c
  gcc -c -Wall -Werror -fpic foo.c -o foo.o
  mv foo.o $i.o
done

for i in libgcc_s1  libgcc1 ; do
  rm -f foo.o foo.c
  echo "void foobar_$i(void) {}" > foo.c
  cat foo.c
  echo "1"
  gcc -c -Wall -Werror  foo.c -o foo.o
  echo "2"
  gcc -shared  -o $i.so foo.o
done

ar rc libgcc1.a foo.o

mv libgcc1.a libgcc.a
mv libgcc_s1.so libgcc_s.so

so i came up with extending environment variables: NIX_LDFLAGS and

export NIX_LDFLAGS="-L /home/joachim/Desktop/projects/nlnet/xml.js/lib -rpath /nix/store/3qhj8rf3j064qbrm5q2yzkb5vd5qnhzj-shell/lib64 -rpath /nix/store/3qhj8rf3j064qbrm5q2yzkb5vd5qnhzj-shell/lib  -L/nix/store/97aj51dl9jvn2qyjapi7m2v2gai9mp27-nodejs-4.3.1/lib -L/nix/store/97aj51dl9jvn2qyjapi7m2v2gai9mp27-nodejs-4.3.1/lib -L/nix/store/g47gdl1h8mzixl9566dnjlwgwvgpyvkr-libtool-2.4.6-lib/lib -L/nix/store/g47gdl1h8mzixl9566dnjlwgwvgpyvkr-libtool-2.4.6-lib/lib -L/home/joachim/Desktop/projects/nlnet/xml.js/lib"

strace -f -s 200 -e execve emconfigure ../libxml2/configure --with-http=no --with-ftp=no --with-python=no --with-threads=no --with-debug

would show something like:


[pid 24762] execve("/nix/store/60a1aqx579bjff3c90cw0l1zwyj2qxhg-binutils-2.26-dev/bin/ld", ["/nix/store/60a1aqx579bjff3c90cw0l1zwyj2qxhg-binutils-2.26-dev/bin/ld", "-dynamic-linker", "/nix/store/iifa17bpi78d0l62463z7sn25jc3545j-glibc-multi-2.23/lib/ld-linux-x86-64.so.2", "--eh-frame-hdr", "-m", "elf_x86_64", "-dynamic-linker", "/lib64/ld-linux-x86-64.so.2", "-o", "conftest", "crt1.o", "crti.o", "crtbegin.o", "/run/user/1000/conftest-904287.o", "-L/home/joachim/Desktop/projects/nlnet/xml.js/build/../lib", "-lgcc", "--as-needed", "-lgcc_s", "--no-as-needed", "-lc", "-lgcc", "--as-needed", "-lgcc_s", "--no-as-needed", "crtend.o", "crtn.o", "-L", "/home/joachim/Desktop/projects/nlnet/xml.js/lib", "-rpath", "/nix/store/3qhj8rf3j064qbrm5q2yzkb5vd5qnhzj-shell/lib64", "-rpath", "/nix/store/3qhj8rf3j064qbrm5q2yzkb5vd5qnhzj-shell/lib", "-L/nix/store/97aj51dl9jvn2qyjapi7m2v2gai9mp27-nodejs-4.3.1/lib", "-L/nix/store/97aj51dl9jvn2qyjapi7m2v2gai9mp27-nodejs-4.3.1/lib", "-L/nix/store/g47gdl1h8mzixl9566dnjlwgwvgpyvkr-libtool-2.4.6-lib/lib", "-L/nix/store/g47gdl1h8mzixl9566dnjlwgwvgpyvkr-libtool-2.4.6-lib/lib", "-L/home/joachim/Desktop/projects/nlnet/xml.js/lib", "-L/nix/store/iifa17bpi78d0l62463z7sn25jc3545j-glibc-multi-2.23/lib", "-L/nix/store/hcndjf2r2rcrf39x77wklckn60sx89dx-gcc-5.3.0-lib/lib", "-rpath", "/nix/store/iifa17bpi78d0l62463z7sn25jc3545j-glibc-multi-2.23/lib", "-rpath", "/nix/store/hcndjf2r2rcrf39x77wklckn60sx89dx-gcc-5.3.0-lib/lib"], [/* 125 vars */]) = 0

the return value of 0 is the key but it didn’t work anyways….

the fix

the emscripten source code mentiones this fix:

export EMCONFIGURE_JS=2 is the key in this case.

# Whether we fake configure tests using clang - the local, native compiler - or not. if not we generate JS and use node with a shebang

Beither approach is perfect, you can try both, but may need to edit configure scripts in some cases
By default we configure in js, which can break on local filesystem access, etc., but is otherwise accurate so we
disable this if we think we have to. A value of '2' here will force JS checks in all cases. In summary:
0 - use native compilation for configure checks
1 - use js when we think it will work
2 - always use js for configure checks
use_js = int(os.environ.get('EMCONFIGURE_JS') or 1)

so every time you fail with ./configure for some weird reason like the one above or other, try export EMCONFIGURE_JS=2, distclean the build and build again.

hf!

summary

with this new knowledge it might get feasible to use nix as a tool chain for more complex emscripten setups!

thanks to matthewbauer and joelmo#nixos@irc.kde.org for their support and ideas!

article source