prev. article next article
emscripten

xmlmirror

21 Jul 2016

motivation

we are happy to announce the initial release of a useful new tool called xmlmirror. as the name more or less spells out, xmlmirror is an XML webeditor with schema validation, based on webforms and implemented with codemirror. xmlmirror further uses a library called Fast-XML-Lint which uses libxml2 for schema verification and which is compiled with emscripten. or in layman’s terms: a web application that really helps you to create complex XML documents from scratch, as well as fix existing documents that are broken.

live demo / source code

features

more details

selenium

unit testing was implemented using selenium 2.53:

nix-shell -p python35Packages.selenium firefox-bin --command "python3 selenium_test.py"

it works like this:

  1. opens a specially crafted html document: schemainfoCreator-test.html in a webbrowser
  2. executes it and looks for “OK” or a 10 seconds timeout

selenium_test.py:

closure-compiler

the google closure compiler was used to ensure strict typing even though it is javascript:

closure_compiler/jcc schemainfoCreator.js

hint: it was no joy to use this tooling due to a lack of documentation and examples.

fastXmlLint.c

it took a bit of time to get into the internals of libxml2 and the antique API documentation is more confusing than helpful. anyway, two interesting results:

  1. even though xmllint can parse xml documents with a multi-document relax-ng schema it can’t be used to parse a multi-document relax-ng schema itself, see discussion

  2. ltrace is your best friend in reverse-engineering shared object/library usage:

    for instance, running:

    ltrace -f xmllint --relaxng html5-rng/xhtml.rng test_fail1.html

    would yield:

    and this is the exact order of libxml2 function calls xmllint issues to parse test_fail1.html!

    note: this helped us a lot and made it possible to discover the secret xmlLineNumbersDefault function!

emscripten

during this project we had the idea to create a c to javascript cross-compiler abstraction using nix for emscripten and we are happy to announce that it is now officially in nixpkgs, see PR 16208.

this means:

  1. you can use nix-build to cross-compile all your dependencies like libz and afterwards use these in your project
  2. since nix runs on all linuxes, mac os x and other unix-like platforms, you can now enojoy full toolchain automation and deployment when doing emscripten.

if using nixpkgs (master), you can check for emscripten targets using:

nix-env -qaP | grep emscriptenPackages

and install using:

nix-env -iA emscriptenPackages.json_c

note: don’t mix json_c (native, x86) with other libs (emscripten, javascript) in your user-profile or you will get weird error messages with object code being in the wrong format and such.

nixos development

nix-shell was the primary development tool along with the default.nix which can basically spawn two different environments:

see Makefile.emEnv and Makefile.nativeEnv respectivly.

let’s have a look at the default.nix:

you will notice that emEnv is a stdenv.mkDerivation and it uses shellHook and buildInputs.

some remarks:

note: the default.nix does contain libz, json-c, xml-js packaging and since this is now in nixpkgs it is kind of obsolete now.

conclusion

we (paul/joachim) want to thank Jos van den Oever (prolific open source contributor and co-chair of the OASIS ODF TC on behalf of the Dutch government) for inspiring the creation of this tool. ODF is a prominent example of a real-world standard that leverages the relax ng standard, and we expect xmlmirror to be very useful in the creation of more ODF autotests. Jos has also graciously offered to provide an initial host repository for xmlmirror.

if you have questions/comments, see nixcloud.io for contact details.