prev. article
nixcloud

nixcloud-webservices-tests

28 Nov 2017

motivation

many programming languages have testing frameworks and for some they are actually used. in nixpkgs i see them lots, most often in libraries written in perl, python and go.

this posting is about how we adapted this concept into nix. AFAIK this hasn't been done before so it might be worth sharing.

how testing works in perl/python/go

the tests are executed during the build of the software and most often implemented in libraries.

sometimes there is doCheck = false; as the tests fail for some reason. often these tests are impure and the build environment can't perform the actions, as for instance, visiting some remote website during build time which is not possible in a nix-build phase.

testing systems i'm refering to:

python example

av = buildPythonPackage rec {
  name = "av-${version}";
  version = "0.2.4";

  src = pkgs.fetchurl {
    url = "mirror://pypi/a/av/${name}.tar.gz";
    sha256 = "bdc7e2e213cb9041d9c5c0497e6f8c47e84f89f1f2673a46d891cca0fb0d19a0";
  };

  buildInputs
    =  (with self; [ nose pillow numpy ])
    ++ (with pkgs; [ ffmpeg_2 git libav pkgconfig ]);

  # Because of https://github.com/mikeboers/PyAV/issues/152
  doCheck = false;

  meta = {
    description = "Pythonic bindings for FFmpeg/Libav";
    homepage = https://github.com/mikeboers/PyAV/;
    license = licenses.bsd2;
  };
};

example taken from https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/python-packages.nix.

perl example

ack = buildPerlPackage rec {
  name = "ack-2.16";
  src = fetchurl {
    url = "mirror://cpan/authors/id/P/PE/PETDANCE/${name}.tar.gz";
    sha256 = "0ifbmbfvagfi76i7vjpggs2hrbqqisd14f5zizan6cbdn8dl5z2g";
  };
  outputs = ["out" "man"];
  # use gnused so that the preCheck command passes
  buildInputs = stdenv.lib.optional stdenv.isDarwin gnused;
  propagatedBuildInputs = [ FileNext ];
  meta = with stdenv.lib; {
    description = "A grep-like tool tailored to working with large trees of source code";
    homepage    = http://betterthangrep.com/;
    license     = licenses.artistic2;
    maintainers = with maintainers; [ lovek323 ];
    platforms   = platforms.unix;
  };
  # tests fails on nixos and hydra because of different purity issues
  doCheck = false;
};

example taken from https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/perl-packages.nix

go example

terraform_0_9 = generic {
  version = "0.9.11";
  sha256 = "045zcpd4g9c52ynhgh3213p422ahds63mzhmd2iwcmj88g8i1w6x";
  # checks are failing again
  doCheck = false;
};

example taken from https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/cluster/terraform/default.nix.

nixos testing

first, let's have a look at how testing in nixos is done traditionally. these tests use KVM, spawn one or serveral virtual machines which interact with each other over the network. they are called explicitly: nix-build -A leaps release.nix.

the leaps test:

import ./make-test.nix ({ pkgs,  ... }:

{
  name = "leaps";
  meta = with pkgs.stdenv.lib.maintainers; {
    maintainers = [ qknight ];
  };

  nodes =
    { 
      client = { };

      server =
        { services.leaps = {
            enable = true;
            port = 6666;
            path = "/leaps/";
          };
          networking.firewall.enable = false;
        };
    };

  testScript =
    ''
      startAll;
      $server->waitForOpenPort(6666);
      $client->waitForUnit("network.target");
      $client->succeed("${pkgs.curl}/bin/curl http://server:6666/leaps/ | grep -i 'leaps'");
    '';
})

nixos tests are described in the manual https://nixos.org/nixos/manual/index.html#sec-nixos-tests nicely. the source of the tests are at https://github.com/NixOS/nixpkgs/blob/master/nixos/release.nix#L217.

nixcloud testing

at nixcloud we also implement tests as we have them in nixos (described above). some tests can be found here: https://github.com/nixcloud/nixcloud-webservices/tree/master/tests but, and that is why this posting was written, we also have different kind of tests, which are similar to those in perl, python and go explained previously.

test usage

we basically extended the nixos module system:

{ config, pkgs, lib, ... } @ args:

{
  options = {
    nixcloud.reverse-proxy = {
      enable = mkEnableOption "reverse-proxy";
      ...
    }
  };
  config = { 
    ... 
    nixcloud.tests.wanted = [ ./test.nix ];
  };
}

see complete usage implementation

note: the nixcloud.reverse-proxy module is similar to nixos modules as services.openssh

if the nixcloud.reverse-proxy module is used and one does a nixos-rebuild switch, it evaluates the ./test.nix during build.

implementation

and the implementation of the test is here https://github.com/nixcloud/nixcloud-webservices/blob/12e51b6bd073acdd78807d0dbb23458267538b48/modules/core/testing.nix

advances

drawbacks

summary

i would love to see this feature coming to nixos/nixpkgs also! oh and thanks to aszlig!

questions?