docker compose vs nixcloud

6 aug 2023

motivation

this is my summary of years working with nix/nixpkgs/nixcloud-webservices and docker.

i compare docker compose to nixcloud.webservices. we’ll also look briefly at kubernetes and nixcloud.containers which is LXC based.

in this context, a complex service would be a LAMP stack, an email service, like nixcloud.email or any ‘service’ which would require several docker containers, started simultaneously, to work.

docker / docker compose

image from docker.com

a list of pros/cons to docker and docker compose:

pro:

  • cross platform development support (linux, mac, windows)
  • docker desktop is so amazing during development
    • clear overview over containers/volumes/images
    • grouping view for docker compose deployments
    • useful for maintenance
      • very easy to get a prompt in a running container
      • good tool to investigate logs
  • yaml based, easy to learn
  • docker FS layers (fabulous workaround for PM shortcomings in reproducibility)
  • best practice assumes container to be stateless: state management in volumes
  • huge community with lots of well designed containers on https://hub.docker.com
  • traefik reverse-proxy is easy to use (with let’s encrypt ACME integration)
  • scaling potential of hosting using docker swarm or even in kubernetes with little adaptions

con:

  • security running containers as root is bad, yet many do exactly that

  • no management node concept (controller in kubernetes)

    it would be helpful if the deployment were controlled from a docker container itself - a controller node. the controller keeps the central configuration and offers this to the other nodes.

    the .env file, the docker-compose.yaml and deployment specific files would reside on the controller and managed by it:

    instead the deployment is controlled from the host, making deployment management platform dependent:

    1. docker compose down
    2. docker compose run --rm setup --domain=example.com --email=admin@example.com
    3. docker compose up -d

  • stateful management tool

    docker compose up alone is not enough, users have to pass explicit command line arguments:

    • docker/docker compose commands get ridiculously long
    • management scripts around docker compose result in platform specific code
    • problematic global state variables with .env
    • hard to build deployment variants with consistency (http/https+let’s encrypt or a different set of components)
    • inspecting running containers give no clue from where they were configured, i.e. the docker-compose.yml file on the filesystem

  • incomplete APIs

    • docker go API: lacks support for some very basic scenarios
      • access logs after a non-persistent container stops, see moby/discussions/44219
      • no way to access state of a container
    • docker compose:
      • has no relevant go bindings
    • docker swarm:
      • didn’t find any relevant go bindings either

  • for a docker compose down you need to be in the right folder or you won’t be able to do this consistently

    https://github.com/compose-spec/compose-spec/issues/94

  • debugging restarting containers is hard

    if container is in a restart loop, one cannot get a shell in it to access the logs what causes the issue

  • git platform specific document encoding: LF vs. LFCR

    when working on Windows vs. working on Linux expect containers to fail in strange ways:

  • Dockerfile: rebuilding images burns lots of bandwidth and wastes download capacity

    as apk add vim or apt install vim can’t cache their downloads as done in nix/nixos

  • yaml syntax is very limited

    yaml is no programming language and has very limited support to override configuration values compared to nixos options

  • shared volumes -v ${PWD}/documents:/documents

    • UID/GID mapping of host to guest has platform specific differences

      windows/mac blend host rights into the container but on linux files created inside the container often have root:root as owner and even worse, sometimes u+rw g-rwx o-rwx as permission.

  • entrypoint.sh not easily overridable

    the concept of entrypoint.sh is amazing but when deriving a container with a Dockerfile the process to write a correct entrypoint.sh is hard and might easily degrade over time. it’d be great if one could have a mult-stage like environment for this.

  • no way to merge two different docker images

    a workaround is multi-stage builds: copy programs from one image to another but this is only applicable for a few ‘selected’ binaries

  • docker desktop

    i’ve been using docker desktop for windows for 18 months years now and though it is a remarkable help during development, my experience is that it is very early days with huge performance issues, memory leaks and the UI can be very lagging.

docker summary

docker and docker compose is really useful as it brings linux development tools onto windows. docker brings reproducibility compared to manual setup of the environment but not as complete as nix.

on the other hand, for the reasons listed above, docker deployment side lacks massively on all platforms supported.

that said, single containers or simple deployments already bring huge benefits, see:

the docker, docker compose, docker swarm golang API bindings have been stagnant, as pointed out above in parts. in contrast: kubernetes comes with this popular library: https://github.com/kubernetes/client-go which features the concept of a controller which can be used to automate the deployment.

nixcloud.webservices & nixcloud.container

nixos can be compared to docker compose and nixops to docker swarm.

let’s start with a list of pros/cons to nix/nixos usage:

pro:

con:

  • nix expression programming

  • nixos tests

  • nix package manager

    • won’t work with windows

      works in WSL but the mix with native windows tools and WSL is complicated.

  • you are using nixos but the require software not in nixpkgs?

    you want clion or goland which is hypothetically not abstracted in nixpkgs? in my experience this can range from an easy fix for nixpkgs or be a nightmare.

  • no IDE integration for language wrapper of nix

    if you use clion for development, you will have to compile the source manually and once you change is final you will have to adapt the nix expression so also nix can build it.

    if you use goland then dependency management for golang will have a different dependency resolution than the nix expression wrapping it.

    https://discourse.nixos.org/t/nixos-bazel-and-clion-other-ide/1288/10

  • you need to build software for windows native

    won’t work with windows but you might be able to setup a cross-compiler for it.

nix summary

kubernetes

kubernetes works on windows, mac and linux and one can use the controller concept to automate the deployment of services.

say you build docker containers but you use kubernetes to deploy them, then you can use the controller concept to automate the deployment of your services.

kubernetes summary

conclusion

my main question would be: “who or what is the controller your services”.

  • for docker the admin is the controller (no automation)
  • kubernetes has a controller concept, so it can be automated in golang
  • on nixos the controller is likely implemented in the nix language, so it can be automated

hopefully this article sheds some light on the options and helps you to make a decision what stack to use.

article source