Building Morph Networks in Hydra

Thu, Sep 1, 2022

Introduction

I use Morph to deploy my NixOS network for a while and I am pretty happy about it. Lately, I’ve decided to add a little machine to the network that is an aarch64/RPI3. However, my custom kernel patches and configurations require a kernel rebuild every once in a while a new version of Linux is released. This made the wait times insurmountable so I decided to give Hydra a try. However, I was not able to find instructions specific to building Morph networks in Hydra. This article is mostly a self-note with the inclusion of a missing piece in the puzzle.

Morph Networks

A Morph network is defined in the following manner according to the example:

let
  pkgs = import (import ../nixpkgs.nix) {};
in
{
  network =  {
    inherit pkgs;
    description = "simple hosts";
    ordering = {
      tags = [ "db" "web" ];
    };
  };

  "web01" = { config, pkgs, ... }: {
    deployment.tags = [ "web" ];

    boot.loader.systemd-boot.enable = true;
    boot.loader.efi.canTouchEfiVariables = true;

    services.nginx.enable = true;

    fileSystems = {
        "/" = { label = "nixos"; fsType = "ext4"; };
        "/boot" = { label = "boot"; fsType = "vfat"; };
    };
  };

  "db01" = { config, pkgs, ... }: {
    deployment.tags = [ "db" ];

    boot.loader.systemd-boot.enable = true;
    boot.loader.efi.canTouchEfiVariables = true;

    services.postgresql.enable = true;

    fileSystems = {
        "/" = { label = "nixos"; fsType = "ext4"; };
        "/boot" = { label = "boot"; fsType = "vfat"; };
    };
  };
}

Basically, it is an attribute set where each attribute other than special ones like network denote a function that which when evaulated with import <nixpkgs/nixos> { configuration = { config, pkgs, libs, …}: … } will return a valid NixOS configuration. Morph evaluates this file via data/eval-machines.nix.

Eiffel - An RPi3 Box

Nothing is particular to my little RPI3 except the architecture.

  "eiffel" = { config, pkgs, ... }: {
 ...omitted...
    nixpkgs.system = "aarch64-linux";
 ...omitted...
  };

So when Morph sees this indication, it looks for a machine that is capable of building the architecture. That is only if buildMachine is the same architecture or the buildMachine has emulation support enabled. This can be achived by the followings:

  boot.binfmt.emulatedSystems = ["aarch64-linux"
  			         "armv6l-linux"
				 "armv7l-linux"];
  nix.buildMachines = [{
    hostName = "localhost";
    systems = ["x86_64-linux"
    	       "aarch64-linux"
	       "armv6l-linux"
	       "armv7l-linux"
	       "i686-linux"];
    #system = "x86_64-linux";
    supportedFeatures = ["kvm" "nixos-test" "big-parallel" "benchmark"];
    maxJobs = 16;
  }];

For instance, this machine has the architecture of x86-64 and we are trying to make it emulate arm architecture for various word size. The first construct enables binfmt emulation support across the operating system via qemu-aarch64 and friends. The second one defines the machine itself as a buildMachine for the specific systems with supported NixOS features. A similar construct should be defined in the box that we wish to execute Morph instructions (ie. my laptop) to tell it to build on the builder (just replace localhost with the builders hostname).

Hydra

Hydra setup is well-defined at NixOS wiki. I believe it boils down to the followings:

  services.hydra = {
    enable = true;
    hydraURL = "http://hydra:PORT";
    listenHost = "IPADDR";
    port = PORT;
    notificationSender = "hydra@core.gen.tr";
    #buildMachinesFiles = [];
    useSubstitutes = true;
  };

This will setup the hydra service on the buildMachine. Replace PORT and IPADDR on RHS’s accordingly.

Next, we will setup a binary-cache for the network, that is the nix-serve daemon.

  services.nix-serve = {
    enable = true;
    bindAddress = "127.0.0.1"; # serve through nginx
    port = PORT;
    secretKeyFile = "/etc/nix/key.private"; 
  };

Replace the PORT according. Then, generate a binary cache key by nix key generate-secret and place it under the services.nix-server.secretKeyFile location.

Final step is to setup nginx as reverse proxy to our binary cache. Pretty straight-forward stuff.

  services.nginx = {
    enable = true;
    virtualHosts = {
      "buildMachine" = {
        #serverAliases = [ "binarycache" ];
        locations."/".extraConfig = ''
        proxy_pass http://localhost:${toString config.services.nix-serve.port};
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      '';
      };
    };
  };

Don’t forget to replace buildMachine with the hostname of the builder.

Note: I’ve tried to use nix-serve binary cache without a front-end but failed. nix-build complains for some derivations that they are not properly signed by a trusted public key.

A Royal Hydra User: Alice

We need to create a Hydra user for managing our builds. The following is given in wiki.

# su - hydra
$ hydra-create-user alice --full-name 'Alice Q. User' \
    --email-address 'alice@example.org' --password-prompt --role admin

Jobsets for the Morph Network

hydra-eiffel

The above is the jobset I use for the machine. The identifier is eiffel. The jobset is Enabled. Nix expression is hydra/eiffel.nix in network. Check interval is 3600 (an hour). The final pieces in this jobset are the inputs. First one is the nixpkgs where I instruct Hydra to pull the nixos-unstable branch. The last one is my local configuration network, still checked out from a git repository. Note that this repo contains the hydra/eiffel.nix file.

hydra/eiffel.nix

Yes, it is simple.

import ./default.nix { machine = "eiffel"; }

hydra/default.nix

Final piece of the puzzle is the following. Takes a machine name and returns a closure that which when evaluated with appropriate parameters evaluates the machine config that is eiffel in this case.

{ machine }:
{ ... }:
let
 nixpkgs = import <nixpkgs> {};
 evaluateMorph = networkExpr: import "${nixpkgs.morph.lib}/eval-machines.nix" {
    inherit networkExpr;
 };
 morphResult = evaluateMorph ../network.nix;
in {
  "${machine}" = morphResult.nodes.${machine}.config.system.build.toplevel;
}

Conclusion

After deploying Hydra to build aarch64 eiffel on my build machine, I no longer have to wait for morph deploy to compile everything from scratch. The binary cache will serve all the files I need and guess what? They will be signed by the builders’ secret key. That’s a very nice property of the nix-store, it allows us to define trusted-public-keys for those only will it accept binaries. Almost like a magic ;)

hydra-result

I hope you enjoyed the tour. If you have trouble finding your way in Hydra, there is a wonderful presentation by P. Simons at youtube.

If you like to have your Linux network managed like this, or better if you like to build a Distributed Quantum Computing Simulator, let me know. I have built one last year and would like to build a better, more capable one. If your institute or corporate has a vision for building one, you can find me at LinkedIn.

Until next time!