tech-ingolf-wagner-de/content/nixos/nix-instantiate.md

title	date	draft	tags	summary
nix-instantiate	2018-12-05T19:09:36+02:00	false	NixOS Ansible Terraform	a tool to create your JSON generator.

nix-instantiate

I like NixOS and the way modules work. I miss them when I do tasks in other languages that have less power than NixOS, for example Ansible and Terraform.

Luckily all these tools can be configured via JSON, and Nix can easily create JSON. The go-to tool for that job is nix-instantiate which every NixOS has installed (yeye!).

Small overview

I will show you how easy it is, with a few lines of nix, to create a JSON renderer for terraform configuration files. But this is only done in a sketchy way, to inspire you to create your own setup for different tools that use JSON.

If you are interessted in a full (or almost full) terraform JSON renderer have a look at my terranix project.

First tests

Lets look what nix-instantiate does.

We create a file test1.nix:

# file test1.nix
rec {
  i = "like Nix";
  you = i;
}

and than we run nix-instantiate to render JSON:

$> nix-instantiate --eval --json --strict test1.nix | jq

{
  "i": "like Nix",
  "you": "like Nix"
}

Nice! This is expected because it is an example from the documentation.

Modules for the win

Modules are one of the things that make NixOS really awesome. So lets us them in combination with nix-instantiate!

# file test2.nix
let
  pkgs = import <nixpkgs> {};

  result =
    with pkgs;
    with lib;
    evalModules {
      modules = [
        # option definition
        {
          options = {
            resource = mkOption {
              type    = with types; attrsOf attrs;
              default = {};
            };
          };
        }
        # config definition
        {
          resource."random_pet" = {
            "house_pet".length = 10;
            "neighbours_pet".length = 10;
          };
        }
      ];
    };
in
  result.config

When running:

nix-instantiate --eval --strict --json test2.nix --show-trace | jq

We get the following JSON:

{
  "_module": {
    "args": {},
    "check": true
  },
  "resource": {
    "random_pet": {
      "house_pet": {
        "length": 10
      },
      "neighbours_pet": {
        "length": 10
      }
    }
  }
}

This is almost what we want to see. the _module value is not needed. So let's remove it with a sanitization function, and move the content path to a different file called config.nix.

# file test3.nix
let
  pkgs = import <nixpkgs> {};

  sanitize =
    with pkgs;
    configuration:
    builtins.getAttr (builtins.typeOf configuration) {
        bool = configuration;
        int = configuration;
        string = configuration;
        list = map sanitize configuration;
        set = lib.mapAttrs
          (lib.const sanitize)
          (lib.filterAttrs (name: value: name != "_module" && value != null) configuration);
      };

  result =
    with pkgs;
    with lib;
    evalModules {
      modules = [ { imports = [ ./config.nix ]; } ];
    };
in
  (sanitize result.config)

In config.nix we can now focus on the configuration content. And we write it just like we would write a NixOS module.

# config.nix
{ config, lib, ... }:
with lib;
{
  options = {
    resource = mkOption {
      type    = with types; attrsOf attrs;
      default = {};
    };
  };

  config = {
    resource."random_pet" = {
      "house_pet".length = 10;
      "neighbours_pet".length = 10;
    };
  };
}

The result of the now well known command

nix-instantiate --eval --strict --json test3.nix --show-trace | jq

looks like the result we want to have:

{
  "resource": {
    "random_pet": {
      "house_pet": {
        "length": 10
      },
      "neighbours_pet": {
        "length": 10
      }
    }
  }
}

Now we have the full power of the NixOS module system to generate JSON. We can write modules to hide complexity and create very well readable terraform or ansible setups without the need of their strange tooling which is not capable of mapping, filtering or hiding complexity.

A Simple Example

Let's make an example so a non-NixOS-veteran can see how to start using this modules system.

hcloud.nix

The following file is a module that let's us create resource entries to create an hcloud server. But it has one parameter additionalFileSize which will automatically add an hcloud_volume and an hcloud_volume_attachment.

# hcloud.nix
{ config, lib, ... }:
with lib;
let
  cfg = config.hcloud.server;
in {
  options.hcloud.server = mkOption {
    default = {};
    type = with types; attrsOf (submodule ( { name, ... }: {
      options = {
        # mandatories : because no default is set
        server_type = mkOption {
          type    = with types; enum ["cx11" "cx21" "cx31" "cx41" "cx51"];
        };
        image = mkOption {
          type    = with types; string;
          example = "ubuntu";
          description = ''
            image to install
          '';
        };
        # optionals
        additionalFileSize = mkOption {
          type    = with types; nullOr ints.positive;
          default = null;
          description = ''
            extra volume (in GB) that should be added.
          '';
        };
      };
    }));
  };

  config =
    let
      serverResources = {
        resource.hcloud_server =
          mapAttrs (name: configuration: {
            inherit (configuration) server_type image;
            name = name;
          } ) cfg;
      };

      additionals = filterAttrs (_: configuration: configuration.additionalFileSize != null) cfg;
      additionVolumesResources = {
        resource."hcloud_volume" = mapAttrs' (name: configuration:
          nameValuePair "${name}" {
            name = "${name}_volume";
            size = configuration.additionalFileSize;
          }
        ) additionals;
      };
      additionVolumesResourcesAttatchments = {
        resource."hcloud_volume_attachment" = mapAttrs' (name: configuration:
          nameValuePair "${name}_volume_attachment" {
            volume_id = "\${hcloud_volume.${name}.id}";
            server_id = "\${hcloud_server.${name}.id}";
            automount = true;
          }
        ) additionals;
      };
    in
      mkMerge [
        serverResources
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResources )
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResourcesAttatchments )
      ];
}

config.nix and Output

Without additionalFileSize

Let's look at the different config.nix results.

{
  imports = [
    ./core.nix       # resource definition
    ./hcloud.nix     # the hcloud_server module
  ];
  # define a hcloud_server
  hcloud.server = {
    "test" = {
      image = "ubuntu";
      server_type = "cx11";
    };
  };
}

$> nix-instantiate --eval --strict --json test3.nix --show-trace | jq
{
  "hcloud": {
    "server": {
      "test": {
        "image": "ubuntu",
        "server_type": "cx11"
      }
    }
  },
  "resource": {
    "hcloud_server": {
      "test": {
        "image": "ubuntu",
        "name": "test",
        "server_type": "cx11"
      }
    }
  }
}

The output is like we expected it to be.

The hcloud parameter should be removed, but for now I will leave it here, to see the original configuration. To make this work in terraform, you have to remove everything except resource.

With additionalFileSize

Let's add some additionalFileSize.

{
  imports = [
    ./core.nix       # resource definition
    ./hcloud.nix     # the hcloud_server module
  ];
  # define a hcloud_server
  hcloud.server = {
    "test" = {
      image = "ubuntu";
      server_type = "cx11";
      additionalFileSize = 100;
    };
  };
}

$> nix-instantiate --eval --strict --json test3.nix --show-trace | jq
{
  "hcloud": {
    "server": {
      "test": {
        "additionalFileSize": 100,
        "image": "ubuntu",
        "server_type": "cx11"
      }
    }
  },
  "resource": {
    "hcloud_server": {
      "test": {
        "image": "ubuntu",
        "name": "test",
        "server_type": "cx11"
      }
    },
    "hcloud_volume": {
      "test": {
        "name": "test_volume",
        "size": 100
      }
    },
    "hcloud_volume_attachment": {
      "test_volume_attachment": {
        "automount": true,
        "server_id": "${hcloud_server.test.id}",
        "volume_id": "${hcloud_volume.test.id}"
      }
    }
  }
}

Whoa, a lot of other resources joined the party. Additionally, the additionalFileSize parameter is properly removed from resource.hcloud_server.test.

You could also create this very simple example with variables, locals and count. By doing that, you already reached the limits of HCL but in Nix this is a very simple example.

A More Complex Example

Lets create something you wouldn't be able to do in HCL anymore.

Imagine you have an inner circle of admins, which need access to all machines created. So when a machine is created we also add all admin keys.

Let's look at the config first how it would look like.

{
  imports = [
    ./core.nix
    ./hcloud.nix
    ./admins.nix
  ];

  # all mighty admins
  admins = {
    palo.ssh_key = "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ palo@someMachine";
    tv.ssh_key = "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ tv@someMachine";
    lass.ssh_key = "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ lass@someMachine";
  };

  hcloud.server = {
    "test" = {
      image = "ubuntu";
      server_type = "cx11";
    };
  };
}

admins.nix

But this time the admins module will not create any resource directly.

# admins.nix
{ lib, ... }:
with lib;
{
  options.admins = mkOption {
    default = {};
    type = with types; attrsOf ( submodule ( { name, ... }: {
      options = {
        ssh_key = mkOption {
          type    = with types; string;
          description = ''
            public key of admin.
          '';
        };
      };
    }));
  };
}

hcloud.nix

These options are used in the hcloud.nix file and of course every other module you write, where you write which servers to create.

{ config, lib, ... }:
with lib;
let
  cfg = config.hcloud.server;
in {

  options.hcloud.server = mkOption {
    default = {};
    type = with types; attrsOf (submodule ( { name, ... }: {
      options = {
        # mandatories : because no default is set
        server_type = mkOption {
          type    = with types; enum ["cx11" "cx21" "cx31" "cx41" "cx51"];
        };
        image = mkOption {
          type    = with types; string;
          example = "ubuntu";
          description = ''
            image to install
          '';
        };
        # optionals
        additionalFileSize = mkOption {
          type    = with types; nullOr ints.positive;
          default = null;
          description = ''
            extra volume (in GB) that should be added.
          '';
        };
      };
    }));
  };

  config =
    let
      serverResources = {
        resource.hcloud_server =
          mapAttrs (name: configuration: {
            inherit (configuration) server_type image;
            name = name;

          # we add the ssh key ids, if admins exist
          } // (optionalAttrs (length adminSshKeyIds > 0) { ssh_keys = adminSshKeyIds; })

          ) cfg;
      };

      additionals = filterAttrs (_: configuration: configuration.additionalFileSize != null) cfg;
      additionVolumesResources = {
        resource."hcloud_volume" = mapAttrs' (name: configuration:
          nameValuePair "${name}" {
            name = "${name}_volume";
            size = configuration.additionalFileSize;
          }
        ) additionals;
      };
      additionVolumesResourcesAttatchments = {
        resource."hcloud_volume_attachment" = mapAttrs' (name: configuration:
          nameValuePair "${name}_volume_attachment" {
            volume_id = "\${hcloud_volume.${name}.id}";
            server_id = "\${hcloud_server.${name}.id}";
            automount = true;
          }
        ) additionals;
      };

      adminSshKeyIds = map (name: "\${hcloud_ssh_key.${name}.id}") (attrNames config.admins);
      adminSshKeys = {
        resource."hcloud_ssh_key" = mapAttrs (name: configuration: {
          name = name;
          public_key = configuration.ssh_key;
        }) config.admins; };
    in
      mkMerge [
        serverResources
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResources )
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResourcesAttatchments )

        # we create hcloud_ssh_keys, if admins exist
        ( mkIf ( length ( attrNames config.admins ) > 0 )
          adminSshKeys)

      ];
}

The hcloud.nix starts to get big now, and it is very similar to the version from the privious section. Focus on the last section on mkMerge and in config look closely at the end of the serverResource definition.

Output

Let's look at the resulting JSON:

$> nix-instantiate --eval --strict --json test3.nix --show-trace | jq
{
  "admins": {
    "lass": {
      "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ lass@someMachine"
    },
    "palo": {
      "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ palo@someMachine"
    },
    "tv": {
      "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ tv@someMachine"
    }
  },
  "hcloud": {
    "server": {
      "test": {
        "image": "ubuntu",
        "server_type": "cx11"
      }
    }
  },
  "resource": {
    "hcloud_server": {
      "test": {
        "image": "ubuntu",
        "name": "test",
        "server_type": "cx11",
        "ssh_keys": [
          "${hcloud_ssh_key.lass.id}",
          "${hcloud_ssh_key.palo.id}",
          "${hcloud_ssh_key.tv.id}"
        ]
      }
    },
    "hcloud_ssh_key": {
      "lass": {
        "name": "lass",
        "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ lass@someMachine"
      },
      "palo": {
        "name": "palo",
        "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ palo@someMachine"
      },
      "tv": {
        "name": "tv",
        "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ tv@someMachine"
      }
    }
  }
}

Nice! All 3 keys will be created by hcloud_ssh_key and they all get wired to the new hcloud_server.

This should give you a feeling how you can maintain your JSON/YAML-configured tools, with nix-instantiate and the NixOS module system.

Happy Hacking!

Thanks

Thanks to tv for his introduction to nix-instantiate. Thanks to lassulus and kmein for polishing this article.