tech-ingolf-wagner-de/content/nixos/nix-instantiate.md

title	date	draft	tags	summary
nix-instantiate	2018-12-05T19:09:36+02:00	false	NixOS Ansible Terraform	a tool to create your JSON generator.

nix-instantiate

I like NixOS and the way modules work. I miss it when I do tasks in other languages, which have less power than NixOS, for example Ansible and Terraform.

Luckily all these tools can be configured via JSON, and Nix can easily create JSON. The tool go to tool for that job is nix-instantiate which every NixOS has installed (yeye!).

Small overview

I will show you how easy it is, with a few lines of nix to create a JSON renderer for terraform configuration files. But this is only done in a sketchy way, to inspire you to create your own setup for different tools that use JSON.

If you are interessted in a full (or almost full) terraform JSON renderer have a look at my terranix project

First tests

Lets look what nix-instantiate does.

We create a file test1.nix

# file test1.nix
rec {
  i = "like Nix";
  you = i;
}

and than we run nix-instantiate to render json

$> nix-instantiate --eval --json --strict test1.nix | jq

{
  "i": "like Nix",
  "you": "like Nix"
}

Nice! this is expected because it is an example from the documentation.

Modules for the win

Modules are one of the things that make NixOS really awesome. So lets us them in combination with nix-instantiate!

# file test2.nix
let
  pkgs = import <nixpkgs> {};

  result =
    with pkgs;
    with lib;
    evalModules {
      modules = [
        # option definition
        {
          options = {
            resource = mkOption {
              type    = with types; attrsOf attrs;
              default = {};
            };
          };
        }
        # config definition
        {
          resource."random_pet" = {
            "house_pet".length = 10;
            "neighbours_pet".length = 10;
          };
        }
      ];
    };
in
  result.config

When running

nix-instantiate --eval --strict --json test2.nix --show-trace | jq

we get the following json.

{
  "_module": {
    "args": {},
    "check": true
  },
  "resource": {
    "random_pet": {
      "house_pet": {
        "length": 10
      },
      "neighbours_pet": {
        "length": 10
      }
    }
  }
}

This is almost what we like to see. the _module value is not wanted. so lets remove it with a sanitize function, and move the content path in a different file called config.nix

# file test3.nix
let
  pkgs = import <nixpkgs> {};

  sanitize =
    with pkgs;
    configuration:
    builtins.getAttr (builtins.typeOf configuration) {
        bool = configuration;
        int = configuration;
        string = configuration;
        list = map sanitize configuration;
        set = lib.mapAttrs
          (lib.const sanitize)
          (lib.filterAttrs (name: value: name != "_module" && value != null) configuration);
      };

  result =
    with pkgs;
    with lib;
    evalModules {
      modules = [ { imports = [ ./config.nix ]; } ];
    };
in
  (sanitize result.config)

in config.nix we can now focus on the configuration content. And it we write it just like we would write a NixOS module.

# config.nix
{ config, lib, ... }:
with lib;
{
  options = {
    resource = mkOption {
      type    = with types; attrsOf attrs;
      default = {};
    };
  };

  config = {
    resource."random_pet" = {
      "house_pet".length = 10;
      "neighbours_pet".length = 10;
    };
  };
}

The result of the now well known command

nix-instantiate --eval --strict --json test3.nix --show-trace | jq

looks like the result we want to have.

{
  "resource": {
    "random_pet": {
      "house_pet": {
        "length": 10
      },
      "neighbours_pet": {
        "length": 10
      }
    }
  }
}

Now we have the full power of the NixOS module system to generate JSON. We can write modules to hide complexity and create very well readable terraform or ansible setups without the need of their strange tooling which is not capable of mapping, filtering or hiding complexity.

a simple example

Let's make an example a none NixOS veteran can see how you would start using this modules system.

hcloud.nix

The following file should show is a module that let's us create resource entries to create a hcloud server. But it has one parameter additionalFileSize which will automatic add a hcloud_volume and a hcloud_volume_attachment.

# hcloud.nix
{ config, lib, ... }:
with lib;
let
  cfg = config.hcloud.server;
in {
  options.hcloud.server = mkOption {
    default = {};
    type = with types; attrsOf (submodule ( { name, ... }: {
      options = {
        # mandatories : because no default is set
        server_type = mkOption {
          type    = with types; enum ["cx11" "cx21" "cx31" "cx41" "cx51"];
        };
        image = mkOption {
          type    = with types; string;
          example = "ubuntu";
          description = ''
            image to install
          '';
        };
        # optionals
        additionalFileSize = mkOption {
          type    = with types; nullOr ints.positive;
          default = null;
          description = ''
            extra volume (in GB) that should be added.
          '';
        };
      };
    }));
  };

  config =
    let
      serverResources = {
        resource.hcloud_server =
          mapAttrs (name: configuration: {
            inherit (configuration) server_type image;
            name = name;
          } ) cfg;
      };

      additionals = filterAttrs (_: configuration: configuration.additionalFileSize != null) cfg;
      additionVolumesResources = {
        resource."hcloud_volume" = mapAttrs' (name: configuration:
          nameValuePair "${name}" {
            name = "${name}_volume";
            size = configuration.additionalFileSize;
          }
        ) additionals;
      };
      additionVolumesResourcesAttatchments = {
        resource."hcloud_volume_attachment" = mapAttrs' (name: configuration:
          nameValuePair "${name}_volume_attachment" {
            volume_id = "\${hcloud_volume.${name}.id}";
            server_id = "\${hcloud_server.${name}.id}";
            automount = true;
          }
        ) additionals;
      };
    in
      mkMerge [
        serverResources
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResources )
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResourcesAttatchments )
      ];
}

config.nix and output

without additionalFileSize

Lets look at the different config.nix results.

{
  imports = [
    ./core.nix       # resource definition
    ./hcloud.nix     # the hcloud_server module
  ];
  # define a hcloud_server
  hcloud.server = {
    "test" = {
      image = "ubuntu";
      server_type = "cx11";
    };
  };
}

$> nix-instantiate --eval --strict --json test3.nix --show-trace | jq
{
  "hcloud": {
    "server": {
      "test": {
        "image": "ubuntu",
        "server_type": "cx11"
      }
    }
  },
  "resource": {
    "hcloud_server": {
      "test": {
        "image": "ubuntu",
        "name": "test",
        "server_type": "cx11"
      }
    }
  }
}

The output is quite like we expected it to be.

The hcloud parameter should be removed, but for now it I leave it here, to see the original configuration. To make this work in terraform, you have to remove everything of course, except resource.

with additionalFileSize

Lets add some additionalFileSize.

{
  imports = [
    ./core.nix       # resource definition
    ./hcloud.nix     # the hcloud_server module
  ];
  # define a hcloud_server
  hcloud.server = {
    "test" = {
      image = "ubuntu";
      server_type = "cx11";
      additionalFileSize = 100;
    };
  };
}

$> nix-instantiate --eval --strict --json test3.nix --show-trace | jq
{
  "hcloud": {
    "server": {
      "test": {
        "additionalFileSize": 100,
        "image": "ubuntu",
        "server_type": "cx11"
      }
    }
  },
  "resource": {
    "hcloud_server": {
      "test": {
        "image": "ubuntu",
        "name": "test",
        "server_type": "cx11"
      }
    },
    "hcloud_volume": {
      "test": {
        "name": "test_volume",
        "size": 100
      }
    },
    "hcloud_volume_attachment": {
      "test_volume_attachment": {
        "automount": true,
        "server_id": "${hcloud_server.test.id}",
        "volume_id": "${hcloud_volume.test.id}"
      }
    }
  }
}

Oha, a lot of other resources joined the party. Additionally the additionalFileSize parameter is properly removed from resource.hcloud_server.test.

You could also create this very simple example with variables, locals and count. But doing that you already reached the limits of HCL but in Nix this is a very simple example.

A more complex Example

Lets create something you wouldn't be able to do in HCL anymore.

Imagine you have a inner circle of admins, which need access to all machines created. So when a machine is created we also add all admin keys.

Let's look at the config first how it would look like.

{
  imports = [
    ./core.nix
    ./hcloud.nix
    ./admins.nix
  ];

  # all mighty admins
  admins = {
    palo.ssh_key = "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ palo@someMachine";
    tv.ssh_key = "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ tv@someMachine";
    lass.ssh_key = "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ lass@someMachine";
  };

  hcloud.server = {
    "test" = {
      image = "ubuntu";
      server_type = "cx11";
    };
  };
}

admins.nix

But this time admins module will not create any resource directly.

# admins.nix
{ lib, ... }:
with lib;
{
  options.admins = mkOption {
    default = {};
    type = with types; attrsOf ( submodule ( { name, ... }: {
      options = {
        ssh_key = mkOption {
          type    = with types; string;
          description = ''
            public key of admin.
          '';
        };
      };
    }));
  };
}

hcloud.nix

These options are used in the hcloud.nix file and of course every other module you write, where you write which create servers.

{ config, lib, ... }:
with lib;
let
  cfg = config.hcloud.server;
in {

  options.hcloud.server = mkOption {
    default = {};
    type = with types; attrsOf (submodule ( { name, ... }: {
      options = {
        # mandatories : because no default is set
        server_type = mkOption {
          type    = with types; enum ["cx11" "cx21" "cx31" "cx41" "cx51"];
        };
        image = mkOption {
          type    = with types; string;
          example = "ubuntu";
          description = ''
            image to install
          '';
        };
        # optionals
        additionalFileSize = mkOption {
          type    = with types; nullOr ints.positive;
          default = null;
          description = ''
            extra volume (in GB) that should be added.
          '';
        };
      };
    }));
  };

  config =
    let
      serverResources = {
        resource.hcloud_server =
          mapAttrs (name: configuration: {
            inherit (configuration) server_type image;
            name = name;

          # we add the ssh key ids, if admins exist
          } // (optionalAttrs (length adminSshKeyIds > 0) { ssh_keys = adminSshKeyIds; })

          ) cfg;
      };

      additionals = filterAttrs (_: configuration: configuration.additionalFileSize != null) cfg;
      additionVolumesResources = {
        resource."hcloud_volume" = mapAttrs' (name: configuration:
          nameValuePair "${name}" {
            name = "${name}_volume";
            size = configuration.additionalFileSize;
          }
        ) additionals;
      };
      additionVolumesResourcesAttatchments = {
        resource."hcloud_volume_attachment" = mapAttrs' (name: configuration:
          nameValuePair "${name}_volume_attachment" {
            volume_id = "\${hcloud_volume.${name}.id}";
            server_id = "\${hcloud_server.${name}.id}";
            automount = true;
          }
        ) additionals;
      };

      adminSshKeyIds = map (name: "\${hcloud_ssh_key.${name}.id}") (attrNames config.admins);
      adminSshKeys = {
        resource."hcloud_ssh_key" = mapAttrs (name: configuration: {
          name = name;
          public_key = configuration.ssh_key;
        }) config.admins; };
    in
      mkMerge [
        serverResources
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResources )
        ( mkIf ( length ( attrNames additionals ) > 0 )
          additionVolumesResourcesAttatchments )

        # we create hcloud_ssh_keys, if admins exist
        ( mkIf ( length ( attrNames config.admins ) > 0 )
          adminSshKeys)

      ];
}

The hcloud.nix starts to get big now, and it is very similar to the version from the privious section. Focus on the last section on mkMerge and in config look closely at the end of the serverResource definition.

output

Lets look at the resulting JSON

$> nix-instantiate --eval --strict --json test3.nix --show-trace | jq
{
  "admins": {
    "lass": {
      "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ lass@someMachine"
    },
    "palo": {
      "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ palo@someMachine"
    },
    "tv": {
      "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ tv@someMachine"
    }
  },
  "hcloud": {
    "server": {
      "test": {
        "image": "ubuntu",
        "server_type": "cx11"
      }
    }
  },
  "resource": {
    "hcloud_server": {
      "test": {
        "image": "ubuntu",
        "name": "test",
        "server_type": "cx11",
        "ssh_keys": [
          "${hcloud_ssh_key.lass.id}",
          "${hcloud_ssh_key.palo.id}",
          "${hcloud_ssh_key.tv.id}"
        ]
      }
    },
    "hcloud_ssh_key": {
      "lass": {
        "name": "lass",
        "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ lass@someMachine"
      },
      "palo": {
        "name": "palo",
        "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ palo@someMachine"
      },
      "tv": {
        "name": "tv",
        "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAA......hKIWndLJ tv@someMachine"
      }
    }
  }
}

Nice all 3 keys will be created by hcloud_ssh_key and they all get wired to the new hcloud_server.

This should give you a feeling how you can maintain your JSON/YAML-configured tools, with nix-instantiate and the NixOS module system.

Happy Hacking!

Thanks

Thanks to tv for his introduction to nix-instantiate.