NixOS: Enabling LXD virtual machines using Flakes
May 13, 2022
Never miss our publications about Open Source, big data and distributed systems, low frequency of one email every two months.
Nixpkgs is an ever-increasing collection of software packages for Nix and NixOS. Even with more than 80,000 packages, you easily run in a situation where there is a functionality that is not yet implemented.
Earlier this year, we wrote a tutorial on how to implement your own package in NixOS. The implementation of the package in the system was tedious and required keeping track of the package you wanted to implement.
Let’s dive into a practical example: LXD is a container and virtual machine manager. It endorses virtual machines natively since LXD 4.0. However, running on NixOS breaks the feature, and we have to find workarounds to use it. To allow LXD to start virtual machines on NixOS, we need a fix, and we want it to be both reproducible and portable.
To do so, we are using the lxd-agent feature which effectively fix the aforementionned issue. It was released by astridyu, a contributor to nixpkgs. It has not yet been implemented in the Master branch of nixpkgs and is, as of 19/04/2022, a pull request.
Our goal is to integrate the pull request to our nixpkgs to be able to start virtual machines. For this, we will use Flakes. It is an experimental feature of the Nix package manager that allows reproducibility in the deployment of dependencies. We use it to assemble different nixpkgs together. It is a trait known as composability.
If the pull request has been merged, this tutorial becomes irrelevant. To get the functionality, update your Nix or NixOS to the current version of the master branch.
Prerequisites
Step 1 - Installing ZFS
ZFS is a filesystem that we use to generate storage pools in LXD. It is the only filesystem supported by this method, for now.
ZFS must be installed on the NixOS machine prior to the installation of LXD. To do so, go to your configuration.nix file located in /etc/nixos/ and add the following lines to your configuration.
boot = {
# Enabling ZFS
initrd.supportedFilesystems = [ "zfs" ];
supportedFilesystems = [ "zfs" ];
zfs.requestEncryptionCredentials = true;
# to avoid boot error with ZFS
loader.grub.copyKernels=true;
# to avoid zfs to use the freeze option and evade fs corruption
kernelParams= [ "nohibernate" ];
};
services.zfs = {
# These are recommended options for zfs
autoScrub.enable = true;
autoSnapshot.enable = true;
};In the file, the networking.hostId property must be set. It is a bit tricky because it only accepts 32bit ID that you generate on the terminal with head -c 8 /etc/machine-id.
With this ID, go to your configuration file and set it.
networking = {
hostId = "<id>";
};Now, rebuild the switch drive. This is achieved by doing sudo nixos-rebuild switch.
If the build fails because you already have LXD installed, refer to this section of the article.
Step 2 - Installing Flakes
To install Flakes, simply put, in configuration.nix:
# Installing flakes
nix = {
package = pkgs.nixFlakes; # or versioned attributes like nix_2_7
extraOptions = ''experimental-features = nix-command flakes'';
};You must rebuild your switch drive again.
Getting started with Flakes
We will draw on the power of Flakes to patch our system by combining different versions of nixpkgs. The versions we will use are:
You can track the pull request with its number: #166858.
Then, we will proceed to override the LXD package from the nixpkgs master branch version with astridyu’s nixpkgs lxd-vms branch version to get the lxd-agent feature applied onto LXD.
To use Flakes, it is required to create a flake.nix file in /etc/nixos. This is done by doing sudo nano /etc/nixos/flake.nix.
In this file, we describe which nixpkgs repository we want to add to our configuration, and how to merge them.
Note: Verify that your hostname, is defined under the
networking.hostNamefield in your configuration. It is required.
{
description = "NixOS Configuration of LXD";
# Input: nixpkgs we want to have in our configuration
inputs = {
# nixpkgs current branch
nixpkgs.url = "nixpkgs/master";
# nixpkgs fork we want (github:username/repository/branch)
nixpkgs-lxdvm = {
url = "github:astridyu/nixpkgs/lxd-vms";
};
};
# Output: creating an overlay with our new version
outputs = { self, nixpkgs, nixpkgs-lxdvm }:
let
system = "x86_64-linux";
# Creating the overlay of our new system
overlay-lxdvm = final: prev: {
lxdvm = import nixpkgs-lxdvm {
inherit system;
# allow unfree apps
config.allowUnfree = true;
};
};
in {
nixosConfigurations."<hostname>" = nixpkgs.lib.nixosSystem {
inherit system;
modules = [
# Creating an overlay module accessible in configuration.nix
({ config, pkgs, ... }: { nixpkgs.overlays = [ overlay-lxdvm ]; })
./configuration.nix
];
};
};
}Installing LXD
We must modify our configuration.nix to enable LXD virtualization.
# Enables the capacity to launch vm with a virtual socket (network)
boot.kernelModules = ["vhost_vsock"];
# Adding lxd and overriding the package
virtualisation = {
lxd = {
enable=true;
# using the package of our overlay
package = pkgs.lxdvm.lxd.override {useQemu = true;};
recommendedSysctlSettings=true;
};
# ... The rest of your virtualization setup
};We rebuild the switch drive. It is required to use the --impure option upon rebuilding if it is asked by the system.
The reason behind this is that Flakes runs in pure evaluation mode which is underdocumented. Flakes forbids the usage of absolute paths, which could cause the evaluation to be impure. This may happen because we are running unsupported packages.
Getting started with LXD and Virtual Machines
To get started with LXD, we recommend that you initialize a storage pool. This is done by sudo lxd init.
Afterwards, questions are asked concerning the first storage pool that LXD creates. We recommend this configuration for the first usage.
Would you like to use LXD clustering? (yes/no) [default=no]: no
Do you want to configure a new storage pool? (yes/no) [default=yes]: yes
Name of the new storage pool [default=default]: test-storage
Name of the storage backend to use (btrfs, dir, zfs) [default=zfs]: zfs
Create a new ZFS pool? (yes/no) [default=yes]:
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]:
Size in GB of the new loop device (1GB minimum) [default=30GB]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]: test-bridge
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like the LXD server to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:When there is no response, default is assumed.
Once the storage pool is configured, new virtual machines can be initialized with lxc launch . It automatically downloads the image of your virtual machine and sets one up.
To check your virtual machines, do lxc ls.
Common issues
If the rebuild fails, it is due to LXD already being installed. This leads to nixos-rebuild switch not succeeding because an LXD storage pool already exists. It means that volumes are still mounted, and you need to delete them because it is necessary to rebuild LXD from scratch.
Start by regenerating your hardware-configuration.nix by doing nixos-generate-config.
In this file, the path to your mounted volumes is mentioned, and you need to dismount and remove them in the following way:
sudo umount -v /var/lib/lxd/storage-pools/to dismount the storage pool.sudo rm -r /var/lib/lxdto delete the whole LXD folder.
Rebuilding your system is necessary again.
Conclusion
Congratulations! You are now able to launch virtual machines using LXD on NixOS, and you learned how to use flakes to create an overlay for your nixpkgs.
This method will become obsolete once the pull request is merged, but it stays relevant if you want to integrate features that are not in the master branch.
