Nix Flake Architecture in Practice

A Beginner-friendly Tour following a Basic Vim Plugin

Getting into Nix & Nix flakes can be a challenge. You may have have heard of Nix’s fame for reproducibility or Nix flake’s composability, but weren’t sure where or how to start. While some folks seem to settle for a devShell when it comes to Nix, going just a bit deeper, Nix can fullfill more project architecture requirements than merely delivering tooling. In this post we will follow journey of requirements from environment setup, to building, testing, & distributing a “Hello World” Vim plugin as the guide for learning the Nix flakes’s API

Our Vim Plugin’s Requirements

Even “Hello, world!” has requirements. For our plugin we will need:

The software requirements
  • a plugin that prints “Hello, world!” to the screen
The architecture requirements to meet the software requirements
  • a clean-slate, isolated environment (or editor (Vim) in this case) for that plugin
  • a way to rapidly update that environment
  • a way to lint and/or test our code
  • a way to distribute/deploy our plugin

Beflaked project

Flakes may be “experimental”, but there has been a push to stabilize them very soon. The noteworthy benefits of flakes are composability, easy management via a lockfile, & a standardized way for users / contributors to understand our code.

If you have yet to enable, flakes on your machine, now is the time!
For NixOS users, edit the configuration.nix
{ pkgs, ... }: {
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
}
For non-NixOS Nix users
$ mkdir -p ${XDG_CONFIG_HOME:-~/.config}/nix
$ echo "experimental-features = nix-command flakes" >> ~/${XDG_CONFIG_HOME:-~/.config}/nix/nix.conf

Echoing the most basic vim-hello-world, a Vim plugin

Since this isn’t a Vim plugin tutorial per se, here’s the most basic echo that can be a stand-in for any script or program.

$ cd $PROJECT
$ mkdir -p plugin
$ echo "echo 'Hello, world!'" >> plugin/hello_world.vim
Tip

If you’re intending to distribute this plugin as free software, now’s a good time choose a license & create a LICENSE.txt or COPYING.txt.

Beginning with a flake.nix

With flakes installed, it’s time to create our $PROJECT’s flake.nix

$ cd $PROJECT
$ nix flake init
$ cat flake.nix
{
  description = "A very basic flake";

  outputs = { self, nixpkgs }: {

    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;

    packages.x86_64-linux.default = self.packages.x86_64-linux.hello;

  };
}

This base can get us started, but it’s in need of tweaking for our projects.

Supporting forAll architectures

Vim (& Neovim) support on a lot of platforms, & we should also support those users (plus Nix will require us to specify these platforms as we can see in the nix flake init output). There are a lot of ways to handle multiple hardware architecture support in Nix, but we will be writing our own utility functions to avoid introducing unnecessary dependencies. When you add to inputs, downstream users will necessarily need to pull our dependencies even if it’s just for a single function so be careful & audit your inputs. It’s never worth it to pull in a Nix utility library for what is effectively a for loop over our supported systems. Here’s one way to write that loop:

# …assumes nixpkgs is in scope such as a flake.nix’s outputs
forAllSystems = nixpkgs.lib.genAttrs nixpkgs.lib.platforms.unix;

nixpkgsFor = forAllSystems (system: import nixpkgs {
  inherit system;
  config = { };
  overlays = [ ];
});
Caution!

Supporting all possible platforms isn’t always the best practice. If you don’t physically own hardware & test on that platform, you can never be sure that it will work—epecially if you’re doing platform-specific or low-level things (but we’ve all been there when an npm package pulled assumed something about our system & put us in a broken state). However, using the editor’s built-in APIs like many plugins do, we should be able to trust the upstream Vim/Neovim testing that our project will work on those platforms.

Tip

If you want to test with the latest package versions from Nix, you may consider nix-unstable

You can use the same technique to point to other versions of Nixpkgs if you need commits that contain specific package versions you need to test against.

How to set nix-unstable in one’s Flake’s inputs
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };
}

Making some packages

packages are the public-facing derivations we want to give access to users. At a minimum we will need two packages with for our software:

  1. Our plugin (our code)
  2. A version of Vim (or Neovim) we will run which only has our dependencies (a modified version of something we are hacking on)

Our plugin

Many languages have handy helpers in Nixpkgs to use existing community tooling as well such as ocamlPackages.buildDuneProject or rustPlatform.buildRustPackage. In the style of many public Vim plugins, our plugin will merely sit in repository’s root directory—no runtimepath switching, no additional build steps. For Vim, we can pull one of those helper in vimUtils.buildVimPlugin (formerly known as vimUtils.buildVimPluginFrom2Nix). This function will behave similarly to how many Vim package managers behave—which is perfect for our basic hello_world. We will add our first package, & we’ll use the name default for it.

{
  description = "A Hello World demo plugin for Vim";

  # not default, but `inputs` pun (`@`) of our input records argument
  outputs = { self, nixpkgs, ... }@inputs:
    let
      forAllSystems = nixpkgs.lib.genAttrs nixpkgs.lib.platforms.unix;

      nixpkgsFor = forAllSystems (system: import nixpkgs {
        inherit system;
        config = { };
        overlays = [ ];
      });
    in
    {
      packages = forAllSystems (system:
        let pkgs = nixpkgsFor."${system}"; in {
          default = pkgs.vimUtils.buildVimPlugin {
             name = "hello-world";
             namePrefix = "vim-";
             src = ./.;
           };
        });
    };
}

Our modified Vim editor

Since we are working from that clean-slate plan, we need to make clean Vim package. We will call this vim-mod to remind ourselves that it’s not the one on the system but the one we have modified. Let’s add it to the packages (leaving out sections we’ve seen to keep it short):

{
  outputs = { self, nixpkgs, ... }@inputs:
    let
      # …
    in
    {
      packages = forAllSystems (system:
        let pkgs = nixpkgsFor."${system}"; in {
          vim-mod = pkgs.vim_configurable.customize {
            vimrcConfig = {
              packages.myPlugin = with pkgs.vimPlugins; {
                start = [
                  # list plugins we need to be compatible with such as:
                  # ale
                  # powerline
                  # …but also include our plugin
                  self.packages."${system}".default
                ];
              };
              customRC = ''
                " A VimL RC file for our default settings for testing

                syntax on

                " Maybe we need flags for our plugin too here
              '';
            };
          };
        });
    };
}
Note

Very similar setting are used for Neovim, just with pkgs.neovim. If it suits your plugin, there’s no reason you couldn’t create & test on both with the same or similar configuration! Refer to the Vim documentation on the NixOS wiki to learn more about both.

Inspecting our output

By running nix-build with our directory as the entry followed with a # & our package name, vim-mod, we’ll instruct the Nix to build just that. Keep in mind that since our modified Vim has our plugin as an input, it will be built as well. After it’s built we can execute our modified Vim.

$ nix build .#vim-mod
$ tree ./result
result/
├── bin
│   ├── eview -> /nix/store/$HASH1-vim-bin/bin/eview
│   ├── evim -> /nix/store/$HASH1-vim-bin/bin/evim
│   ├── ex -> /nix/store/$HASH1-vim-bin/bin/ex
│   ├── gview -> /nix/store/$HASH1-vim-bin/bin/gview
│   ├── gvim -> /nix/store/$HASH1-vim-bin/bin/gvim
│   ├── gvimdiff -> /nix/store/$HASH1-vim-bin/bin/gvimdiff
│   ├── gvimtutor -> /nix/store/$HASH2-vim-full-9.0.1562/bin/gvimtutor
│   ├── rgview -> /nix/store/$HASH1-vim-bin/bin/rgview
│   ├── rgvim -> /nix/store/$HASH1-vim-bin/bin/rgvim
│   ├── rview -> /nix/store/$HASH1-vim-bin/bin/rview
│   ├── rvim -> /nix/store/$HASH1-vim-bin/bin/rvim
│   ├── vi -> /nix/store/$HASH1-vim-bin/bin/vi
│   ├── view -> /nix/store/$HASH1-vim-bin/bin/view
│   ├── vim -> /nix/store/$HASH1-vim-bin/bin/vim
│   ├── vimdiff -> /nix/store/$HASH1-vim-bin/bin/vimdiff
│   ├── vimtutor -> /nix/store/$HASH2-vim-full-9.0.1562/bin/vimtutor
│   └── xxd -> /nix/store/$HASH2-vim-full-9.0.1562/bin/xxd
└── share -> /nix/store/$HASH2-vim-full-9.0.1562/share
$ result/bin/vim
result/bin/vim
Hello, world!
Press ENTER or type command to continue
Hint

The dot, ., in .#vim-mod refers to the current working directory or $PWD, just like when you ls . - the # is the flake in that directory, and vim-mod is the package name in the flake.

Using apps

An app is a derivation that’s meant to be run by a users—this can be as simple as a CLI tool, or a server, or complex orchestration of tools, servers, & services. Since our use case is basic, the app we want is a modified Vim executable preinstalled plugin with just our plugin & its dependencies. At the present, every time we need to make a change to our plugin we will need to rebuild & then execute the result/bin/vim command to test. This can be tedious. Instead we can build a app that will launch our modified editor.

{
  outputs = { self, nixpkgs, ... }@inputs:
    let
      # …
    in
    {
      packages = forAllSystems (system: {
        # …
      });

      apps = forAllSystems (system:
        let pkgs = nixpkgsFor.${system}; in {
          vim-mod = {
            type = "app";
            program = "${self.packages.${system}.vim-mod}/bin/vim";
          };
        });
    };
}

This lets us reference a named application vim-mod any time we call nix run .#vim-mod.

$ nix run .#vim-mod
Hello, world!
Press ENTER or type command to continue

A simple, quick way to do development might be to make tweaks to our plugin followed by a nix run in another terminal to compile in our changes & spawn a new Vim instance. In more complex apps, the app can be instructed to recompile on file changes or (re)start systemd services.

Tip

Arguments can be passed with -- such as nix run .#vim-mod -- $filename. With Vim you can run commands/functions too with nix run .#vim-mod -- +MyFunction.

Getting linting & testing feedback

Nix’s checks

checks can be used to set up automated tests, linters, formatters, or really anything we want to assert about our code base. While we can implement as many checks as we want, for this demonstration we only set up vint, a Vim Script linter.

{
  outputs = { self, nixpkgs, ... }@inputs:
    let
      # …
    in
    {
      checks = forAllSystems (system:
        let pkgs = nixpkgsFor.${system}; in {
          vint = pkgs.runCommand "vint-lint"
            {
              src = ./.;
              nativeBuildInputs = with pkgs; [ vim-vint ];
            } ''
              set -euo pipefail
              vint --style-problem $src/**/*.vim | tee $out
            '';
        });
    };
}

Now in our Nix environment, we can have vint run across all of our code with a one-liner

$ nix flake check

We could choose to run this in CI or attach it to certain VCS hooks or run manually. We can also append even more checks such as vader.vim for tests or editorconfig-checker to verify our white space rules are followed.

Uses for devShells

A lot of folks introduce Nix flakes via the devShells more akin to a replacement for the asdf version manager, a way to get specific versions of tooling into your environment or $PATH. That can be a good place to start, however, when a project is fully beflaked with Nix, there is less need to run anything in the mutable environment of devShells rather than the immutable environment of Nix derivations. Where devShell usage can help is is when working with tools outside your project’s environment. Let’s say we are using our system’s version of (Neo)Vim to edit plugin because it’s been fully modfied for our needs, & we’d like more immediate feedback about our linting errors from vint. In this scenario we would want our editor to have access to vint for a plugin to execute displaying warnings automatically. Because we don’t want version conflicts between nix flake check & the version of vint our editor is using, we can get put the exact same version onto our path for our editor to hook into.

{
  outputs = { self, nixpkgs, ... }@inputs:
    {
      # …

      devShells = forAllSystems (system:
        let pkgs = nixpkgsFor.${system}; in {
          default = pkgs.mkShell {
            buildInputs = with pkgs; [
              vint
            ];
          };
        });
    };
};

Now, if we enter a project, we can run

$ nix develop

(or use flake from direnv to do this automatically). Inside this shell, our editor to execute can execute vint (or your language server, or whatever tool you need) at the same version as specified by our Nix Flake inputs for Nixpkgs.

Distributing software

Distributing an via overlays

Our vim-hello-world is working as expected, but we want others to be able to use it too! Overlays are an a easy way for other Nix enjoyers to have our software directly from the flake.nix in our project root, often fetched from a repository, which will modify their nixpkgs. To make using our plugin as painless as possible, all we need to do is append our plugin to the user’s Nixpkgs’s vimPlugins attrset which they can use in their personal Vim setup. You can use a similar method to append or override any sofware in the user’s environment.

{
  outputs = { self, nixpkgs, ... }@inputs:
    let
      # …
    in
    {
      overlays = {
        # final: the output of Nixpkgs after being lazily evaluated
        # prev: the input of previous Nixpkgs
        vimPlugins = final: prev: {
          vimPlugins = prev.vimPlugins // {
            ${name} = self.packages.${prev.system}.vimPlugins.${name};
          };
        };
      };
    };
}

Now users can import our Flake into their own via Input!

{
  input = {
    # …
    vim-hello-world = {
      url = "protocol://repository_or_tarball";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, ... }@inputs: {
    # …wherever you do your overlays…
    overlays = [
      inputs.vim-hello-world.overlays.vimPlugins
    ];
  };
}

Distributing as a tarball

Unfortunately, not everyone is using Nix or Nix+flakes for their text editor management so we’ll need an alternative (besides many would prefer upstreamed packages in Nixpkgs to overlays). While it may be common to just use the whole version control repository for editor plugins (and in many cases this is simpler/recommended for maintenance), it may not be the best experience for your users. A basic, non-trivial plugin repository is likely going to contain files the user will never interact with such as settings files, CI & other build scripts, images & other media for HTML documentation, source files for compilation to Lua in the case of Neovim, as well as the entire repositories history. Rather than telling users “just go install Fennel, append its compiler to your $PATH, & add this build step command to your plugin manager’s config & it should work”, we could instead build it, strip it, tree-shake it, minify it, etc. for the user. To do this, it would be time to break out into a ‘real’ Nix derivation—which is out of scope since it’s not specific to flakes, but the basic idea would be…

{
  outputs = { self, nixpkgs, ... }@inputs:
    let
      # …
    in
    {
      packages = forAllSystems (system:
        let pkgs = nixpkgsFor."${system}"; in {
          # The ‘kind’ of derivation can vary depending on language &
          # complexity, but all boil down to mkDerivation. The
          # `trivial-builders` can get you pretty far in many cases
          default = pkgs.mkDerivation {
            pname = "vim-hello-world";
            version = "0.0.1";
            src = ./.;
            # …& add some buildPhase, etc.
          };
        });
    };
}

We could build our package like before locally or in CI, tar the ./result folder (following links!), & push a build artifact to wherever you plan to host tarballs for users.

View our whole Flake for vim-hello-world
{
 description = "A Hello World demo plugin for Vim";

 outputs = { self, nixpkgs, ... }@inputs:
   let
     forAllSystems = nixpkgs.lib.genAttrs nixpkgs.lib.platforms.unix;

     nixpkgsFor = forAllSystems (system: import nixpkgs {
       inherit system;
       config = { };
       overlays = [ ];
     });
   in
   {
     overlays = {
       vimPlugins = final: prev: {
         vimPlugins = prev.vimPlugins // { ${name} = self.packages.${prev.system}.${name}; };
       };
     };

     packages = forAllSystems (system:
       let pkgs = nixpkgsFor."${system}"; in {
         default = pkgs.vimUtils.buildVimPlugin {
           name = "hello-world";
           namePrefix = "vim-";
           src = ./.;
           meta.license = pkgs.lib.licenses.isc;
         };

         vim-mod = pkgs.vim_configurable.customize {
           name = "vim";
           vimrcConfig = {
             packages.myplugins = with pkgs.vimPlugins; {
               start = [
                 # list plugins we need to be compatible with such as:
                 # ale
                 # powerline
                 # …but also include our plugin
                 self.packages."${system}".default
               ];
               # optional plugins to not be automatically loaded
               opt = [ ];
             };
             customRC = ''
               " A VimL RC file for our default settings for testing

               syntax on

               " Maybe we need flags for our plugin too here
             '';
           };
         };
       });

     apps = forAllSystems (system:
       let pkgs = nixpkgsFor.${system}; in {
         default = self.apps.${system}.vim-mod;

         vim-mod = {
           type = "app";
           program = "${self.packages.${system}.vim-mod}/bin/vim";
         };
       });

     checks = forAllSystems (system:
       let pkgs = nixpkgsFor.${system}; in {
         vint = pkgs.runCommand "vint-lint"
           {
             src = ./.;
             nativeBuildInputs = with pkgs; [ vim-vint ];
           } ''
             set -euo pipefail
             vint --style-problem $src/**/*.vim | tee $out
           '';
       });

     devShells = forAllSystems (system:
       let pkgs = nixpkgsFor.${system}; in {
         default = pkgs.mkShell {
           buildInputs = with pkgs; [
             vint
           ];
         };
       });
   };
}
Tip

If you get stuck, consider checking out the REPL documentation to help debug any issue. You would start by loading the flake & then inspecting around something like:

$ nix repl
Welcome to Nix 2.17.0. Type :? for help.

nix-repl> :l flake.nix

Where to go from here?

In touring the basic parts of the Nix flake API we can see with this one tool, we can artictect & orchestrate our entire build setup + distribution + testing. All of these actions have been done in a stateless manner too without relying on the system’s packaging state which affords us certain guarantees about our software. Each part of the flake API is ready for extending from the simple to the complex & it can be done piecemeal.

A demo repository has been set up to look at if you want to read through its history.

Exercises for reader if you would like to continue hacking on Vim plugins

  • Set up nix flake check with a Vim testing framework like vader.vim.
  • Use various Nixpkgs pinned version to test older versions of Vim
  • Set up a Neovim configuration + nvim-mod to test our vim-hello-world VimL plugin in both editors
  • Try a lua/ folder & write a basic print("Hello, world!") from Neovim
  • Build a Neovim plugin in a different language (could be Haskell, OCaml, Rust, JavaScript, Python, et al.) or try a compile-to-Lua options, like Fennel or LunarML or MoonScript, to build Lua.