Introducing Secrix

Here at Platonic.Systems, all of our systems out in the wild are NixOS, and we deploy them with a single tool. Previously, we relied on agenix for secrets management, which sufficed for our needs. Unfortunately, agenix became cumbersome due to requiring an extra file dedicated to listing its secrets alongside the system configuration declarations. This duplication was rather annoying and unneeded, so I set out to streamline the process by creating a library on top of agenix that would build the secrets.nix file on the fly. This module was called xinega and turned out to be harder than expected—until my coworker John Bargman pointed out that agenix wasn’t needed at all, and that I could use age (or, in this case, rage) directly to encrypt and decrypt secrets. This idea piqued my interest, so off I went to build a new module sans agenix.

A New Way of Looking at the Secrets Management Problem

In the past, we’d used Nixops and its internal secret management mechanism for system deployments. This worked well for quite a while. Eventually, we ended up moving off of Nixops to Nixinate, which is working for us while we continue to develop and finalize our in-house solution for deployments. No, we don’t suffer from Not-Invented-Here Syndrome; we simply want to push the boundaries of Nix. Nixinate doesn’t have a mechanism for managing secrets, so we ended up with agenix.

Agenix is great at managing secrets until a system restart is needed or a wild power outage strikes—in which case, by default, you’ll need to redeploy to get your secrets back. This situation is not ideal, because no matter how much time I’ve spent trying to control the entire universe so I can predict exactly when my systems will end up restarting, I have thus far been unable. The encrypted secrets, however, are stored in the nix store. This is sufficient enough for me to decrypt them when my system starts again; however, agenix doesn’t do this. Additionally, services specifically rely upon many of our secrets, so we’d have to make those services dependent upon the secrets existing. This created a lot of repetition, so why not simply tie the secrets to the services in the module itself?

Upon this realization, I began playing video games, because who wants to work on a Saturday? Approximately 5 minutes into playing, I instead began working on secrix because I apparently want to work on a Saturday. Please send help.

Making Secrets Management Easy

Tying secrets to services actually allows us to do a few things for free, such as make assumptions about the owner of the files and bind the keys to the services via systemd. Interestingly enough, this arrangement also allows us to assign lifetimes to the decrypted secrets, as systemd has directives specific to execution environments, such as RuntimeDirectory. As we can see in the RuntimeDirectory= documentation:

In case of RuntimeDirectory= the innermost subdirectories are removed when the unit is stopped.

If there are no inner directories, the directory itself will be removed, causing the secrets to disappear with it. Beyond this, the RuntimeDirectory is in /run, which is a tmpfs, meaning that the secrets live in virtual memory. This means that even when an unexpected event occurs, the files still disappear, but the secrets never appear in a decrypted state on disk.

Decrypting and encrypting files merely entails storing pertinent decryption/encryption information in the system configuration, so no duplication is needed. In fact, for many of our services, adding a single secret simply means adding two lines to the configuration and being done with it.

The core essence of secrix remains that exact idea: being able to add secrets to our systems easily and tie them to services while keeping their lifetimes only as long as absolutely necessary. That said, not all secrets can or should exist for a determined amount of time, nor should they have to be tied to a service, so I added the concept of system secrets. System secrets are secrets that get decrypted when the system starts and go away when the system shuts down. As of right now, these secrets all get decrypted and removed together, so there is no fine control over the lifetimes of any given secret. That is the intended functionality for a future release.

Keeping Decrypted Information Out of the Nix Store

Agenix provided secrets to the system; however, keeping them out of the nix store sometimes involved a lot of tom-foolery that included substituting the secret into some other file via bash and, often, directly modifying the systemd service definition itself. To avoid this, I added a secret builder option to secrix that allows you to manipulate the secret however you’d like (still with bash) before placing it in its final location. This doesn’t necessarily remove all the finagling related to keeping secrets out of the store, but it removes a great deal.

A Note on Permissions

Upon release, service secrets will be stored in a directory that has the permissions 0755 unless RuntimeDirectoryMode= happens to be defined by the service the secrets are bound to. In almost all cases, this is unremarkable. What is remarkable, however, are the system secrets. System secrets are not bound to an existing service and, thus, reside in a directory with a different set of permissions: 0111. This arrangement was specifically chosen so that no user, including the owner of the directory, can view the directory contents but can still access files owned within it. This is worth noting as I can see confusion happening when someone is unable to list out the files in the directory. These default permissions can be changed with secrix.system.secretsDir.permissions. Soon, in an upcoming release, changes to service secret directory permissions will be allowed.

Managing Secrets

Managing the encryption of secrets happens as a flake app, meaning there is no need to add a shell to work with secrets. To see how to use the command line, you can simply use nix run .#secrix -- -h or even simply nix run .#secrix. Unfortunately at the current moment, secrix cannot infer which users or hosts will be used for which secret, so those must be specified manually. It also cannot infer the private key to use for decryption, so that will have to be specified manually as well when rekeying or editing. I am looking into fixing these issues, and they will hopefully be addressed soon.

Current Status

Secrix is currently at 0.9.0 until I’ve knocked out the last few hurdles. If you wish to contribute, please feel free. The source code is here. Despite its beta status, we currently use it in production, and it’s just worked so far.

Special Thanks

Special thanks to John Bargman and David Lyon at Platonic.Systems for helping me finalize this library for release.