RFD: Modular Dynamicism in NixOS

Introduction

Nix is a structured system configuration language designed to reproducibly implement system configurations; and is often paired with solutions such as Colmena¹ or Morph², which provide the capability to completely and reproducibly control the remote software running across a cluster of machines by evaluating a configuration, building packages and their relevant configuration data, and then deploying them to a collection of target machines. This paradigm has seen successful industry adoption³⁴⁵.

This paradigm of using Nix -- in which a static configuration is generated and then deployed to specified physical machines -- is often less applicable to companies which utilize "scalable" or "elastic" cloud paradigms. These paradigms vary the computing resources available -- such as number of hosts, allocated (virtual) CPU-time, and RAM allocation -- in order to efficiently match resource utilization to the current computational demand⁶. Unfortunately, the paradigms currently used by Nix tooling result in primarily static configurations which much be re-created and re-deployed ¹ in order to scale to changes in resource availability, leading many users (e.g. ⁷) to believe Nix is a competitor to container orchestration platforms, rather than a natural complement.

In this work, we propose a mechanism for adding structured dynamicism to Nix, which is designed to allow runtime reconfiguration without the significant computational load and service interruption associated with 'traditional' Nix re-deployment, and without compromising the core Nix values of reproducibility and rapid change reversion. To accomplish this, we propose a mechanism for selective re-evaluation and activation which modifies only the pieces of running host necessary to quickly and efficiently effect a configuration change.

Definitions

Applying a NixOS configuration goes through multiple stages. For our purposes, these are:

• Evaluation (eval), during which the Nix interpreter evaluates Nix language source code, turning a web of NixOS modules into derivations for the system
  • This is what you can observe in the REPL
  • nixos-rebuild dry-build
  • nixos-rebuild repl
  • nix eval ".#nixosConfigurations.$HOSTNAME.config.system.build.toplevel.outPath"
• Build, during which the Nix daemon runs derivation builders, which output collections of symlinks, generated config files, activation script files, etc
  • This is when files and directories are organized into the tree for the system configuration
  • nixos-rebuild build
  • nix build ".#nixosConfigurations.$HOSTNAME.config.system.build.toplevel"
• Activation, during which a built configuration is applied via activation scripts
  • This is when /run/current-system changes, when systemd units are reloaded, when agenix sets up /run/agenix, etc
  • nixos-rebuild switch, nixos-rebuild test
  • ./result/activate
• Runtime, when the NixOS configuration has been applied and the system is running normally

In this document these will be respectively called eval time, build time, activation time, and runtime.

Motivation

A NixOS system configuration is a monolith. This is kind of good! It's how we get such reproducibility, and it means you can introspect a lot about the configuration just by looking at the config.system.build.toplevel derivation. But it also means eval times are long (9.9 seconds for a small desktop system closure on a high-end Ryzen 7950X), and the smallest change must rebuild the entire system-path (/run/current-system/sw). Tweaking a single value in a service's config explodes into rebuild the entire systemd service tree, the entire /etc/static tree, etc.

Modern scalable/elastic environments -- such as those driven by Kubernetes⁶ -- often leverage a set of programmable rules to attempt to automatically scale hardware resources to match current computational demand. These systems often attempt to leverage performance predictors (such as CPU and memory utilization) in order to rapidly address spikes and drops in demand, scaling computational resources in order to minimize unused ('wasted') computing capability without suffering performance issues in the event of unexpected load.

These dynamic computing environments require a rapid ability to respond to performance predictors⁸; in many cases, these systems have a goal of adapting to predicted load before their services' users can notice a performance impact. These types of dynamic mutations are not well-supported by Nix; both due to Nix's tendency towards long evaluation times, which dramatically reduce the efficiency of feedback systems, and due to Nix's limited capability for programmatic mutations.

There currently exist only very limited facilities to allow mutation from Nix; and these facilities are currently poorly-matched for scalable environments, instead targeting specific problems such as keeping secret data out of the Nix store. Accordingly, the few systems that do provide mutability are are either highly domain specific (such as agenix ⁹), or merely capable of allowing runtime dynamicism without ensuring reproducibility or traceability (such as ¹⁰ /system.etc.overlay.mutable). It is not currently possible to, for example, reactively change the configured threads available to a web service in response to current load.

This RFD proposes a modular solution to dynamically and programmatically modify the effective NixOS configuration of a running system without sacrificing Nix's core benefits of reproducibility and introspectability.

Proposal

In order to address the described need, we propose a solution based in three core tenets:

1. The tightly-coupled monolith that is a single system configuration (often interacted with as a 'NixOS generation') should be broken into more composable configuration pieces. When configuration pieces are less tightly coupled to the final system configuration, changes can be made that do not require significant re-evaluation times, as described below.
2. Simple reconfigurations should be easily appended as small/trivial Nix expressions, which allow both programmatic manipulation of the running environment, and which still generate activation profiles allowing generation-based rollback.
3. Configuration changes should be able to be applied without evaluating the whole system closure, in order to provide scalable systems the ability to quickly respond to changes in environment/load.

This section describes our proposal for each of these tenets; the following section provides benchmark evidence substantiating this section's rationales.

Partitioning the Configuration 'Monolith'

Currently, evaluating and entire system configuration (for example, by evaluating a node's config.system.build.toplevel) requires evaluating all defined NixOS options and forcing the value of all values referenced by the top-level configuration (i.e. all attrs directly referenced in .toplevel). Further, several operations commonly performed to generate final system and application configurations result in the forced serialization of a set of attributes -- for example, by generating a text configuration representation of the provided set of attributes; such as when generating a systemd service file or an application's TOML configuration. Together, these results in significant evaluation times when attempting to build a deployable system configuration; but evaluation of configuration subtrees -- such as individual options -- can require significantly less time.

In general, the smaller the evaluated subtree, and fewer data transformations entailed, the faster the evaluation is performed. As a result, a set of service configuration attributes (a Nix "attrset") with no external references tends to evaluate fairly quickly.

Consider the following example evaluations:

• Evaluating config.boot.kernel.sysctl is fast, but config.environment.etc."sysctl.d/60-nixos.conf".text is much slower, as it requires gathering and forcing options from other configuration locations; resulting in a large evaluated subtree even prior to text serialization.
• Evaluating config.fileSystems is fast, but config.environment.etc.fstab.text is much slower, as all referenced attributes in the system's filesystem configuration must be forced and serialized into a final text fstab.
• Evaluating config.services.openssh.settings is fast, but serializing it into a final text form is config.systemd.services.sshd.runner.text is much slower. When this is used by further configuration, such as config.systemd.units."sshd.service".text, the evaluation times compound as the higher subtree is evaluated.

NixOS options are used to model service and application settings -- typically as compositions of native Nix types -- and these options can be evaluated individually without pulling in the rest of the system. This makes all the difference: on our reference machine, it 10 seconds for to evaluate a system closure, but only 500 milliseconds to evaluate the structured Nix values contained in config.nix.settings.

From this, we can observe that the NixOS module system is not inherently slow; but realization of final outputs after referencing many tightly-coupled options is; as are the many file format serializers provided by Nixpkgs (lib.generators/pkgs.formats) and written in the Nix language. As a result, with proper structuring, we can cheaply re-evaluate specified parts of a system configuration, especially if we keep our options structured (as opposed to serialized) and local.

Append-based Reconfiguration

Current scalable systems, such as Kubernetes'-based container orchestrations⁶, are capable of directly modifying configuration parameters of their target nodes at runtime. These executions are fast, but sacrifice both the formal introspectability of using a structured language such as Nix, and the seamless rollback and redeployment endemic to Nix's reproducibility¹¹.

In our system, we propose implementing append-based reconfiguration, where individual segments of trivial Nix configuration are appended to an existing configuration. For example, to adjust the number of threads used by a theoretical server, we might adjust the value of its server_threads configuration parameter using the following appended Nix module:

{...}: 
{
  # Replace the existing number of server threads by 24.
  #
  # The (-1) here indicates the priority of our change; a subsequent addition
  # might use the value (-2) to take priority over this appended change; or,
  # when easily identified, we might remove this append in order to add another
  # modification to the same parameter.
  services.some-web-service.settings.server_threads = lib.mkOverride (-1) 24;
}

The Nixpkgs/NixOS module system already provides the capability of programmatically overwriting previous values. In the service module definition, parameters for NixOS services are typically written using an override system, which allows individual assignments to assign arbitrary priorities to any value assigned to an option. Accordingly, inside the definition of our theoretical web service's module, we might see:

  config.services.some-web-service.settings.server_threads = lib.mkDefault 2; 

Here, lib.mkDefault is a helper function that provides a default, low-priority value. The following code is directly equivalent:

  # This is equivalent to mkDefault; the priority 1000 is a reasonably low priority;
  # with lower integers (including negatives) accepted over higher ones.
  config.services.some-web-service.settings.server_threads = lib.mkOverride (1000) 2;
  
  # The nixpkgs library also provides lib.mkForce, which is equivalent to using lib.mkOverride
  # with a given priority of 50. The value "100" is used as a default priority.

As our new priority is lower than any existing priority, appending it to the list of modules included in any configuration will result in the final value of config.services.some-web-server.server_threads evaluating to our new value of 24. This is a core strength of the Nixpkgs system; as use of this mechanism turns out to be extremely inexpensive: appending configurations with hundreds of overrides has a negligible impact on evaluation time; well under the measurement's noise-floor. Even the application of millions of overrides has a minimal impact on the option's evaluation time; with evaluation times increasing by only 10x.

We accordingly can re-compute the configuration parameters used by a specific service quickly; enabling the method of rapid parameter adjustment mentioned in the next section. By performing a full system evaluation after this adjustment is applied, we continue to provide the traditional generational-rollback and reproducible deployment benefits of a Nix system, without incurring the latency of a full evaluation.

Rapid, dynamic reconfiguration

The final piece of our proposed mechanism is designed to allow rapid reconfiguration, allowing a node to adapt to newly-available 'elastic' computing resources without waiting for a full evaluation and Nix configuration. To enable this, we modify the NixOS module system to mark each dynamically updatable parameter with a special annotation that indicates it should be runtime-modifiable.

As an example, let us consider our theoretical some-web-service. Let us assume that, like Mastodon¹² and many other applications, our server accepts the number of service threads it should run from the environment variable SERVICE_THREADS. A NixOS module for our service might thus be written to "pass in" this value via the environment section of its systemd service.

  # Here, we're focusing only on the definition of the parameter we care about.
  #
  # In reality, these are often found as attributes in long service definitions, such as the definition
  # for the systemd service for the Mastodon web frontend: 
  #
  # https://github.com/NixOS/nixpkgs/blob/8bb5646e0bed5dbd3ab08c7a7cc15b75ab4e1d0f
  # /nixos/modules/services/web-apps/mastodon.nix#L1009
  #
  # This settings attrset contains the `service_threads` value that we want to update.
  systemd.services.some-web-service.environment = 
      config.services.some-web-service.settings;

When services are generated, the NixOS systemd service module takes the contents of the environment attribute, resolves it, and then serializes it into the relevant systemd service file. While the actual mechanism is complex due to the variety of keys found in a systemd service file, for our purposes, it can be represented by the following transform:

    # While generating the text of the systemd unit file, the environment is coalesced into a set of
    # `Environment=<key>=<value>` entries by concat'ing values together. We'll simplify this transform
    # down to a fictional lib.mkSystemdEnvironment.
    #
    # You can check out the -actual- code that would be equivalent to our fictional function here:
    # https://github.com/NixOS/nixpkgs/blob/4c104bbc3613937f2ff6eb277b25c5b883b1223d/nixos/lib/systemd-lib.nix#L775
    #
    
    let
      serviceTextBefore = "<ommitting all non-environment service text for brevity>";
      serviceTextAfter  = "<ommitting all non-environment service text for brevity>";
    in
    systemd.units."some-web-service.service".text = ''
        ${serviceTextBefore}
        
        ${lib.mkSystemdEnvironment config.settings.some-web-service.settings}
        
        ${serviceTextAfter}
    '';

In our proposed model, we might add a special annotation to our web service's option definition, indicating that modification of our relevant value will result in a needed modification to its systemd service file.

  # Note: the exact format and mechanism for these are TBD; and not the focus of this RFD.
  # We expect them to significantly vary as we make new discoveries during implementation.
  #
  config.dynamicism.some-web-service = {
      # This parameter might specify which settings would need to change in order to result in a dynamic change.
      source-parameters = config.services.some-web-service.settings;  
      
      # This parameter might be one of many that indicate what changes need to be performed dynamically;
      # here, we indicate that we need to update the some-web-service unit file.
      systemd-services-updated = ["some-web-service.service"];
  };

When the value is changed, we might walk the config.dynamicism sections, and compare the provided parameters in the new configuration from the values in the current-generation's configuration. If they differ, we know that we need to re-generate the some-web-service.service systemd unit file, push the new unit file to the target node, and then apply it.

Under this model, in place of a re-evaluation of the entire system, we now have to perform only three steps in order to apply our new changes:

1. Re-evaluate systemd.units."some-web-service".text, yielding a derivation that produces a single new systemd unit file, and evaluating only the smaller subtree (and single serializer) used to specify the unit file.
2. Push the new derivation to the target node, e.g. by using nix-copy-closure or nix copy.
3. Instruct systemd to load the new unit definition, e.g. by loading this as a runtime/transient unit.

Once this rapid change is made, the coordinator may evaluate and build a new system generation, which can be pushed to the node in the background without immediate activation.

Programmatic Use

Eventually, we hope to combine all elements of our proposed system into one easily accessible from a "scalable cloud" control plane; so the supervisor might be able to orchestrate Nix-based configuration updates, as in the following pseudocode:

utilization = self.cpu_utilization()

if utilization > CPU_UTILIZATION_HIGH_THRESHOLD:
    
    # Double the amount of vCPUs...
    node.add_vcpus(node.current_cpus())
    
    # ... and update our web service to take advantage of those new CPUs.
    node.update_nix_parameter(f"services.some-web-service.settings.server_threads", node.current_cpus())
    
elif utilization < CPU_UTILIZATION_LOW_THRESHOLD:

    # Halve the amount of vCPUs...
    node.remove_vcpus(node.current_cpus() // 2)
    
    # ... and update our web service to take advantage of those new CPUs.
    node.update_nix_parameter(f"services.some-web-service.settings.server_threads", node.current_cpus())
    

Benchmarks using NixOS Tests

Nixpkgs' NixOS tests themselves use NixOS configurations, which we can query like any other NixOS configuration to get a minimal but reasonable example of usage of these services. The benchmarks below are performed with hyperfine --shell=none, on an AMD Ryzen 9 9050X, and using nix-instantiate --quiet --eval --no-eval-cache --json --strict.

Jenkins

Options are relative to nixpkgs#nixosTests.jenkins.nodes.master.

NixOS Option	Eval Time
system.build.toplevel.outPath	3.257 s
systemd.units."jenkins.service".text	773.3 ms
services.jenkins.jobBuilder	401.0 ms

Grafana

Options are relative to nixpkgs#nixosTests.grafana.provision.nodes.provisionNix. I've split Grafana's settings options into two benchmarks: one with all the option values, and one with all the option values except exactly one: services.grafana.settings.paths.provisioning. The default value for that option is a derivation that depends on multiple other values in services.grafana.settings.*, and uses pkgs.formats.yaml. That single option adds 512 ms, a 42% increase!

NixOS Option	Mean Eval Time
system.build.toplevel.outPath	2.224 s
systemd.units."grafana.service".text	1.320 s
services.grafana.settings w/o drv	718.0 ms
services.grafana.settings w/ drv	1.230 s

GotoSocial

Options are relative to nixpkgs#nixosTests.gotosocial.nodes.machine.

NixOS Option	Mean Eval Time
system.build.toplevel.outPath	1.704 s
systemd.units."gotosocial.service".text	880.7 ms
services.gotosocial.settings	445.3 ms

Alternatives

As we've researched the best way to achieve dynamicism, several other approaches have been considered; however, these currently are not believed to meet our goals of reproducibility, inspectability, and rapid deployment.

Approaches considered, but ultimately discarded, include:

• Writing a CLI tool with a Nix language parser to manipulate option values in source text directly.
• Moving all dynamicism to runtime, taking activation time out of the equation.
• Use classic monolithic NixOS configurations and simply continuously re-deploy.
• Using other Nix features such as impure derivations.

Citations

---

1. Colmena, a Nix-based deployment system written in rust: https://github.com/zhaofengli/colmena ↩︎

2. Morph, a tool to deploy Nix configurations to remote servers: https://github.com/DBCDK/morph ↩︎

3. Shopify, a major company using Nix for backend deployment: https://shopify.engineering/what-is-nix ↩︎

4. Mercury, a financial institution, uses Nix for backend orchestration: https://serokell.io/blog/haskell-in-production-mercury ↩︎

5. Tweag have provided similar solutions to many clients using Nix: https://www.tweag.io/group/nix/ ↩︎

6. Kubernetes, overview: https://kubernetes.io/docs/concepts/overview/ ↩︎

7. https://news.ycombinator.com/item?id=22858558 ↩︎

8. Control system stability theory; as exemplified by considering response delay as loss of phase margin: https://en.wikipedia.org/wiki/Phase_margin ↩︎

9. Agenix, a secret manager in Nix: https://github.com/ryantm/agenix ↩︎

10. https://mynixos.com/nixpkgs/option/system.etc.overlay.mutable ↩︎

11. Users with complex Kubernetes stacks will likely object that tools exist to make "introspection" and "rollback" possible; but these rely on an online, authenticated server connection to etcd for introspection and use the underlying version control software for rollback. These neither provide the granular introspectibility offered by Nix, nor Nix's ability concretely re-construct a system's state during rollback. ↩︎

12. Mastodon uses a simple environment variable present in its service to adjust the number of web-server processes and threads: https://github.com/NixOS/nixpkgs/blob/nixos-25.11/nixos/modules/services/web-apps/mastodon.nix#L28 ↩︎