evolive

evolive is a REST API for live evaluations of Nix expressions, geared towards nixpkgs-style repository.

It provides:

• a streaming endpoint to dispatch evaluation as we go
• cache evaluations and upload them to a S3 bucket for archival in Parquet files
  • "keep" flag as in Hydra that pushes the drvs into a store and the results into a parquet file. Otherwise, accept that the data is lost once the worker dies (todo)
  • persist machine/instance id (also first step towards provenance)
• a way to retrieve derivation recipes that were evaluated
• a way to stream the evaluation log for a given ongoing evaluation

Setup

Prerequisites

• An S3 endpoint with two buckets. When using Garage, the setup can be done as follows:

garage key create evolive
garage bucket create evolive-reports
garage bucket create evolive-drvs
garage bucket website --allow evolive-drvs
garage bucket allow --read --write --owner evolive-reports --key evolive
garage bucket allow --read --write --owner evolive-drvs --key evolive

Running

Start the dev server from the nix-shell like this:

fastapi run src/api/main.py

Implementation notes

Fetcher

Instead of using libfetchers, a custom implementation is used that mimics its behavior: that has the benefit that we can implement optimizations ourselves. For instance, the default behavior is that a shallow clone is made into a repository and the evaluation happens in a temporary directory that is a worktree of said checkout.

That has the benefit that additional operations like dumping the path into the store are not necessary and the temporary checkout is removed after evaluation.

S3 upload

Once instantiated, all derivations of a job are pushed into an S3 bucket in Nix's binary-cache format. For that, derivations and paths of a job's build-closure are serialized into a NAR, compressed with zstd and uploaded.

Implementation notes & design considerations

• The NAR streaming and compression is implemented in Python. For a large amount of jobs, this is significantly faster than regularly shelling out to Lix.

  To achieve that, evolive has its own minimal daemon client that supports a single operation, queryPathInfo that helps to compute the closure of a job and a generator that dumps store-paths into a NAR stream. evolive doesn't support other eval-stores as that'd slow down evaluation too much.

• The overall idea is to be able scale the evaluator up and down as much as needed. Hence, no persistent state should be on evolive's end. Also, copying from the local store would mean that we'd have to automatically distribute SSH keys for each evolive instance. Hence, a "cloud-native" storage solution is used for storing derivations.

• To avoid unnecessary uploads into S3, the following checks & caches are employed:

  • An evaluation gets a fingerprint that consists of the VCS revisions the fetchers got during fetch and a hash of the repositories and Nix & dictionary expressions used. If a fingerprint already exists, the evaluation isn't run.

  • Paths already existing aren't touched anymore.

• To keep the store consistent at all times (i.e. for each store-path its entire closure is also part of the store), store-paths are grouped into topological generations where each generation is uploaded concurrently.

Known problems

• The Git fetcher is missing full compatibility with libfetchers with regards to e.g. submodules. Also, it's not supported to have e.g. a correct revCount currently because the fetchers work with shallow clones. shallow must be explicitly turned off.