Use "baonix" to read secrets (test workloads only) #362

Open
raito wants to merge 4 commits from consume-machine-identity-secrets into main
Owner

The aim here is to have an end-to-end first use of workload identity to retrieve secrets and make them available to services.

The first production target are monitoring and backups which are easy and global to every machines that will come in another PR.

Notable differences with agenix:

  • No support for per-generation secrets, i.e. no /run/agenix.d/18/...
  • No support for symlink secrets
  • No support for Darwin
  • No support for home-manager secrets
  • Obvious dependency on networking and spire-agent.service — introduce a secret at that level and the system will not like it.

Planned features:

  • Abstraction to retrieve classical type of secrets, possibly in the form of Golang templating à la OpenBao agent.
  • Automatic reloading of secrets (not sure yet how I plan to do that.)
  • Automatic policy debugging, not sure what policy do you need? systemctl set-environment DEBUG_BAONIX_POLICIES=1 and restart and the required policies will be printed for every secret.
  • Pre-flight check on whether we can access a secret and read it before attempting to read.
  • Partial retrieval of secrets: do not fail totally if you cannot fetch all the secrets.
  • Per-generation secrets if we need them.
  • Symlink secrets if we need them.
  • Darwin support.
  • Initrd-level secrets if we need them (basically: run baonix in initrd + node attest the initrd rather than the stage 2).

Depends on #360.

The aim here is to have an end-to-end first use of workload identity to retrieve secrets and make them available to services. The first production target are monitoring and backups which are easy and global to every machines that will come in another PR. Notable differences with agenix: - No support for per-generation secrets, i.e. no `/run/agenix.d/18/...` - No support for symlink secrets - No support for Darwin - No support for home-manager secrets - Obvious dependency on networking and `spire-agent.service` — introduce a secret at that level and the system will not like it. Planned features: - Abstraction to retrieve classical type of secrets, possibly in the form of Golang templating à la OpenBao agent. - Automatic reloading of secrets (not sure yet how I plan to do that.) - Automatic policy debugging, not sure what policy do you need? `systemctl set-environment DEBUG_BAONIX_POLICIES=1` and restart and the required policies will be printed for every secret. - Pre-flight check on whether we can access a secret and read it before attempting to read. - Partial retrieval of secrets: do not fail totally if you cannot fetch all the secrets. - Per-generation secrets if we need them. - Symlink secrets if we need them. - Darwin support. - Initrd-level secrets if we need them (basically: run baonix in initrd + node attest the initrd rather than the stage 2). ~~Depends on #360.~~
We use this piece of software to let OpenBao register a new OIDC method
for login which will be used by machines.

Signed-off-by: Raito Bezarius <raito@afnix.fr>
Signed-off-by: Raito Bezarius <raito@afnix.fr>
This initialize a simple NGINX based SNI module.

Taken from @thubrecht's DGNum infrastructure repository.

Signed-off-by: Raito Bezarius <raito@afnix.fr>
Signed-off-by: Raito Bezarius <raito@afnix.fr>
Signed-off-by: Raito Bezarius <raito@afnix.fr>
Required now that we are running a SPIRE server for the WebPKI trust
bootstrap and an OIDC discovery provider for OpenBao.

Signed-off-by: Raito Bezarius <raito@afnix.fr>
This activates OIDC federation with OVH SBG node.

Signed-off-by: Raito Bezarius <raito@afnix.fr>
An helper module abstraction to define machine identities policies in a
static fashion.

Signed-off-by: Raito Bezarius <raito@afnix.fr>
Signed-off-by: Raito Bezarius <raito@afnix.fr>
This initialize a variant of agenix "module compatible" that relies on
all the work we have done with machine identity.

Signed-off-by: Raito Bezarius <raito@afnix.fr>
raito force-pushed consume-machine-identity-secrets from ab981876a9 to f5b3259a98 2025-09-25 23:15:00 +00:00 Compare
raito force-pushed consume-machine-identity-secrets from f5b3259a98 to 6ed5c2c01a 2025-09-25 23:23:16 +00:00 Compare
raito force-pushed consume-machine-identity-secrets from 6ed5c2c01a to 84492f2a01 2025-09-26 00:09:20 +00:00 Compare
raito force-pushed consume-machine-identity-secrets from 84492f2a01 to d55bf1397f 2025-09-26 02:58:32 +00:00 Compare
raito force-pushed consume-machine-identity-secrets from d55bf1397f to 1f732a1f91 2025-09-26 17:02:06 +00:00 Compare
@ -0,0 +101,4 @@
example = [ "ovh-sbg-hv01-sts-jwt" "ovh-lim-hv01-sts-jwt" ];
description = "What is the authentication path for the JWT method?";
};
baoRole = mkOption {
Author
Owner

I wish this wouldn't be necessary to specify as this is automatically calculated on the Vault side…

I wish this wouldn't be necessary to specify as this is automatically calculated on the Vault side…
@ -0,0 +96,4 @@
default = "https://vault.afnix.fr";
description = "Address to the OpenBao instance";
};
baoAuthPath = mkOption {
Author
Owner

Also would be nice not to specify that.

Also would be nice not to specify that.
@ -0,0 +91,4 @@
Where secrets are created.
'';
};
baoAddress = mkOption {
Author
Owner

Perhaps putting all bao related options under an bao tree would be good.

Perhaps putting all bao related options under an bao tree would be good.
@ -0,0 +108,4 @@
};
secrets = mkOption {
type = types.attrsOf secretType;
default = { };
Author
Owner

An example would be nice here.

An example would be nice here.
@ -0,0 +11,4 @@
{ lib, config, ... }:
let
inherit (lib) mkIf concatMapAttrs nameValuePair listToAttrs;
trustDomain = config.networking.fqdn;
Author
Owner

This could be grabbed directly from the SPIRE module.

This could be grabbed directly from the SPIRE module.
@ -0,0 +23,4 @@
# NOTE(Raito): it's absolutely crucial this keeps being synchronized
# with Vault policies and their naming scheme.
# Any divergence will surface as failure to obtain a JWT-SVID.
spiffeID = "spiffe://${trustDomain}/tenants/${tenant}/svc/${svc}";
Author
Owner

This could be put behind a common library file that generates spiffe ID names, avoiding the divergence.

This could be put behind a common library file that generates spiffe ID names, avoiding the divergence.
@ -108,11 +107,21 @@ in
${concatStringsSep "\n" (mapAttrsToList (name: upstream: "${name} ${upstream};") cfg.redirects)}
}
log_format basic '$remote_addr [$time_local] '
Author
Owner

@thubrecht if you're interested to take back these changes or criticize them :-)

@thubrecht if you're interested to take back these changes or criticize them :-)
@ -48,0 +48,4 @@
test = {
policy = {
"secret/data/afnix/test/*".capabilities = [
"create" "update" "read" "delete" "list"
Author
Owner

Should probably be "read "list"" only.

Should probably be `"read` "list"" only.
@ -42,0 +51,4 @@
# We want to verify that we can load test secrets.
baonix = {
audience = "afnix";
baoAuthPath = "jwt-ovh-sbg-hv01-sts-jwt";
Author
Owner

Ideally, we should drop the last -jwt, it's redundant, this needs to happen at the Vault level.

Ideally, we should drop the last `-jwt`, it's redundant, this needs to happen at the Vault level.
@ -41,1 +50,4 @@
# We want to verify that we can load test secrets.
baonix = {
audience = "afnix";
Author
Owner

Audience should be URIs. This needs to be fixed at the Vault level.

Audience should be URIs. This needs to be fixed at the Vault level.
raito changed title from WIP: use "baonix" to read secrets for monitoring and backups to Use "baonix" to read secrets for monitoring and backups 2025-09-26 17:07:58 +00:00
raito changed title from Use "baonix" to read secrets for monitoring and backups to Use "baonix" to read secrets (test workloads only) 2025-09-26 17:08:14 +00:00
raito force-pushed consume-machine-identity-secrets from 1f732a1f91 to 14afc4f914 2025-09-26 17:25:38 +00:00 Compare
@ -108,11 +107,21 @@ in
${concatStringsSep "\n" (mapAttrsToList (name: upstream: "${name} ${upstream};") cfg.redirects)}
}
log_format basic '$remote_addr [$time_local] '
Owner

I think it would be cool to log the hostname as well

I think it would be cool to log the hostname as well
This pull request has changes conflicting with the target branch.
  • services/baremetal/hypervisors/machine-identity/host.nix
  • services/default.nix
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin consume-machine-identity-secrets:consume-machine-identity-secrets
git switch consume-machine-identity-secrets

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff consume-machine-identity-secrets
git switch consume-machine-identity-secrets
git rebase main
git switch main
git merge --ff-only consume-machine-identity-secrets
git switch consume-machine-identity-secrets
git rebase main
git switch main
git merge --no-ff consume-machine-identity-secrets
git switch main
git merge --squash consume-machine-identity-secrets
git switch main
git merge --ff-only consume-machine-identity-secrets
git switch main
git merge consume-machine-identity-secrets
git push origin main
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: afnix/infra#362
No description provided.