Xen support #387
No reviewers
Labels
No labels
bug
dependency
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
review-next
security
stub
tool
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: raito/lanzaboote#387
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "CertainLach/feat/xen"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
https://github.com/NixOS/nixpkgs/pull/324911 implements Xen hypervisor support in NixOS, this PR implements Xen support in lanzaboote when used together with mentioned PR.
It is impossible to have xen enabled for specialization, only primary system can have it, due to https://github.com/DeterminateSystems/bootspec/issues/147
Current implementation produces very large EFI binaries (>160M, prepare your /boot partitions), because Xen can't verify linux image by itself, and it needs to be included in Xen unified kernel image (https://xenbits.xenproject.org/docs/unstable/misc/efi.html), which only supports unpacked (vmlinux) kernels (as far as I can see, using any other format (vmlinuz/bzImage) results in "Not an ELF binary" error).
Because there is no path to vmlinux image present in boot.json, extra configuration is required:
boot.bootspec.extensions."org.xenproject.bootspec.v1".vmlinux = "${config.system.build.kernel.dev}/vmlinux";
Created xen boot entries are located lower than entries obtained from plain nixos UKI images, because I haven't figured how to provide systemd-boot with sort key.
Garbage collection doesn't work for /boot/loader/entries (I'm not sure how it is supposed to work).
Cc: @SigmaSquadron
This is wonderful; thank you!
I'm using the same
sort_keyin the systemd-boot builder. Due to the(with Xen Hypervisor)in parenthesis right next to the normal title, systemd-boot bumps it higher up in the list.I asked Ryan Lahfa in the Bootspec room when I was figuring out the stuff needed for the systemd-boot builder, and they mentioned that you can look into the
boot.jsonfor each specialisation, and while it looks like a normalboot.json, it doesn't have the specialisation extension, and the top-level arguments are specific to that specialisation. I imagined booting Xen in specialisations would also work, as if the specialisation is meant to be a Xen dom0, the specialisation-specificboot.jsonwould have Xen entries.I don't have "with Xen Hypervisor" here, sort-key was not enough, I have added version & renamed entry file name (== entry id) to make it sort correctly.
My primary system has xen enabled, and I have specialization with noxen tag, and it now orders entries exactly as with plain UKI images.
Lanzaboote uses top-level boot.json, where specializations are described using "org.nixos.specialisation.v1" key.
It is possible to look directly in specialization directories (/run/current-system/specialisation/X/boot.json), but it looks wrong, toplevel definition (/run/current-system/boot.json) should work too.
For reference, current
bootctl listoutput, where primary system has xen enabled, and noxen specialization has xen disabled. The ordering in bootctl list matches one in bootloader.Marking as ready for review, as I don't plan to change too much implementation-wise here.
I have experimented with adding hash validation to xen.efi, making it possible to not embed bzImage/ramdisk inside of UXI, but this is far from being done, and will go to upstream xen first, right now it will consume 40MB of /boot storage per generation.
Only toplevel system having xen sucks, but I believe it may need to be resolved in some more generic way, maybe it would be better to have xen installed the same way as memtest/netboot.xyz, instead of intercepting nixos entry generation, and then the solution will be the same as for #273?
I also want to hear bootspec people feedback on https://github.com/DeterminateSystems/bootspec/issues/147, and my proposed solution (Comment in the same issue).
Anyway, opening this PR for feedback on those topics, hoping some better solution might arise for those issues.
That was the original idea before the UKI shenanigans in https://github.com/NixOS/nixpkgs/pull/324911. This isn't an ideal solution, as it's expected that end users be able to roll back to a previous generation while keeping their Xen domains running.
No, I didn't meant it like that, I meant some generic way of producing per-entry
/boot/loader/entries/Xfiles with corresponding signed EFIs in lanzaboote. Currently, only xen generations can produce Boot Loader Specification Type #1 (.conf) files in lanzaboote.netboot.xyz/memtest don't need to be per-entry, but the solution might be generic enough to enable both.
I.e
configurationLimitfor non-nixos entries might allow to have 20 generations of nixos, having 5 generations of xen to reduce space consumption, and then 1 generation of memtest? Idk.How it may look in bootspec:
@CertainLach thanks for your work here! Is this a working example you posted? What would be needed to give this PR a test run with an existing
lanzabooteconfig?@gador The thing I posted is for the better, future implementation of the same functionality.
Xen itself will just work if you use lanzaboote built from this PR, no changes to NixOS config are required.
Awesome, thanks, Ill give it a try 👍
Rebased & matched behavior with nixpkgs implementation - it is now produces two boot entries per generation if xen is enabled.
CI failures seem to be unrelated, looking at the other PRs.
I also don't see a way to add test for this PR's changes, as it won't be possible to boot xen system in CI.
I finally had time to try and boot into this PR.
I still get the exact same error described by me here https://github.com/NixOS/nixpkgs/issues/350051
I have lanzaboote flake input pointed to
github:CertainLach/lanzaboote/feat/xen";andvirtualisation.xen.enable = true;and it still gives me the same error messages.I also now got
systemd-logind[2455]: Parsed PE file '/boot/EFI/Linux/xen-generation-395-4jk5grj5thec6zlqt5jfhpplmes6lsb5tdwyevfyx37hqfqgehcq.efi' is not a UKI.which is new.Hope this helps. Let me know if I can/should do more debugging
This is normal, as generated image is not a kernel image (K in UKI), instead it is booted from .conf entry in /boot/entries
Do you see new (With xen hypervisor) entry appearing in bootloader menu?
You got it. I didn't select the new generation on boot. I have xen enabled now, thanks!
Do you know how to set it as default? Usually, the new generations always get selected by default, but this time, it didn't?
Also, maybe unrelated, I got a kernel bug, which I've never had before:
(With xen support) entry should be on top, it isn't for you?
It is. It just isn't chosen as default (The second entry is selected by default)
After rebase, noticed that GC does no longer work, fixed that.
Latest commit might also change the ordering issue, though I haven't encountered that, probably due to customized loader.conf
Tried after your change: Now, no new entry is generated and also the old ones do not get removed
Entries generated using the old version need to be removed manually,
rm /boot/loader/entries/xen-* /boot/efi/Linux/xen-*
As for entries not generating - can you try to change your nixos configuration a little, it is possible it doesn't think that is is the new system generation for some reason.
Removing the entries manually did work, thanks.
I did change my config (.e.g I added the suggested kernel lines from https://github.com/NixOS/nixpkgs/issues/350051#issuecomment-2471604666 ) and a new generation is created and selected by default, but nothing with
xenonly the normal entry.And commands like
xldon't work, just as I get errors during boot likeFailed to insert module 'xenfs': No such deviceHuh, are you sure you install correct lanzaboote?
inputs.lanzaboote.url = "github:CertainLach/lanzaboote/feat/xen";
Do you have xen extension in your /run/current-system/boot.json?
arrg. Im dump. When I switched back to the original branch I commented your input and forgot to uncomment :-/
So, now I have the new xen generations and it didn't freeze up until now, so fingers crossed
can we rebase this branch to
0.4.2to work with current systemd? (or better yet, merge it) ;-)Rebased to latest lanzaboote master.
As for merging...
@blitz @RaitoBezarius sorry for pinging, but this is really unfortunate that Xen in NixOS doesn't work with lanzaboote, and this PR is required for that. I have failed to get anyone's attention in the matrix room, and I still want some feedback about my implementation and/or just to have this PR merged.
For general architecture guidance, we do not accept system-specific code in the systemd (and shared crate when possible, e.g. bootspec extensions).
To make progress on this PR, I'd like you to do the following things:
(1) Move the Xen specific code into a
xen/srcsubcrate inrust/tool, even if you need to copy or move "more" code fromsystemd/srcinto theshared/srccrate when it makes more sense.(2) Add a README in that Xen crate to explain that "Lanzaboote Xen backend" is maintained by your team and is not officially supported (beyond the fact that it's available here in this codebase as a matter of convenience).
The idea is the following, the lanzaboote maintainers officially maintain most of the codebase but we recognize that some folks want to support different backends out there:
etc.
Xen is yet one another special backend to me here. Feel free to ping me over Matrix for sync discussions if needed, my bandwidth is not great these days alas.
This is not a backend, it can be treated as another nixos boot configuration, xen replaces standard nixos linux kernel here.
In NixOS, you enable Xen (virtualisation.xen.enable), and nixpkgs's systemd-boot can just boot into it without any additional configuration. However, with lanzaboote, this doesn't work, as lanzaboote boots normal linux kernel instead.
Its implementation can be moved into another crate, but systemd-boot backend will have it as a dependency, I don't see how it is possible to decouple them.
If Xen was simply a different kernel, the modifications to the shared crate should directly go into NixOS and lanzaboote should not be aware of Xen as a different kernel. But from what I gather of this PR, it's more complicated than that.
On the top of that, lzbt-systemd is modified to support bespoke entries that seems also required in Xen case which are not necessary with standard partial kernel images.
We would like to keep lzbt-systemd standard and close to the concept of unified kernel images. If Xen cannot accommodate for this, it cannot mix in the lzbt-systemd backend. It can be a lzbt-systemd-xen backend FWIW.
It's fine if code gets copied at some point, but the idea here is to get the responsibilities & scope right for the different "backend" however you want to call it.
If you enable Xen + systemd-boot, you can easily turn on the lzbt-systemd-xen implementation in the upstream NixOS modules or anything. That's fine as well.
Linux: kernel + boot params get packed into UKI, UKI gets found by systemd-boot and then booted
Xen: kernel + boot params + xen efi + initrd get packed into UXKI (Unified Xen Kernel Image, https://xenbits.xenproject.org/docs/unstable/misc/efi.html), resulting in efi image, which can be booted by systemd, but won't be detected automatically as this is not a UKI, thus a dedicated type #1 entry file is required.
The problem is, enabling Xen should not disable NixOS entries generation.
I've got an idea, lzbt-systemd-xen should only generate Xen-dependent files, and then delegate rest of the work to the lzbt-systemd.
There still will be changes to the shared partition of the tool code, but they can be then feature-gated for xen.
The only problem remains is the garbage collector. lzbt-systemd still needs to be taught to garbage-collect /boot/loader/entries, even if that directory is not populated by it right now (Until lanzaboote supports managing memtest and other plain efi binaries).
Is that ok from you?
Still, it would be much cleaner to just add this code to lzbt-systemd, but I understand that Xen might be too niche, and not everyone wants to deal with code supporting it.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.