Hello L4 hackers,
I'm currently evaluating whether we can use Fiasco/L4Re for some of our future projects and ran into some problems/questions that you might help me out with. The general idea is to isolate multiple Linux VMs running Debian/Ubuntu (unmodified kernel) on x86/amd64 hardware with L4Re.
I already did most of the tutorials on the kernkonzept/manifest github wiki. (https://github.com/kernkonzept/manifest/wiki) There were some issues with the hardware I'm using, since it maps its BIOS to an address that L4 likes to use for page tables. (at least on amd64, x86 did work out of the box) I fixed it by adjusting the base address in the linker script. (Hopefully, this should not have any bad side effects on L4.)
First question: In the VM tutorials, I always see the use of kernel image + cpio ramdisk in the startup code. Is this also applicable if I want to use a full-fledged Debian/Ubuntu installation? The guest system should also be able to modify the installation. (i. e. run apt-get upgrade etc.)
Second question: The general architecture for the VM scenario seems to be Fiasco/L4 -> uvmm -> Linux VMs. Is it also possible to use L4Linux as Host for the Linux VMs? My assumption would be, that it might be easier to use the Linux VMs from Linux instead of uvmm. (maybe with kvm/libvirt support?) Especially when it comes to passing the hardware to the VMs.
Third question: This is a direct follow up from the last point. Passing the HW is the hard part for me. I'm not that familiar with device tree files yet and as far as I know x86/amd64 usually does not use device tree because it can auto-detect via PCI/ACPI. But it looks like uvmm/io does require device tree to pass the HW to the VMs even though L4 itself is detecting it via PCI/ACPI. Maybe I can pass the whole PCI bus, but then I would like to isolate at least some parts. For example VM 1 should only get one port of the network adapter and VM 2 should get another one. (exclusively) And while both VMs should be able to read the RTC, they should not be able to modify it. (or maybe only one of them) I know, it's not really a question, but maybe you can give me some advice on how best to approach this on x86/amd64 hardware.
Fourth question: Does Fiasco/L4 support SR-IOV? I checked the code + commits and there seems to be some development in the last year, but not sure how "mature" it is.
That's all for now, thanks for reading and I hope the questions are not too embarrassing/annoying. :)
Best regards,
Andreas Hübner Software Developer Business Fullfillment | Research & Development aconnic group OFFICE: 01157 Dresden, Am Lehmberg 54, GERMANY Phone +49 3722 7351-1010 E-Mail andreas.huebner@aconnic.com Web www.aconnic.com Court Munich Registration HRB 292019 Chairman of the Supervisory Board: Dr. Stefan Hagemann Executive Board: Werner Neubauer, Markus Königshofer aconnic AG (Munich), aconnic system Germany GmbH (Hartmannsdorf), aconnic service GmbH (Kornwestheim), aconnic sistemas de telecomunicaciones S.A. de C.V. (Mexico City), aconnic system Switzerland AG (Zurich), aconnic engineering AG (Zurich), aconnic carbonzero GmbH (Starnberg) are direct affiliates of the aconnic group. We refer to our data protection rules at: https://www.aconnic.com/privacy/%C2%A0 _______________________________________________ l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de
Hello Andreas,
On Wed Apr 02, 2025 at 06:55:12 +0000, Hübner, Andreas wrote:
I'm currently evaluating whether we can use Fiasco/L4Re for some of our future projects and ran into some problems/questions that you might help me out with. The general idea is to isolate multiple Linux VMs running Debian/Ubuntu (unmodified kernel) on x86/amd64 hardware with L4Re.
Wonderful :)
I already did most of the tutorials on the kernkonzept/manifest github wiki. (https://github.com/kernkonzept/manifest/wiki) There were some issues with the hardware I'm using, since it maps its BIOS to an address that L4 likes to use for page tables. (at least on amd64, x86 did work out of the box) I fixed it by adjusting the base address in the linker script. (Hopefully, this should not have any bad side effects on L4.)
Is it some more special hardware? But if it work I guess it's good.
First question: In the VM tutorials, I always see the use of kernel image + cpio ramdisk in the startup code. Is this also applicable if I want to use a full-fledged Debian/Ubuntu installation? The guest system should also be able to modify the installation. (i. e. run apt-get upgrade etc.)
Yes, that's currently also applicable when booting full distributions. Obviously this is not ideal for this kine of use-case. We are working on it.
Second question: The general architecture for the VM scenario seems to be Fiasco/L4 -> uvmm -> Linux VMs.
Yes.
Is it also possible to use L4Linux as Host for the Linux VMs? My assumption would be, that it might be easier to use the Linux VMs from Linux instead of uvmm. (maybe with kvm/libvirt support?)
Good idea :) However, there's no working support for this approach (even if you might spot some code in this area in L4Linux).
Especially when it comes to passing the hardware to the VMs.
But that should not be a problem. Hardware pass-through it pretty common.
Third question: This is a direct follow up from the last point. Passing the HW is the hard part for me. I'm not that familiar with device tree files yet and as far as I know x86/amd64 usually does not use device tree because it can auto-detect via PCI/ACPI. But it looks like uvmm/io does require device tree to pass the HW to the VMs even though L4 itself is detecting it via PCI/ACPI.
It is correct that uvmm does need a device tree, also on x86. However, you can just take the virt-pc.dtb. All the pass-through of PCI devices will just work through the virtual PCI bridge, thus has nothing to do anymore with the device tree. Just for putting any other devices to the VM requires adding those to the device tree (such as more virtio device).
In the corresponding vbus config you select with PCI devs to make available on this vbus which will consequently show up in the VM.
Maybe I can pass the whole PCI bus, but then I would like to isolate at least some parts.
That's possible too, just to mention it.
For example VM 1 should only get one port of the network adapter and VM 2 should get another one. (exclusively)
Yes.
And while both VMs should be able to read the RTC, they should not be able to modify it. (or maybe only one of them)
There's an L4Re RTC service for this.
I know, it's not really a question, but maybe you can give me some advice on how best to approach this on x86/amd64 hardware.
Fourth question: Does Fiasco/L4 support SR-IOV? I checked the code + commits and there seems to be some development in the last year, but not sure how "mature" it is.
It's work in progress and I think there are some bits still missing. But should not take long anymore.
That's all for now, thanks for reading and I hope the questions are not too embarrassing/annoying. :)
Not at all :)
Adam _______________________________________________ l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de
Hello Adam,
thanks for your answers.
Is it some more special hardware? But if it work I guess it's good.
It's a whitebox from Armortec. Something like this: https://www.armortec.net/products/network-appliances/x86-desktop/2023/0725/3...
I did see the following error messages from IO while booting, but I don't think it is related to the changed base address.
error: EC: error missing resources error: ACPI LID0: could not evaluate _PRW object (AE_NOT_FOUND), not enabled as wake source
It is correct that uvmm does need a device tree, also on x86. However, you can just take the virt-pc.dtb. All the pass-through of PCI devices will just work through the virtual PCI bridge, thus has nothing to do anymore with the device tree. Just for putting any other devices to the VM requires adding those to the device tree (such as more virtio device).
In the corresponding vbus config you select with PCI devs to make available on this vbus which will consequently show up in the VM.
Okay, so it turns out I made two mistakes. (It is working fine now.) One was the naming of the vbuses where they have to match in the lua config. (I took the example where there is only one bus named vbus and added another VM, but must have overlooked how the matching is done.) And the second one is using the wrong kernel or rather the wrong network driver in the kernel. I did not see any network devices in the VM and assumed the pass-through did not work. But the PCI device was actually there under /sys/bus/pci. (The standard e100/e1000 driver included in your kernel does not handle the network interfaces on the whitebox, had to use igc.)
Maybe I can pass the whole PCI bus, but then I would like to isolate at least some parts.
That's possible too, just to mention it.
So I can pass the whole bus but then blacklist specific devices on that bus that should not be passed to the VM? Is there an example config for this?
And while both VMs should be able to read the RTC, they should not be able to modify it. (or maybe only one of them)
There's an L4Re RTC service for this.
Could you elaborate a bit on this? I did see the RTC driver, which seems to work as a multiplexer. But I didn't see how I can limit read/write access with it. (The documentation said there are no config options.) Or is this modeled via capabilities?
From what I read, the capabilities give access to the server/device they point to.
But how would the server know (without configuration) if the client is allowed to write or if it's read-only? (unless there are separate capabilities)
Thanks again for help,
Andreas _______________________________________________ l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de
Hi Andreas,
On Thu Apr 10, 2025 at 12:42:05 +0000, Hübner, Andreas wrote:
thanks for your answers.
Is it some more special hardware? But if it work I guess it's good.
It's a whitebox from Armortec. Something like this: https://www.armortec.net/products/network-appliances/x86-desktop/2023/0725/3...
I did see the following error messages from IO while booting, but I don't think it is related to the changed base address.
error: EC: error missing resources error: ACPI LID0: could not evaluate _PRW object (AE_NOT_FOUND), not enabled as wake source
Rather normal, some ACPI configs could be better.
It is correct that uvmm does need a device tree, also on x86. However, you can just take the virt-pc.dtb. All the pass-through of PCI devices will just work through the virtual PCI bridge, thus has nothing to do anymore with the device tree. Just for putting any other devices to the VM requires adding those to the device tree (such as more virtio device).
In the corresponding vbus config you select with PCI devs to make available on this vbus which will consequently show up in the VM.
Okay, so it turns out I made two mistakes. (It is working fine now.) One was the naming of the vbuses where they have to match in the lua config. (I took the example where there is only one bus named vbus and added another VM, but must have overlooked how the matching is done.) And the second one is using the wrong kernel or rather the wrong network driver in the kernel. I did not see any network devices in the VM and assumed the pass-through did not work. But the PCI device was actually there under /sys/bus/pci. (The standard e100/e1000 driver included in your kernel does not handle the network interfaces on the whitebox, had to use igc.)
Ok, thanks for the explanation.
Maybe I can pass the whole PCI bus, but then I would like to isolate at least some parts.
That's possible too, just to mention it.
So I can pass the whole bus but then blacklist specific devices on that bus that should not be passed to the VM? Is there an example config for this?
No, I meant that the whole PCIe controller could be given to the VM, such that io does not see or know about PCIe. Honestly, on x86 this is rather unusual to be done.
And while both VMs should be able to read the RTC, they should not be able to modify it. (or maybe only one of them)
There's an L4Re RTC service for this.
Could you elaborate a bit on this? I did see the RTC driver, which seems to work as a multiplexer. But I didn't see how I can limit read/write access with it. (The documentation said there are no config options.) Or is this modeled via capabilities?
From what I read, the capabilities give access to the server/device they point to.
But how would the server know (without configuration) if the client is allowed to write or if it's read-only? (unless there are separate capabilities)
Indeed it's done via caps. If a client has the rtc cap with write rights, it can write. If just with read rights, then it can only read. I've added this to the docs.
Thanks, Adam _______________________________________________ l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de
l4-hackers@os.inf.tu-dresden.de
-
Adam Lackorzynski -
Hübner, Andreas