On Tuesday 10. September 2019 12.42.36 Paul Boddie wrote:
P.S. I have also been pursuing the user mode kernel work, which for various reasons has been proceeding slowly. Aside from errors I have introduced myself, there have been plenty of challenges getting things like data structures allocated and initialised, and it feels like guesswork at times.
I haven't been really able to dedicate myself to the above work recently, but on a somewhat related topic, I did try and get KVM to work with QEMU on MIPS. The intention was to take advantage of the native architecture and to delegate only privileged operations to KVM and/or QEMU.
On the MIPS Creator CI20, the only real option for KVM is the trap-and-emulate mode, partly because the JZ4780 SoC does not have VZ support, also because the last kernel properly supporting the hardware (3.18) only has KVM support to this level. In principle, starting a payload in QEMU should only see a trap to the kernel occur when a privileged operation takes place, and this does indeed seem to be what happens.
Since Fiasco (or other privileged code) would normally run in the kernel address space (KSEG0), a curious arrangement is involved in MIPS KVM where such kernel code has to run in a special "guest KSEG0" space (starting at 0x40000000 instead of 0x80000000). This gives such code a chance to run in user mode until it performs a privileged instruction or accesses memory regions that are privileged. Meanwhile, code running in the truncated user address space is presumably prevented from accessing "guest KSEG0" using normal memory mapping techniques.
Sadly, I couldn't get KVM to work for me. I recompiled the Linux kernel to enable KVM and recompiled the Fiasco kernel having changed references to KSEG0, the kernel load address, and so on. I also changed L4Re to take the revised KSEG0 address into account. However, within QEMU, the built-in bootloader for the Malta platform would fail to read the first faulting instruction accessing I/O memory. In effect, QEMU would direct the execution of the code, a trap would occur for an access to I/O memory, but the KVM code would fail to access the memory containing the instruction.
There is a caveat noted in the MIPS KVM support in the Linux kernel which indicates that a 16K page size is required both in the host and guest kernels. I recompiled Linux and Fiasco to comply with this, and I then briefly saw the QEMU-directed execution enter the L4Re bootstrap code, only to see the same problem occurring with the instruction stream. A subsequent attempt saw the problem occur once again within the Malta bootloader, suggesting some intermittent issue. I think the KVM code was refactored somewhat in later Linux kernels.
I don't know if anyone else has tried any of this at any point. It seems to me that there are few enough people using native MIPS systems for development that there can hardly be any at all using KVM on MIPS and perhaps no-one who has tried to boot Fiasco in this way. But it seems reasonable to see if anyone gave it any consideration (or even uses KVM on other architectures).
Paul