QEMU/KVM on MIPS (was Re: Fiasco.OC-UX on MIPS?)

[Paul Boddie]

On Tuesday 10. September 2019 12.42.36 Paul Boddie wrote:
> 
> P.S. I have also been pursuing the user mode kernel work, which for various
> reasons has been proceeding slowly. Aside from errors I have introduced
> myself, there have been plenty of challenges getting things like data
> structures allocated and initialised, and it feels like guesswork at times.

I haven't been really able to dedicate myself to the above work recently, but 
on a somewhat related topic, I did try and get KVM to work with QEMU on MIPS. 
The intention was to take advantage of the native architecture and to delegate 
only privileged operations to KVM and/or QEMU.

On the MIPS Creator CI20, the only real option for KVM is the trap-and-emulate 
mode, partly because the JZ4780 SoC does not have VZ support, also because the 
last kernel properly supporting the hardware (3.18) only has KVM support to 
this level. In principle, starting a payload in QEMU should only see a trap to 
the kernel occur when a privileged operation takes place, and this does indeed 
seem to be what happens.

Since Fiasco (or other privileged code) would normally run in the kernel 
address space (KSEG0), a curious arrangement is involved in MIPS KVM where 
such kernel code has to run in a special "guest KSEG0" space (starting at 
0x40000000 instead of 0x80000000). This gives such code a chance to run in 
user mode until it performs a privileged instruction or accesses memory 
regions that are privileged. Meanwhile, code running in the truncated user 
address space is presumably prevented from accessing "guest KSEG0" using 
normal memory mapping techniques.

Sadly, I couldn't get KVM to work for me. I recompiled the Linux kernel to 
enable KVM and recompiled the Fiasco kernel having changed references to 
KSEG0, the kernel load address, and so on. I also changed L4Re to take the 
revised KSEG0 address into account. However, within QEMU, the built-in 
bootloader for the Malta platform would fail to read the first faulting 
instruction accessing I/O memory. In effect, QEMU would direct the execution 
of the code, a trap would occur for an access to I/O memory, but the KVM code 
would fail to access the memory containing the instruction.

There is a caveat noted in the MIPS KVM support in the Linux kernel which 
indicates that a 16K page size is required both in the host and guest kernels. 
I recompiled Linux and Fiasco to comply with this, and I then briefly saw the 
QEMU-directed execution enter the L4Re bootstrap code, only to see the same 
problem occurring with the instruction stream. A subsequent attempt saw the 
problem occur once again within the Malta bootloader, suggesting some 
intermittent issue. I think the KVM code was refactored somewhat in later 
Linux kernels.

I don't know if anyone else has tried any of this at any point. It seems to me 
that there are few enough people using native MIPS systems for development 
that there can hardly be any at all using KVM on MIPS and perhaps no-one who 
has tried to boot Fiasco in this way. But it seems reasonable to see if anyone 
gave it any consideration (or even uses KVM on other architectures).

Paul