Hello again,
Having been using the MIPS-based CI20 single-board computer a bit more recently, I wondered if it might be possible to get the user mode Fiasco kernel working on MIPS. As I understand it, the UX variant of Fiasco only works on IA32 (x86, 32-bit), and although MIPS isn't particularly popular, it would at least be useful to me.
However, having looked at the code, it seems like quite a bit of work to unpick the IA32-related aspects from the UX-specific code. There appears to be a fair amount of descriptor table usage, for instance, and despite reading the thesis about it [1], I don't feel I have a decent-enough overview of what would really need doing to port the UX variant to another architecture.
So far, I've reviewed the MIPS code which employs a Proc abstraction and has a number of functions in the Mips namespace which could be replaced to maintain a "virtual" processor in the user mode kernel. It seems that the existing UX variant employs an Emulation abstraction for things like the processor status, and perhaps the Proc abstraction could take over these responsibilities. Quite how many privileged CPU registers would need maintaining probably depends on how much other code would still be using them.
This brings me to an awkward difference between IA32 and MIPS: that of the page fault handling and memory management. MIPS has no inherent notion of page table structures that are held by the CPU and consulted by it when addressing errors occur. Instead, an exception handler is invoked to populate various privileged registers and to perform privileged instructions. However, it seems to me that Fiasco needs to maintain a page directory, anyway, and that the user mode kernel is not likely to be attempting the actual memory management operations, either.
Consequently, it is tempting to think that the existing UX code might also be usable on MIPS. This would actually involve replacing a dedicated page directory implementation that appears to be present for MIPS with the one that other architectures appear to use. One detail that might need preserving is the code path dealing with TLB lookups (as a consequence of addressing errors), but it may be the case that page faults are dealt with more directly when signals arrive - perhaps what the UX code does now - and that the TLB exception vectors will remain unused.
When considering some of these issues, I did wonder about the role of certain architecture-specific aspects of the MIPS implementation. The existing UX code seems to deal with "vectors" and seems rather connected to interrupt descriptor tables (or their emulation). On MIPS, exception vectors are typically resident in kernel memory and have specific addresses.
It seems to me that the handling of exceptions (caused by signals) would direct execution via these vectors and thus preserve the existing exception.S implementation, albeit relocated to user memory. I guess the alternative would involve rewriting such things and using emulated processor features directly instead of trapping and handling privileged instructions. (I did also wonder to what extent the assembly language could also be migrated to C++ even for the native MIPS implementation.)
That brings me to a discussion about the purpose of the user mode kernel variant. As I understand it, one of the aims is to preserve architecture- specific code so as to give that code some possible testing and general exercise: it might actually help in the development of the kernel more generally. Is this a reasonable understanding? Initially, I thought that UX would actually be architecture-neutral (in terms of mechanisms; obviously the code would be native), but it appears that a stated aim is to be able to run existing binaries unchanged. Would an architecture-neutral UX variant make any sense?
I suppose I could go on and on, here, but as far as I can tell there are quite a few things that would need deciding in order to have a realistic chance of ever getting this working, amongst others that I will have forgotten or not considered...
* Redefining the memory layout for kernel and task processes (so as to work within the host system constraints)
* Separating generic and native functionality for various central abstractions
* Replacing the memory management with a mechanism similar to that employed by the existing UX code
* Defining a usable interrupt and exception paradigm (here, I am tempted to introduce a second-level controller for user mode interrupts)
* Determining which modules are required and how the code should be built (largely using the existing UX configurations)
* Testing and integrating the host system functionality required for the user mode kernel to function (process tracing, signal handling, system call invocations using the trampoline, thread management)
On this last point, I investigated the trampoline mechanism on a MIPS Linux system, and it does seem viable. There are a few tricks to getting system calls working (that can be learned from glibc and the kernel itself), although the trick employed to cancel them may not work. Also, the paper [1] describes the need to acquire signal context information by deploying a signal handler and having processes capture context data on the kernel's behalf. This may not be necessary on MIPS since the interesting details - those that seem to be posted on the stack when IA32 exceptions occur - are available via the registers exposed by ptrace on MIPS.
Indeed, I learned that ptrace is something of a fickle friend, not necessarily providing the same level of support on all systems. And thus, I wondered if there is still much sustained interest in the user mode approach or if people regard other technologies to be more practical and/or interesting. If you have read this far, maybe you might be able to share your thoughts on the topic.
It has been frustrating trying to get a good overview of the code - I'm not sure the preprocessor really helps, to be honest - and while I would like to be able to get the user mode kernel working on another architecture, I would rather like to know if it is really worth my while. Getting a better understanding is therefore the first step in finding out.
Paul
[1] http://os.inf.tu-dresden.de/papers_ps/steinberg-beleg.pdf
Hi Paul,
UX is an interesting way of virtualizing a kernel and gives interesting insights into both Fiasco and Linux. I guess with some work it would be well possible to make this work on MIPS. But, before going on this endeavor what about QEMU? It should be much quicker and simpler to run it through QEMU,
Adam
On Mon Aug 12, 2019 at 00:35:10 +0200, Paul Boddie wrote:
Having been using the MIPS-based CI20 single-board computer a bit more recently, I wondered if it might be possible to get the user mode Fiasco kernel working on MIPS. As I understand it, the UX variant of Fiasco only works on IA32 (x86, 32-bit), and although MIPS isn't particularly popular, it would at least be useful to me.
However, having looked at the code, it seems like quite a bit of work to unpick the IA32-related aspects from the UX-specific code. There appears to be a fair amount of descriptor table usage, for instance, and despite reading the thesis about it [1], I don't feel I have a decent-enough overview of what would really need doing to port the UX variant to another architecture.
So far, I've reviewed the MIPS code which employs a Proc abstraction and has a number of functions in the Mips namespace which could be replaced to maintain a "virtual" processor in the user mode kernel. It seems that the existing UX variant employs an Emulation abstraction for things like the processor status, and perhaps the Proc abstraction could take over these responsibilities. Quite how many privileged CPU registers would need maintaining probably depends on how much other code would still be using them.
This brings me to an awkward difference between IA32 and MIPS: that of the page fault handling and memory management. MIPS has no inherent notion of page table structures that are held by the CPU and consulted by it when addressing errors occur. Instead, an exception handler is invoked to populate various privileged registers and to perform privileged instructions. However, it seems to me that Fiasco needs to maintain a page directory, anyway, and that the user mode kernel is not likely to be attempting the actual memory management operations, either.
Consequently, it is tempting to think that the existing UX code might also be usable on MIPS. This would actually involve replacing a dedicated page directory implementation that appears to be present for MIPS with the one that other architectures appear to use. One detail that might need preserving is the code path dealing with TLB lookups (as a consequence of addressing errors), but it may be the case that page faults are dealt with more directly when signals arrive - perhaps what the UX code does now - and that the TLB exception vectors will remain unused.
When considering some of these issues, I did wonder about the role of certain architecture-specific aspects of the MIPS implementation. The existing UX code seems to deal with "vectors" and seems rather connected to interrupt descriptor tables (or their emulation). On MIPS, exception vectors are typically resident in kernel memory and have specific addresses.
It seems to me that the handling of exceptions (caused by signals) would direct execution via these vectors and thus preserve the existing exception.S implementation, albeit relocated to user memory. I guess the alternative would involve rewriting such things and using emulated processor features directly instead of trapping and handling privileged instructions. (I did also wonder to what extent the assembly language could also be migrated to C++ even for the native MIPS implementation.)
That brings me to a discussion about the purpose of the user mode kernel variant. As I understand it, one of the aims is to preserve architecture- specific code so as to give that code some possible testing and general exercise: it might actually help in the development of the kernel more generally. Is this a reasonable understanding? Initially, I thought that UX would actually be architecture-neutral (in terms of mechanisms; obviously the code would be native), but it appears that a stated aim is to be able to run existing binaries unchanged. Would an architecture-neutral UX variant make any sense?
I suppose I could go on and on, here, but as far as I can tell there are quite a few things that would need deciding in order to have a realistic chance of ever getting this working, amongst others that I will have forgotten or not considered...
Redefining the memory layout for kernel and task processes (so as to work within the host system constraints)
Separating generic and native functionality for various central abstractions
Replacing the memory management with a mechanism similar to that employed by the existing UX code
Defining a usable interrupt and exception paradigm (here, I am tempted to introduce a second-level controller for user mode interrupts)
Determining which modules are required and how the code should be built (largely using the existing UX configurations)
Testing and integrating the host system functionality required for the user mode kernel to function (process tracing, signal handling, system call invocations using the trampoline, thread management)
On this last point, I investigated the trampoline mechanism on a MIPS Linux system, and it does seem viable. There are a few tricks to getting system calls working (that can be learned from glibc and the kernel itself), although the trick employed to cancel them may not work. Also, the paper [1] describes the need to acquire signal context information by deploying a signal handler and having processes capture context data on the kernel's behalf. This may not be necessary on MIPS since the interesting details - those that seem to be posted on the stack when IA32 exceptions occur - are available via the registers exposed by ptrace on MIPS.
Indeed, I learned that ptrace is something of a fickle friend, not necessarily providing the same level of support on all systems. And thus, I wondered if there is still much sustained interest in the user mode approach or if people regard other technologies to be more practical and/or interesting. If you have read this far, maybe you might be able to share your thoughts on the topic.
It has been frustrating trying to get a good overview of the code - I'm not sure the preprocessor really helps, to be honest - and while I would like to be able to get the user mode kernel working on another architecture, I would rather like to know if it is really worth my while. Getting a better understanding is therefore the first step in finding out.
Paul
[1] http://os.inf.tu-dresden.de/papers_ps/steinberg-beleg.pdf
l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
On Monday 19. August 2019 00.31.16 Adam Lackorzynski wrote:
Hi Paul,
UX is an interesting way of virtualizing a kernel and gives interesting insights into both Fiasco and Linux. I guess with some work it would be well possible to make this work on MIPS.
After writing my message, I did think of pursuing a simpler, iterative approach to making the kernel run as a Linux process, ultimately employing most of the same techniques but without trying to adapt the existing UX code.
But, before going on this endeavor what about QEMU? It should be much quicker and simpler to run it through QEMU,
This did occur to me: after all, what is mostly needed is a way of emulating the privileged instructions, and QEMU is a general framework for achieving this. (The documentation isn't completely clear about whether I would end up emulating everything even on the same architecture.)
So, I have built Fiasco and L4Re for the MIPS Malta platform, and it seems to be possible to launch a system using the Makefile. For instance:
make O=mybuild qemu E=hello
However, I imagine that either a graphical example is needed where the SDL interface for QEMU is being used, or that some options are required to get QEMU to use a simple serial console. There appear to be some options and the possibility to do something like this:
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic'
Maybe even the machine type is necessary amongst the options as well. This doesn't seem to produce output, though.
I have to ask: what do people actually use when developing L4Re and Fiasco?
Paul
Hi Paul,
On [22-08-2019 13:33], Paul Boddie wrote:
On Monday 19. August 2019 00.31.16 Adam Lackorzynski wrote:
Hi Paul,
UX is an interesting way of virtualizing a kernel and gives interesting insights into both Fiasco and Linux. I guess with some work it would be well possible to make this work on MIPS.
After writing my message, I did think of pursuing a simpler, iterative approach to making the kernel run as a Linux process, ultimately employing most of the same techniques but without trying to adapt the existing UX code.
But, before going on this endeavor what about QEMU? It should be much quicker and simpler to run it through QEMU,
This did occur to me: after all, what is mostly needed is a way of emulating the privileged instructions, and QEMU is a general framework for achieving this. (The documentation isn't completely clear about whether I would end up emulating everything even on the same architecture.)
So, I have built Fiasco and L4Re for the MIPS Malta platform, and it seems to be possible to launch a system using the Makefile. For instance:
make O=mybuild qemu E=hello
Did you set the correct qemu options in your Makeconf.boot? And you may have to specify the correct platform type because the default afik 'boston', like
make O=mybuild qemu PT=malta E=hello
Also check the qemu cmdline that it contains "-M malta".
However, I imagine that either a graphical example is needed where the SDL interface for QEMU is being used, or that some options are required to get QEMU to use a simple serial console. There appear to be some options and the possibility to do something like this:
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic'
Maybe even the machine type is necessary amongst the options as well. This doesn't seem to produce output, though.
I have to ask: what do people actually use when developing L4Re and Fiasco?
Qemu is always a good start. It allows very quick development and test cycles. In the end you of course have to test on hardware.
Regards, Matthias.
Paul
l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
On Thursday 22. August 2019 14.15.55 Matthias Lange wrote:
Hi Paul,
On [22-08-2019 13:33], Paul Boddie wrote:
So, I have built Fiasco and L4Re for the MIPS Malta platform, and it seems to be possible to launch a system using the Makefile. For instance:
make O=mybuild qemu E=hello
Did you set the correct qemu options in your Makeconf.boot? And you may have to specify the correct platform type because the default afik 'boston', like
make O=mybuild qemu PT=malta E=hello
Also check the qemu cmdline that it contains "-M malta".
I set the qemu options on the command line (noted below), but I imagine that Makeconf.boot would also be a reasonable place. I tend to forget about all these different files after a while.
In the Fiasco and L4Re "make config" procedures, I did set the platform type to Malta, but it seems that you are suggesting also indicating this on the command line, too.
However, I imagine that either a graphical example is needed where the SDL interface for QEMU is being used, or that some options are required to get QEMU to use a simple serial console. There appear to be some options and the possibility to do something like this:
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic'
Maybe even the machine type is necessary amongst the options as well. This doesn't seem to produce output, though.
Here, I also added '-M malta' to QEMU_OPTIONS, having seen that qemu probably wasn't supplied those tokens by the Makefile.
I have to ask: what do people actually use when developing L4Re and Fiasco?
Qemu is always a good start. It allows very quick development and test cycles. In the end you of course have to test on hardware.
If I can get it to work, maybe it will be more useful than UX, although I did find UX to be fairly useful on my IA32 machine. Previously I have tended to steer clear of qemu because it has seemed to be a rather heavyweight solution to various problems that can be solved in other ways (such as people building entire software distributions by running native compilers under emulation rather than supporting cross-compilation instead).
Thanks to you and Adam for the hints so far!
Paul
On Thursday 22. August 2019 15.34.05 Paul Boddie wrote:
On Thursday 22. August 2019 14.15.55 Matthias Lange wrote:
Did you set the correct qemu options in your Makeconf.boot? And you may have to specify the correct platform type because the default afik 'boston', like
make O=mybuild qemu PT=malta E=hello
Also check the qemu cmdline that it contains "-M malta".
I set the qemu options on the command line (noted below), but I imagine that Makeconf.boot would also be a reasonable place. I tend to forget about all these different files after a while.
In the Fiasco and L4Re "make config" procedures, I did set the platform type to Malta, but it seems that you are suggesting also indicating this on the command line, too.
Just a quick update: I have since tried running qemu via the Makefile with the machine and console output parameters indicated using QEMU_OPTIONS; the emulator does start up and run code at the expected bootstrap locations but it doesn't seem to produce any output.
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic -M malta'
I suspect that there is some kind of restarting happening because when using the monitor to check the program counter, it showed the bootstrap code start address even after being allowed to run for a while. I guess I could investigate debugging options in the bootstrap code, especially since I was doing that with real hardware last year.
The bootstrap address is, of course, defined in the following file:
src/l4/pkg/bootstrap/server/src/Make.rules
It is set by this definition:
DEFAULT_RELOC_mips32:= 0x802d0000
And it appears to be this address that appears in the qemu monitor.
[...]
Qemu is always a good start. It allows very quick development and test cycles. In the end you of course have to test on hardware.
If I can get it to work, maybe it will be more useful than UX, although I did find UX to be fairly useful on my IA32 machine.
I will add that I have been able to build Fiasco to run as a process in my MIPS GNU/Linux environment, having hopefully figured out usable linker options, modules, and so on. In order to satisfy the kernel, I have provided some emulated support for privileged registers and the memory management unit, although the latter will need integrating with (or be replaced by) host-level memory mapping operations under normal operation.
Currently, I am working through adding the UX functionality as it is needed and, with some physical memory having been provided for the kernel (using mmap and indicating memory descriptors), I now need to handle signals and jump to the appropriate exception handlers, starting off with handling the timer interrupt being produced by the irq0 program.
Paul
Hi,
On 8/28/19 4:07 PM, Paul Boddie wrote:
On Thursday 22. August 2019 15.34.05 Paul Boddie wrote:
On Thursday 22. August 2019 14.15.55 Matthias Lange wrote:
Did you set the correct qemu options in your Makeconf.boot? And you may have to specify the correct platform type because the default afik 'boston', like
make O=mybuild qemu PT=malta E=hello
Also check the qemu cmdline that it contains "-M malta".
I set the qemu options on the command line (noted below), but I imagine that Makeconf.boot would also be a reasonable place. I tend to forget about all these different files after a while.
In the Fiasco and L4Re "make config" procedures, I did set the platform type to Malta, but it seems that you are suggesting also indicating this on the command line, too.
Just a quick update: I have since tried running qemu via the Makefile with the machine and console output parameters indicated using QEMU_OPTIONS; the emulator does start up and run code at the expected bootstrap locations but it doesn't seem to produce any output.
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic -M malta'
You must also add the CPU type for Mips QEMU. For Mips 32 R2 this would be:
-M malta -cpu P5600
(Unrelated, I usually also set: -nographic -monitor none -serial stdio That gives you the output directly on the console, no extra windows.)
Please have a look at the conf/Makeconf.boot.example which provides a working QEMU configurations for various platforms. I strongly recommend copying it to conf/Makeconf.boot and then adapting it to your needs. It makes life a lot easier and serves as a handy reminder what your favorite options are.
Kind regards
Sarah
On Wednesday 28. August 2019 16.47.31 Sarah Hoffmann wrote:
On 8/28/19 4:07 PM, Paul Boddie wrote:
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic -M malta'
You must also add the CPU type for Mips QEMU. For Mips 32 R2 this would be:
-M malta -cpu P5600
(Unrelated, I usually also set: -nographic -monitor none -serial stdio That gives you the output directly on the console, no extra windows.)
This is very useful to know. I will give it a try and report back on the outcome.
Please have a look at the conf/Makeconf.boot.example which provides a working QEMU configurations for various platforms. I strongly recommend copying it to conf/Makeconf.boot and then adapting it to your needs. It makes life a lot easier and serves as a handy reminder what your favorite options are.
I'll take a look at this as well. Makeconf.boot is something I tend to set up only after realising that things don't work without it (and that it isn't set up by default, or maybe not in a usable way) and then I forget about it, but obviously it contains the options that things like UX (on IA32) need.
Thanks for the advice, and apologies for not figuring this out myself!
Paul
On Thursday 29. August 2019 12.31.49 Paul Boddie wrote:
On Wednesday 28. August 2019 16.47.31 Sarah Hoffmann wrote:
On 8/28/19 4:07 PM, Paul Boddie wrote:
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic -M malta'
You must also add the CPU type for Mips QEMU. For Mips 32 R2 this would be: -M malta -cpu P5600
(Unrelated, I usually also set: -nographic -monitor none -serial stdio That gives you the output directly on the console, no extra windows.)
This is very useful to know. I will give it a try and report back on the outcome.
Well, the short story is that I got it to work, so many thanks are due for the help. Of course, had I looked at the Makeconf.boot.example file, I would have learned about the necessary -cpu option, but that was not the only obstacle here.
(The -serial stdio option is apparently the default, but with qemu having so many options and switches, I imagine that it is worth specifying in case defaults change or option combinations have side-effects.)
What I found was that the bootstrap code was not completing. I added the -S option to qemu to stop the CPU at start-up time, and I used the -s option to set up remote debugging. Then, I ran gdb in another terminal, using the following commands in the gdb session:
target remote localhost:1234 set architecture mips:isa32r2 hbreak *0x802d0000 c
Using the si command to step through the code, it appeared that execution failed at the point where a load was first made relative to the gp register. Using the info registers command, I could see that the initialisation of this register was not done properly. And looking at the registers at the start of bootstrap routine execution, I could see that t9 was being used to initialise gp but had not been set up.
If this sounds familiar it is because there were similar issues with other assembly language routines that I ended up patching to run L4Re on the different Ingenic devices I have been using. However, I never needed to patch this code. An explanation for this might be that on the actual hardware, U- Boot is involved and it might well initialise t9 when jumping to the bootstrap code. Here, the CPU firmware does not set up t9.
Previously, it was noted that other compilers had been used to develop L4Re for MIPS platforms, and I suspect that there must be a difference in the behaviour of the .cpload macro between the assemblers employed by these compilers. With my Debian-provided GCC toolchains, .cpload doesn't seem to be setting up t9, and it may be that it will only do so if there is a frame declared, which is not the case in the code affected by this problem (in the different places in L4Re).
(I would have to remind myself about what the .cp* macros actually do because I don't remember at this point in time.)
So maybe the approach for initialising t9 could be reviewed so that it is not toolchain-specific. Here, as before, I ended up doing something like this:
_start: lui $25, %hi(_realstart) ori $25, $25, %lo(_realstart) _realstart: ...
As for the eventual outcome, I managed to get the payload working with qemu which is encouraging, although the performance didn't seem particularly great. I imagine that the performance might improve if and when qemu realises that most of the code can just run directly on the hardware, and the just-in-time compilation should figure this out, so perhaps more testing is needed.
Thanks once again for the help!
Paul
Hi Paul,
On Fri Aug 30, 2019 at 13:43:10 +0200, Paul Boddie wrote:
On Thursday 29. August 2019 12.31.49 Paul Boddie wrote:
On Wednesday 28. August 2019 16.47.31 Sarah Hoffmann wrote:
On 8/28/19 4:07 PM, Paul Boddie wrote:
make O=mybuild qemu E=hello QEMU_OPTIONS='-nographic -M malta'
You must also add the CPU type for Mips QEMU. For Mips 32 R2 this would be: -M malta -cpu P5600
(Unrelated, I usually also set: -nographic -monitor none -serial stdio That gives you the output directly on the console, no extra windows.)
This is very useful to know. I will give it a try and report back on the outcome.
Well, the short story is that I got it to work, so many thanks are due for the help. Of course, had I looked at the Makeconf.boot.example file, I would have learned about the necessary -cpu option, but that was not the only obstacle here.
(The -serial stdio option is apparently the default, but with qemu having so many options and switches, I imagine that it is worth specifying in case defaults change or option combinations have side-effects.)
What I found was that the bootstrap code was not completing. I added the -S option to qemu to stop the CPU at start-up time, and I used the -s option to set up remote debugging. Then, I ran gdb in another terminal, using the following commands in the gdb session:
target remote localhost:1234 set architecture mips:isa32r2 hbreak *0x802d0000 c
Using the si command to step through the code, it appeared that execution failed at the point where a load was first made relative to the gp register. Using the info registers command, I could see that the initialisation of this register was not done properly. And looking at the registers at the start of bootstrap routine execution, I could see that t9 was being used to initialise gp but had not been set up.
If this sounds familiar it is because there were similar issues with other assembly language routines that I ended up patching to run L4Re on the different Ingenic devices I have been using. However, I never needed to patch this code. An explanation for this might be that on the actual hardware, U- Boot is involved and it might well initialise t9 when jumping to the bootstrap code. Here, the CPU firmware does not set up t9.
Previously, it was noted that other compilers had been used to develop L4Re for MIPS platforms, and I suspect that there must be a difference in the behaviour of the .cpload macro between the assemblers employed by these compilers. With my Debian-provided GCC toolchains, .cpload doesn't seem to be setting up t9, and it may be that it will only do so if there is a frame declared, which is not the case in the code affected by this problem (in the different places in L4Re).
(I would have to remind myself about what the .cp* macros actually do because I don't remember at this point in time.)
So maybe the approach for initialising t9 could be reviewed so that it is not toolchain-specific. Here, as before, I ended up doing something like this:
_start: lui $25, %hi(_realstart) ori $25, $25, %lo(_realstart) _realstart: ...
Thanks for the update! Indeed it seems to work for me without any adjustments both with mips32 on malta and with mips64 on the boston platform (both with a pretty recent QEMU). I probably using the "wrong" compiler. I'll try to follow up here once sid's cross compilers are installable again.
Adam
On Monday 9. September 2019 23.53.13 Adam Lackorzynski wrote:
On Fri Aug 30, 2019 at 13:43:10 +0200, Paul Boddie wrote:
So maybe the approach for initialising t9 could be reviewed so that it is not toolchain-specific. Here, as before, I ended up doing something like this:> _start: lui $25, %hi(_realstart) ori $25, $25, %lo(_realstart)
_realstart: ...
Thanks for the update! Indeed it seems to work for me without any adjustments both with mips32 on malta and with mips64 on the boston platform (both with a pretty recent QEMU). I probably using the "wrong" compiler. I'll try to follow up here once sid's cross compilers are installable again.
At the moment, I'm using the native compiler in the Stretch (oldstable) version of Debian, but I first had this problem with the Sid (unstable) cross- compilers about two years ago, and I imagine that there is something about Debian's compilers or even generic GCC which is different from whatever you are using. (I should perhaps check the output of my Buildroot cross-compilers to see what they actually produce, too.)
Still, the compilers - of both kinds - in Stretch (oldstable) and Buster (stable) should be usable. I haven't done much with the unstable cross- compilers in a couple of months.
Paul
P.S. I have also been pursuing the user mode kernel work, which for various reasons has been proceeding slowly. Aside from errors I have introduced myself, there have been plenty of challenges getting things like data structures allocated and initialised, and it feels like guesswork at times.
On Tuesday 10. September 2019 12.42.36 Paul Boddie wrote:
P.S. I have also been pursuing the user mode kernel work, which for various reasons has been proceeding slowly. Aside from errors I have introduced myself, there have been plenty of challenges getting things like data structures allocated and initialised, and it feels like guesswork at times.
I haven't been really able to dedicate myself to the above work recently, but on a somewhat related topic, I did try and get KVM to work with QEMU on MIPS. The intention was to take advantage of the native architecture and to delegate only privileged operations to KVM and/or QEMU.
On the MIPS Creator CI20, the only real option for KVM is the trap-and-emulate mode, partly because the JZ4780 SoC does not have VZ support, also because the last kernel properly supporting the hardware (3.18) only has KVM support to this level. In principle, starting a payload in QEMU should only see a trap to the kernel occur when a privileged operation takes place, and this does indeed seem to be what happens.
Since Fiasco (or other privileged code) would normally run in the kernel address space (KSEG0), a curious arrangement is involved in MIPS KVM where such kernel code has to run in a special "guest KSEG0" space (starting at 0x40000000 instead of 0x80000000). This gives such code a chance to run in user mode until it performs a privileged instruction or accesses memory regions that are privileged. Meanwhile, code running in the truncated user address space is presumably prevented from accessing "guest KSEG0" using normal memory mapping techniques.
Sadly, I couldn't get KVM to work for me. I recompiled the Linux kernel to enable KVM and recompiled the Fiasco kernel having changed references to KSEG0, the kernel load address, and so on. I also changed L4Re to take the revised KSEG0 address into account. However, within QEMU, the built-in bootloader for the Malta platform would fail to read the first faulting instruction accessing I/O memory. In effect, QEMU would direct the execution of the code, a trap would occur for an access to I/O memory, but the KVM code would fail to access the memory containing the instruction.
There is a caveat noted in the MIPS KVM support in the Linux kernel which indicates that a 16K page size is required both in the host and guest kernels. I recompiled Linux and Fiasco to comply with this, and I then briefly saw the QEMU-directed execution enter the L4Re bootstrap code, only to see the same problem occurring with the instruction stream. A subsequent attempt saw the problem occur once again within the Malta bootloader, suggesting some intermittent issue. I think the KVM code was refactored somewhat in later Linux kernels.
I don't know if anyone else has tried any of this at any point. It seems to me that there are few enough people using native MIPS systems for development that there can hardly be any at all using KVM on MIPS and perhaps no-one who has tried to boot Fiasco in this way. But it seems reasonable to see if anyone gave it any consideration (or even uses KVM on other architectures).
Paul
l4-hackers@os.inf.tu-dresden.de
-
Adam Lackorzynski -
Matthias Lange -
Paul Boddie -
Sarah Hoffmann