Hello,
I have been trying to get L4Re working on another MIPS-based SoC and found myself struggling to get the kernel to boot. Initially, I wondered about various exotic concurrency issues, particularly since the SoC has two cores (each with two threads, apparently), but I started to suspect that our old nemesis, cache management, was to blame.
Anyway, to save you the long story, I had a closer look at the syncICache routine that resides in the bootstrap package/module:
pkg/bootstrap/server/src/ARCH-mips/crt0.S
It turns out that syncICache is ineffective due to an erroneous loop continuation test. Here's the offending code:
10: synci 0($a0)
addu $a0, $a0, $v0 sltu $v1, $a0, $a1 bne $v1, $0, 10b nop
Here, $a0 is the region start and $a1 is the region size, with $v0 being the cache line size. The above code is equivalent to the following:
address = start; do { synci 0(address)
address += linesize; } while (address < size);
The address should, of course, be compared to start + size. So, cache entries may not be flushed, particularly later in regions and at higher addresses, omitting coverage even of the early kernel regions, making kernel execution very unpredictable and unreliable. Perhaps the larger caches make this fault more obvious on this particular SoC.
In the startup routine in the bootstrap package/module...
pkg/bootstrap/server/src/startup.cc
...there is a declaration for syncICache, along with code to flush the caches, introduced purely for MIPS. To test my suspicions, I wrote a C++ version of the routine with inline assembly language for the machine instructions, fixing the loop continuation test. This seemed to allow the kernel to boot and the system to start up as I originally expected.
Would you like me to contribute this fix? I think that unless inline assembly is problematic for some toolchain choices, it might be clearer to maintain this code in C++ rather than entirely in assembly language, thus avoiding such faults in the first place.
This problem may have been around since the introduction of MIPS support into L4Re, appearing first in r70 of the public Subversion repository from back in May 2016.
Paul