Hello,
I have been trying to get L4Re working on another MIPS-based SoC and found myself struggling to get the kernel to boot. Initially, I wondered about various exotic concurrency issues, particularly since the SoC has two cores (each with two threads, apparently), but I started to suspect that our old nemesis, cache management, was to blame.
Anyway, to save you the long story, I had a closer look at the syncICache routine that resides in the bootstrap package/module:
pkg/bootstrap/server/src/ARCH-mips/crt0.S
It turns out that syncICache is ineffective due to an erroneous loop continuation test. Here's the offending code:
10: synci 0($a0)
addu $a0, $a0, $v0 sltu $v1, $a0, $a1 bne $v1, $0, 10b nop
Here, $a0 is the region start and $a1 is the region size, with $v0 being the cache line size. The above code is equivalent to the following:
address = start; do { synci 0(address)
address += linesize; } while (address < size);
The address should, of course, be compared to start + size. So, cache entries may not be flushed, particularly later in regions and at higher addresses, omitting coverage even of the early kernel regions, making kernel execution very unpredictable and unreliable. Perhaps the larger caches make this fault more obvious on this particular SoC.
In the startup routine in the bootstrap package/module...
pkg/bootstrap/server/src/startup.cc
...there is a declaration for syncICache, along with code to flush the caches, introduced purely for MIPS. To test my suspicions, I wrote a C++ version of the routine with inline assembly language for the machine instructions, fixing the loop continuation test. This seemed to allow the kernel to boot and the system to start up as I originally expected.
Would you like me to contribute this fix? I think that unless inline assembly is problematic for some toolchain choices, it might be clearer to maintain this code in C++ rather than entirely in assembly language, thus avoiding such faults in the first place.
This problem may have been around since the introduction of MIPS support into L4Re, appearing first in r70 of the public Subversion repository from back in May 2016.
Paul
Hi Paul,
thanks. Could probably be also done in C/C++, but your description is so good that this should be enough?
diff --git a/server/src/ARCH-mips/crt0.S b/server/src/ARCH-mips/crt0.S index b8fa86ba..f4c8d5dd 100644 --- a/server/src/ARCH-mips/crt0.S +++ b/server/src/ARCH-mips/crt0.S @@ -49,6 +49,7 @@ syncICache: ; li $v1, 2 # size is 2 << min(DL, IL) sllv $v0, $v1, $v0
+ addu $a1, $a0, $a1 # a1 is end = start + size # run the loop 10: synci 0($a0)
Thanks, Adam
On Sun Jun 08, 2025 at 00:35:06 +0200, Paul Boddie wrote:
Hello,
I have been trying to get L4Re working on another MIPS-based SoC and found myself struggling to get the kernel to boot. Initially, I wondered about various exotic concurrency issues, particularly since the SoC has two cores (each with two threads, apparently), but I started to suspect that our old nemesis, cache management, was to blame.
Anyway, to save you the long story, I had a closer look at the syncICache routine that resides in the bootstrap package/module:
pkg/bootstrap/server/src/ARCH-mips/crt0.S
It turns out that syncICache is ineffective due to an erroneous loop continuation test. Here's the offending code:
10: synci 0($a0)
addu $a0, $a0, $v0 sltu $v1, $a0, $a1 bne $v1, $0, 10b nopHere, $a0 is the region start and $a1 is the region size, with $v0 being the cache line size. The above code is equivalent to the following:
address = start; do { synci 0(address) address += linesize; } while (address < size);The address should, of course, be compared to start + size. So, cache entries may not be flushed, particularly later in regions and at higher addresses, omitting coverage even of the early kernel regions, making kernel execution very unpredictable and unreliable. Perhaps the larger caches make this fault more obvious on this particular SoC.
In the startup routine in the bootstrap package/module...
pkg/bootstrap/server/src/startup.cc
...there is a declaration for syncICache, along with code to flush the caches, introduced purely for MIPS. To test my suspicions, I wrote a C++ version of the routine with inline assembly language for the machine instructions, fixing the loop continuation test. This seemed to allow the kernel to boot and the system to start up as I originally expected.
Would you like me to contribute this fix? I think that unless inline assembly is problematic for some toolchain choices, it might be clearer to maintain this code in C++ rather than entirely in assembly language, thus avoiding such faults in the first place.
This problem may have been around since the introduction of MIPS support into L4Re, appearing first in r70 of the public Subversion repository from back in May 2016.
Paul
l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de
Adam
On Monday, 9 June 2025 22:34:02 CEST Adam Lackorzynski wrote:
thanks. Could probably be also done in C/C++, but your description is so good that this should be enough?
diff --git a/server/src/ARCH-mips/crt0.S b/server/src/ARCH-mips/crt0.S index b8fa86ba..f4c8d5dd 100644 --- a/server/src/ARCH-mips/crt0.S +++ b/server/src/ARCH-mips/crt0.S @@ -49,6 +49,7 @@ syncICache: ; li $v1, 2 # size is 2 << min(DL, IL) sllv $v0, $v1, $v0
addu $a1, $a0, $a1 # a1 is end = start + size # run the loop10: synci 0($a0)
I would think so. $a1 remains the region size and $a0 is still the region start at that point, so adding the two should yield the region end. Then, testing $a0 against $a1 should have the desired outcome.
I should probably have made this simple fix myself, but I wanted to be absolutely sure that I understood the behaviour and the intent.
Paul
l4-hackers@os.inf.tu-dresden.de
-
Adam Lackorzynski -
Paul Boddie