L4Re: MIPS SMP cache initialisation may be broken for some CPUs
Hello, Pushing on with my efforts to port L4Re to another MIPS-based SoC, I have been attempting to activate more than one CPU core. Although there is some support in Fiasco for the MIPS Coherence Manager (CM) and Cluster Power Controller (CPC), these technologies are not supported in this particular SoC, which is related to the JZ4780 used in the MIPS Creator CI20. However, the Platform_control abstraction (in src/kern/mips/platform_control- mips.cpp) does seem to provide some flexibility in the way secondary CPUs or cores may be started. It seems that even in situations where the CM is not available, the alloc_secondary_boot_code method will assemble a bootstrap routine from snippets including one, _tramp_mp_cache, that initialises the caches. Unfortunately, it appears that this cache initialisation code does not work on this particular SoC. The method used by_tramp_mp_cache is the familiar one where the cache tags are invalidated for all cache index entries using the Index_Store_Tag operation for each cache type. This is supposed to work on any MIPS CPU, but through some tedious troubleshooting, I discovered that it does not work for the data cache. If the data cache variant of the Index_Store_Tag operation is performed for a range of addresses starting from the base of KSEG0, I found that some kind of side-effect would occur, possibly affecting the rather critical addresses in that range. This would prevent Fiasco from continuing to start up, and things like timers and the UART would not function, although the processor was not halted and would still run code, illuminating LEDs as a last-resort debugging tool. There is some preliminary support for this SoC, as well as the JZ4780, in Linux and it seems that instead of using Index_Store_Tag, the tag records are explicitly invalidated and updated. However, it also seems to be the case that caches can be invalidated using Hit_Invalidate. Changing the cache operations in _tramp_mp_cache to use this invalidation operation in its two variants appears to work, avoiding the problems described above. Does anyone have any thoughts or recollections about this cache initialisation routine? On the topic of actually starting secondary cores, this SoC has its own set of registers for resetting cores, interrupt control, mailboxes, and so on. I have developed a fairly simple abstraction that performs similar core bootstrapping work to that of the Cm abstraction (in src/kern/mips/cm.cpp). I then extend the Platform_control abstraction so that it will call the alloc_secondary_boot_code method, followed by the appropriate method to start the secondary cores. Is there any documentation for the SMP architecture in Fiasco and L4Re? I note that interprocessor interrupts (IPI) are one element that I might need to support, probably using the mailboxes. Thanks in advance for any advice or feedback that may be given! Paul
Hi Paul, On Mon Jun 23, 2025 at 00:34:41 +0200, Paul Boddie wrote:
Pushing on with my efforts to port L4Re to another MIPS-based SoC, I have been attempting to activate more than one CPU core. Although there is some support in Fiasco for the MIPS Coherence Manager (CM) and Cluster Power Controller (CPC), these technologies are not supported in this particular SoC, which is related to the JZ4780 used in the MIPS Creator CI20.
However, the Platform_control abstraction (in src/kern/mips/platform_control- mips.cpp) does seem to provide some flexibility in the way secondary CPUs or cores may be started. It seems that even in situations where the CM is not available, the alloc_secondary_boot_code method will assemble a bootstrap routine from snippets including one, _tramp_mp_cache, that initialises the caches.
Yes, the routines should be flexible in this regards, allowing to do what's necessary.
Unfortunately, it appears that this cache initialisation code does not work on this particular SoC. The method used by_tramp_mp_cache is the familiar one where the cache tags are invalidated for all cache index entries using the Index_Store_Tag operation for each cache type. This is supposed to work on any MIPS CPU, but through some tedious troubleshooting, I discovered that it does not work for the data cache.
If the data cache variant of the Index_Store_Tag operation is performed for a range of addresses starting from the base of KSEG0, I found that some kind of side-effect would occur, possibly affecting the rather critical addresses in that range. This would prevent Fiasco from continuing to start up, and things like timers and the UART would not function, although the processor was not halted and would still run code, illuminating LEDs as a last-resort debugging tool.
There is some preliminary support for this SoC, as well as the JZ4780, in Linux and it seems that instead of using Index_Store_Tag, the tag records are explicitly invalidated and updated. However, it also seems to be the case that caches can be invalidated using Hit_Invalidate. Changing the cache operations in _tramp_mp_cache to use this invalidation operation in its two variants appears to work, avoiding the problems described above.
Does anyone have any thoughts or recollections about this cache initialisation routine?
I'm afraid, I have not.
On the topic of actually starting secondary cores, this SoC has its own set of registers for resetting cores, interrupt control, mailboxes, and so on. I have developed a fairly simple abstraction that performs similar core bootstrapping work to that of the Cm abstraction (in src/kern/mips/cm.cpp). I then extend the Platform_control abstraction so that it will call the alloc_secondary_boot_code method, followed by the appropriate method to start the secondary cores.
Sounds good. If it's just for this single BSP, all the code could also be in there.
Is there any documentation for the SMP architecture in Fiasco and L4Re? I note that interprocessor interrupts (IPI) are one element that I might need to support, probably using the mailboxes.
Yes. some cross-core notification is needed, i.e., some mechanism to trigger an interrupt on another code. Cache-coherency should work. And then it should be fine I guess.
Thanks in advance for any advice or feedback that may be given!
Sorry for the delay. Adam
On 2025-07-20 20:42, Adam Lackorzynski wrote:
On Mon Jun 23, 2025 at 00:34:41 +0200, Paul Boddie wrote:
Does anyone have any thoughts or recollections about this cache initialisation routine?
I'm afraid, I have not.
MIPS cache initialisation always seems to involve recipes and caveats. Perhaps the level 2 cache is the thing that makes this SoC different to various other ones. MIPS is becoming a rather niche architecture now, its one-time corporate guardian now having pivoted to RISC-V and coming under the ownership of GlobalFoundries, who with one of their more advanced fabs must count as some kind of neighbour of yours.
On the topic of actually starting secondary cores, this SoC has its own set of registers for resetting cores, interrupt control, mailboxes, and so on. I have developed a fairly simple abstraction that performs similar core bootstrapping work to that of the Cm abstraction (in src/kern/mips/cm.cpp). I then extend the Platform_control abstraction so that it will call the alloc_secondary_boot_code method, followed by the appropriate method to start the secondary cores.
Sounds good. If it's just for this single BSP, all the code could also be in there.
I will probably try and consolidate the code for this family of SoCs.
Is there any documentation for the SMP architecture in Fiasco and L4Re? I note that interprocessor interrupts (IPI) are one element that I might need to support, probably using the mailboxes.
Yes. some cross-core notification is needed, i.e., some mechanism to trigger an interrupt on another code. Cache-coherency should work. And then it should be fine I guess.
It seems that the IPI mechanism in src/kern/mips/ipi-mips.cpp works fine, or at least sufficiently well for the debugger to use it. I will need to reacquaint myself with the reliability of the debugger in the next few days because there are some problems with the UART on the board I am using that prevent effective testing. The IPI mechanism as provided appears to use some memory as the channel through which information is communicated. It seems that IPI signalling in src/kern/mips/gic.cpp involves the MIPS GIC, but that is not provided by this particular SoC. Instead, the SoC provides a number of core-specific mailboxes with interrupts raised when the mailboxes are written to. I thought that I might replace the implementation in ipi-mips.cpp with something that makes use of these mailboxes more effectively. Currently, a non-zero value written to a mailbox causes an IPI, but I don't make use of this value, instead relying on the existing IPI implementation to set the condition in its _rq array, which appears to maintain the status of the different IPI conditions (which itself employs words to hold boolean values, presumably to facilitate use of the architecture's atomic update primitives). I imagine that I could instead have an IPI implementation that updates the mailbox for a given core, setting a bit in the mailbox register corresponding to the condition. Since there is some form of rudimentary locking provided by the SoC, a read-update-write sequence should be fairly safe. This was not a priority when I just wanted to get it all working. The state of this work reached the point where I think I had implemented core-specific interrupt conditions correctly, along with the appropriate semantics for the set_cpu operation. Here, my mental model is a bit cloudy. If I create an IRQ object in L4Re, I wonder if it is bound to a particular CPU/core or whether it may be made to schedule any CPU/core upon an interrupt condition occurring. (I have been wondering similar things about the Linux kernel where there is a substantial framework for interrupt handling but relatively little coherent documentation about how it all fits together.)
Thanks in advance for any advice or feedback that may be given!
Sorry for the delay.
No worries! I haven't been able to look at this for a while, myself. I appreciate the reply! Paul
participants (2)
-
Adam Lackorzynski -
Paul Boddie