Dear Adam,

thanks for your reply!

I'm not sure about the QEMU part (works fine for me)

For the record, this is what I did:

(1) Start with the vanilla base 21.07.0 snapshot.

(2) Manually update the respective components (i.e. src/fiasco, src/l4/mk, src/l4/pkg/bootstrap, etc.) from the respective GitHub repositories. Maybe I've missed something, but I was unable to find any integration repository that would just point to the other repos as submodules or something in that style. And I don't know where is the upstream of things like bin/setup.d and alike.

(3) Compile the resulting composition for arm64-virt-el2. The compilation finishes OK.

(4) Run the compilation output in QEMU. See the log attached. The UART still echoes the input, thus I believe the kernel does not crash, but there is no forward progress.

I am sure that I would be able to find and fix the problem eventually. But I simply prefer a working baseline before doing some development and therefore I have stuck to the vanilla snapshot :)

but on the HiKey my guess would be that the cache on those cores is not enabled? Could you check this?

Like I have written before, I have tried to confirm that the memory is mapped with the correct attributes. The JTAG debugger reports that the memory region where _tramp_mp_spinlock is located in a memory region that is inner shareable, inner write-back, outer write-back, read allocate, write allocate, non-transient.

I have also tried to confirm this from the code:

(a) MAIR_EL2 is set to 0x00ff4400. Which means that the attribute index 2 represents normal cacheable memory.

(b) TCR_EL2 is set to 0x80853510. Which means that the memory is inner shareable, normal outer write-back read allocate write allocate, normal inner write-back read allocate write allocate.

(c) SCTLR_EL2 is set to 0x30c51835. Which means that the instruction cache, the data cache and the memory translation are enabled.

(d) I must say that the code in kern/arm/paging-arm.cpp is extremely hard to understand and analyze (compared to most kernels I've ever seen), not just because it is plagued by non-symbolic constants. But I still believe that the value of 0x008 on the line 835 translates to using the attribute index 2 (see above).

Am I missing something?

Best regards

Martin Decky

Dear Jakub,

Do you need something from the snapshot other than the packages you then update from GitHub?

Yes, I actually need the gnu-efi package to build the UEFI boot images.

If you only need the GitHub packages, the tool for maintaining all of the individual repos at once is called ham and its use is described in our tutorial at:

https://github.com/kernkonzept/manifest/wiki/BUILDING

OK, thanks for the hint! That actually worked nicely and I can build a working L4Re both for QEMU and for HiKey960.

Now I have individual repositories [1][2][3][4] that can closely follow the upstream. Unfortunately, the problem with the additional CPU cores still persists.

[1] https://github.com/martin-decky/fiasco-hikey960 [2] https://github.com/martin-decky/mk-hikey960 [3] https://github.com/martin-decky/bootstrap-hikey960 [4] https://github.com/martin-decky/L4Re-HiKey960-deploy

Best regards

Martin Decky

Dear Jakub,

From the symptoms in the log it seems to me like you were running a virt-enabled kernel with the L4Re build configured for a non-virt-enabled kernel. Can you check that CONFIG_KERNEL_CPU_VIRT is set to y in your .kconfig?

Yes, you are correct. I confirm that this configuration mismatch was the root cause of the issue. I must've overlooked something while I've been updating the files in the snapshot with the files from the upstream repos.

I'm wondering whether it would be useful to have a compile-time (or at least run-time) check to catch such configuration mismatch between Fiasco and L4Re.

Well, updating the snapshot was a poor man's solution anyway and using Ham is IMHO a better approach. Thanks!

Best regards

Martin Decky