On Tuesday 28. May 2019 18.45.15 Lei Zhou wrote:

HI, I'm looking for the information and procedure how to compile and deploy L4Re on Raspberry PI. Any feedbacks would be greatly appreciated!

I guess you should start here: https://l4re.org/download.html https://l4re.org/build.html https://l4re.org/fiasco/build.html If you are using a modern Debian-based system, you can get away with something like this to obtain all the tools needed for compilation: apt-get install build-essential subversion libncurses5-dev gawk bison flex This is a bit different from what the above pages say: apt-get install make gawk g++ binutils pkg-config g++-multilib subversion \ flex bison But getting things like build-essential is less likely to cause surprises later on. Plus, I found the libncurses5-dev package to be necessary. The online instructions have you downloading and running the repomgr tool to get the sources. If you are a bit cautious, like I am, you can do this in two steps: svn cat https://svn.l4re.org/repos/oc/l4re/trunk/repomgr > repomgr ./repomgr init https://svn.l4re.org/repos/oc/l4re fiasco l4re This lets you inspect the repomgr script and also allows you to keep it around in an obvious place. At this point, you can build Fiasco.OC with something like this: cd src/kernel/fiasco/ make BUILDDIR=mybuild cd mybuild/ make config make Then, building L4Re should be possible by going back up to the src/l4 directory: cd ../../../l4 You may need to create a conf/Makeconf.boot file. I have the following in mine: MODULE_SEARCH_PATH += $(L4DIR_ABS)/../kernel/fiasco/mybuild MODULE_SEARCH_PATH += $(L4DIR_ABS)/conf/examples MODULE_SEARCH_PATH += $(L4DIR_ABS)/pkg/io/io/config BOOTSTRAP_SEARCH_PATH = $(L4DIR_ABS)/conf/examples BOOTSTRAP_SEARCH_PATH += $(L4DIR_ABS)/../kernel/fiasco/mybuild BOOTSTRAP_SEARCH_PATH += $(L4DIR_ABS)/pkg/io/io/config BOOTSTRAP_MODULES_LIST = $(L4DIR_ABS)/conf/modules.list And then you can try and do something like this: make B=mybuild make O=mybuild config make O=mybuild If everything worked, you can try and build an image. This tends to be done as follows: mkdir mybuild/images make O=mybuild uimage E=hello Deploying the image should be a matter of copying a file onto a suitable media device for the Raspberry Pi (SD card, I guess): cp mybuild/images/bootstrap_hello.uimage /media/sdcard Here, only you will really know where to copy the file to. I hope this helps! Paul

Hi, On [28-05-2019 21:21], Lei Zhou wrote:

Thanks Paul for your prompt response. I will give it try and see how it goes. Regards, Lei

I just want to add two things to Paul's very good description. First, have you checked our build instructions and how tos on our github [1] ? And second, which Raspberry Pi model are you targeting? Newer (> gcc 4.7) cross-toolchains seem to have dropped support for ARMv6k. While you still can compile L4Re/Fiasco for that architecture the binaries will not run later. Best, Matthias. [1] https://github.com/kernkonzept/manifest/wiki

________________________________________ From: l4-hackers [l4-hackers-bounces@os.inf.tu-dresden.de] on behalf of Paul Boddie [paul@boddie.org.uk] Sent: Tuesday, May 28, 2019 5:09 PM To: l4-hackers@os.inf.tu-dresden.de Subject: Re: Want to run L4Re on Raspberry PI.

On Tuesday 28. May 2019 18.45.15 Lei Zhou wrote:

...

HI, I'm looking for the information and procedure how to compile and deploy L4Re on Raspberry PI. Any feedbacks would be greatly appreciated!

I guess you should start here:

https://urldefense.proofpoint.com/v2/url?u=https-3A__l4re.org_download.html&... https://urldefense.proofpoint.com/v2/url?u=https-3A__l4re.org_build.html&d=D... https://urldefense.proofpoint.com/v2/url?u=https-3A__l4re.org_fiasco_build.h...

If you are using a modern Debian-based system, you can get away with something like this to obtain all the tools needed for compilation:

apt-get install build-essential subversion libncurses5-dev gawk bison flex

This is a bit different from what the above pages say:

apt-get install make gawk g++ binutils pkg-config g++-multilib subversion \ flex bison

But getting things like build-essential is less likely to cause surprises later on. Plus, I found the libncurses5-dev package to be necessary.

The online instructions have you downloading and running the repomgr tool to get the sources. If you are a bit cautious, like I am, you can do this in two steps:

svn cat https://urldefense.proofpoint.com/v2/url?u=https-3A__svn.l4re.org_repos_oc_l... > repomgr ./repomgr init https://urldefense.proofpoint.com/v2/url?u=https-3A__svn.l4re.org_repos_oc_l... fiasco l4re

This lets you inspect the repomgr script and also allows you to keep it around in an obvious place.

At this point, you can build Fiasco.OC with something like this:

cd src/kernel/fiasco/ make BUILDDIR=mybuild cd mybuild/ make config make

Then, building L4Re should be possible by going back up to the src/l4 directory:

cd ../../../l4

You may need to create a conf/Makeconf.boot file. I have the following in mine:

MODULE_SEARCH_PATH += $(L4DIR_ABS)/../kernel/fiasco/mybuild MODULE_SEARCH_PATH += $(L4DIR_ABS)/conf/examples MODULE_SEARCH_PATH += $(L4DIR_ABS)/pkg/io/io/config BOOTSTRAP_SEARCH_PATH = $(L4DIR_ABS)/conf/examples BOOTSTRAP_SEARCH_PATH += $(L4DIR_ABS)/../kernel/fiasco/mybuild BOOTSTRAP_SEARCH_PATH += $(L4DIR_ABS)/pkg/io/io/config BOOTSTRAP_MODULES_LIST = $(L4DIR_ABS)/conf/modules.list

And then you can try and do something like this:

make B=mybuild make O=mybuild config make O=mybuild

If everything worked, you can try and build an image. This tends to be done as follows:

mkdir mybuild/images make O=mybuild uimage E=hello

Deploying the image should be a matter of copying a file onto a suitable media device for the Raspberry Pi (SD card, I guess):

cp mybuild/images/bootstrap_hello.uimage /media/sdcard

Here, only you will really know where to copy the file to.

I hope this helps!

Paul

_______________________________________________ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de https://urldefense.proofpoint.com/v2/url?u=http-3A__os.inf.tu-2Ddresden.de_m...

_______________________________________________ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

-- Matthias Lange, matthias.lange@kernkonzept.com, +49-351-41 888 614 Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. Geschäftsführer: Dr.-Ing. Michael Hohmuth

On Wednesday 29. May 2019 08.39.11 Matthias Lange wrote:

On [28-05-2019 21:21], Lei Zhou wrote:

...

Thanks Paul for your prompt response. I will give it try and see how it goes. Regards, Lei I just want to add two things to Paul's very good description.

First, have you checked our build instructions and how tos on our github [1] ?

Maybe there should be some updates to l4re.org to communicate any new advice.

And second, which Raspberry Pi model are you targeting? Newer (> gcc 4.7) cross-toolchains seem to have dropped support for ARMv6k. While you still can compile L4Re/Fiasco for that architecture the binaries will not run later.

One thing I did forget to mention was the cross-toolchains, which was regrettable. I believe that the Debian toolchains in Debian 9 (Stretch), 10 (Buster) or unstable (Sid) should be good enough, but as is noted above, the architecture variant of earlier Raspberry Pi models (and the Pi Zero) is not supported in Debian any more (which is why Raspbian exists). I suppose that L4Re could be compiled natively within Raspbian if that is already deployed. I don't remember if there are any complications for ARM about code output, but looking at the src/l4/mk/arch/Makeconf.arm, it seems that a fairly standard arm-linux-gnueabihf toolchain is indicated. (I suppose I could be thinking about certain "bare metal" configurations requiring differently-configured compilers.) Otherwise, I had some success with Buildroot in generating toolchains, these being needed for my soft-float MIPS configuration. Paul

[1] https://github.com/kernkonzept/manifest/wiki

Thanks Paul and Matthias for valuable inputs and comments, which greatly save me a lot of time to searching and googling. I am using pi2 or pi3. Has been mostly collecting various information and today will try based on all your inputs. Much appreciated. Lei Sent from my BlackBerry — the most secure mobile device — via the Bell Network From: matthias.lange@kernkonzept.com Sent: May 29, 2019 9:20 AM To: l4-hackers@os.inf.tu-dresden.de Cc: lezhou@blackberry.com; paul@boddie.org.uk Subject: Re: Want to run L4Re on Raspberry PI. On [29-05-2019 14:35], Paul Boddie wrote:

On Wednesday 29. May 2019 08.39.11 Matthias Lange wrote:

...

On [28-05-2019 21:21], Lei Zhou wrote:

...

Thanks Paul for your prompt response. I will give it try and see how it goes. Regards, Lei I just want to add two things to Paul's very good description.

First, have you checked our build instructions and how tos on our github [1] ?

Maybe there should be some updates to l4re.org to communicate any new advice.

...

And second, which Raspberry Pi model are you targeting? Newer (> gcc 4.7) cross-toolchains seem to have dropped support for ARMv6k. While you still can compile L4Re/Fiasco for that architecture the binaries will not run later.

One thing I did forget to mention was the cross-toolchains, which was regrettable. I believe that the Debian toolchains in Debian 9 (Stretch), 10 (Buster) or unstable (Sid) should be good enough, but as is noted above, the architecture variant of earlier Raspberry Pi models (and the Pi Zero) is not supported in Debian any more (which is why Raspbian exists).

I suppose that L4Re could be compiled natively within Raspbian if that is already deployed. I don't remember if there are any complications for ARM about code output, but looking at the src/l4/mk/arch/Makeconf.arm, it seems that a fairly standard arm-linux-gnueabihf toolchain is indicated. (I suppose I could be thinking about certain "bare metal" configurations requiring differently-configured compilers.)

The issue we have with "standard" cross toolchains is the libgcc they are shipping. It is compiled for ARMv7 and contains instructions that are unknown / illegal on ARMv6k. The mean thing is, that our build system tells GCC via "-march=armv6zk" what code to generate but then links the wrong libgcc. Matthias.

Otherwise, I had some success with Buildroot in generating toolchains, these being needed for my soft-float MIPS configuration.

Paul

...

[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kernkonzept_...

-- Matthias Lange, matthias.lange@kernkonzept.com, +49-351-41 888 614 Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. Geschäftsführer: Dr.-Ing. Michael Hohmuth

On Wednesday 29. May 2019 15.20.01 Matthias Lange wrote:

The issue we have with "standard" cross toolchains is the libgcc they are shipping. It is compiled for ARMv7 and contains instructions that are unknown / illegal on ARMv6k. The mean thing is, that our build system tells GCC via "-march=armv6zk" what code to generate but then links the wrong libgcc.

Some of these compiler configuration issues are awkward. My problem was that although there are switches for hard- and soft-float, appropriate libgcc variants need to be available, and the Debian toolchains do not provide them all. I actually don't remember if there is a need for completely separate compilers, but I would hope not (and yet not be surprised if it were the case). Of course, it is one thing to be able to avoid certain instructions in one's own code, quite another to have to deal with library code liberally using "illegal" instructions. I got quite far with bare metal code, even targeting microcontrollers like the PIC32 with the MIPS compilers, for instance. So, the Buildroot compiler it had to be. Paul P.S. I have a selfish interest in following this as I could imagine also trying out L4Re on the Raspberry Pi at some point. I guess that the framebuffer isn't currently supported though.

Another error I'm running into is at last step to generate bootable uimage. $make E=hello uimage MODULE_SEARCH_PATH="${PWD}/../../kernel/fiasco/build-ukernel-pi3" PLATFORM_TYPE=rpi_b I have two questions: 1. What RAM_BASE should I pass in make cmdline? or leave it as is? 2. Seems missing mkimage tool in the src tree. I used repomgr pulling all the source automatically. What did I miss here? ------------------------------------------------------------------------------------------------------------------------------------------------ make[1]: Entering directory '/home/lezhou/workspace/l4re/src/l4' =========== Updating RAM_BASE for platform rpi_b to 0x0 ========= ... Regenerating RAM_BASE settings [sigma0] ==> Linking sigma0 [sigma0] ==> sigma0 built [sigma0] ==> Installing sigma0 to local build-tree [moe] ==> Linking moe [moe] ==> moe built [moe] ==> Installing moe to local build-tree /home/lezhou/workspace/l4re/src/l4/pkg/bootstrap/server/src/Make.rules:263: *** mkimage(mkimage) host tool missing, cannot build bootstrap.uimage/itb. Stop. ../../../../mk/binary.inc:149: recipe for target '/home/lezhou/workspace/l4re/src/l4/build-pi3/pkg/bootstrap/server/src/OBJ-arm64_armv8a' failed make[2]: *** [/home/lezhou/workspace/l4re/src/l4/build-pi3/pkg/bootstrap/server/src/OBJ-arm64_armv8a] Error 2 Makefile:537: recipe for target 'uimage' failed make[1]: *** [uimage] Error 2 make[1]: Leaving directory '/home/lezhou/workspace/l4re/src/l4' Makefile:6: recipe for target 'do-all-make-goals' failed make: *** [do-all-make-goals] Error 2 ------------------------------------------------------------------------------------------------------------------------------------------------------------- Thanks, Lei ________________________________________ From: Lei Zhou Sent: Wednesday, May 29, 2019 4:39 PM To: Paul Boddie; l4-hackers@os.inf.tu-dresden.de Subject: RE: Want to run L4Re on Raspberry PI. When I cross compiled Fiasco for Raspberry PI3 B, running into this compiling error as below. Here are what I did: 1. use Linaro cross compiler 2. // cross compiling for Rpi 3b+ $make BUILDDIR=build-rpi3-ukernel $make O=build-rpi3-ukernel config // pick aarch64, broadcom 2837, and rpi 3b $make O=build-rpi3-ukernel -j4 Compiling error: ======================== ... Making arm_control-arm-bcm283x.o ... Making irq_handler-arm-bcm283x.o ... Making uart_console.o ... Making vgic.o ==> Linking tcboffset.bin ... Making jdb.o In file included from auto/vgic.cc:4:0: /home/lezhou/workspace/l4re/src/kernel/fiasco/src/kern/arm/vgic.cpp: In constructor ‘Gic_h_init::Gic_h_init(Cpu_number)’: /home/lezhou/workspace/l4re/src/kernel/fiasco/src/kern/arm/vgic.cpp:153:59: error: ‘Gic_h_phys_base’ is not a member of ‘Mem_layout’ Gic_h::gic.construct(Kmem::mmio_remap(Mem_layout::Gic_h_phys_base)); ^~~~~~~~~~~~~~~ /home/lezhou/workspace/l4re/src/kernel/fiasco/src/Makerules.global:118: recipe for target 'vgic.o' failed <<<====== compiling error make[2]: *** [vgic.o] Error 1 make[2]: *** Waiting for unfinished jobs.... make[2]: Leaving directory '/home/lezhou/workspace/l4re/src/kernel/fiasco/build-ukernel-pi3' /home/lezhou/workspace/l4re/src/kernel/fiasco/src/Makefile:153: recipe for target 'all' failed ====================================================================== Any inputs would be appreciated! Lei ________________________________________ From: l4-hackers [l4-hackers-bounces@os.inf.tu-dresden.de] on behalf of Paul Boddie [paul@boddie.org.uk] Sent: Wednesday, May 29, 2019 12:27 PM To: l4-hackers@os.inf.tu-dresden.de Subject: Re: Want to run L4Re on Raspberry PI. On Wednesday 29. May 2019 15.20.01 Matthias Lange wrote:

The issue we have with "standard" cross toolchains is the libgcc they are shipping. It is compiled for ARMv7 and contains instructions that are unknown / illegal on ARMv6k. The mean thing is, that our build system tells GCC via "-march=armv6zk" what code to generate but then links the wrong libgcc.

Some of these compiler configuration issues are awkward. My problem was that although there are switches for hard- and soft-float, appropriate libgcc variants need to be available, and the Debian toolchains do not provide them all. I actually don't remember if there is a need for completely separate compilers, but I would hope not (and yet not be surprised if it were the case). Of course, it is one thing to be able to avoid certain instructions in one's own code, quite another to have to deal with library code liberally using "illegal" instructions. I got quite far with bare metal code, even targeting microcontrollers like the PIC32 with the MIPS compilers, for instance. So, the Buildroot compiler it had to be. Paul P.S. I have a selfish interest in following this as I could imagine also trying out L4Re on the Raspberry Pi at some point. I guess that the framebuffer isn't currently supported though. _______________________________________________ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de https://urldefense.proofpoint.com/v2/url?u=http-3A__os.inf.tu-2Ddresden.de_m...

On Wednesday 29. May 2019 18.27.03 Paul Boddie wrote:

P.S. I have a selfish interest in following this as I could imagine also trying out L4Re on the Raspberry Pi at some point. I guess that the framebuffer isn't currently supported though.

Well, I got round to trying this, but it hasn't been entirely successful. First of all, I made a toolchain using Buildroot 2019.02.2 and what are effectively the raspberrypi0_defconfig settings. The generated gcc is version 7.4.0, ld is version 2.29.1. Note that I am using the Zero, not the Zero-W, model. This doesn't appear to have any real toolchain implications, as far as I can tell from looking at the configurations. Some pertinent settings: BR2_GCC_TARGET_ABI="aapcs-linux" BR2_GCC_TARGET_CPU="arm1176jzf-s" BR2_GCC_TARGET_FPU="vfp" BR2_GCC_TARGET_FLOAT_ABI="hard" BR2_GCC_TARGET_MODE="arm" BR2_ARM_CPU_HAS_FPU=y BR2_ARM_CPU_HAS_VFPV2=y BR2_ARM_CPU_HAS_ARM=y BR2_ARM_CPU_HAS_THUMB=y BR2_ARM_CPU_ARMV6=y BR2_arm1176jzf_s=y BR2_ARM_EABIHF=y BR2_ARM_FPU_VFPV2=y Then, I built Fiasco with the Raspberry Pi Zero W selected: CONFIG_PF_BCM283X=y CONFIG_BSP_NAME="bcm283x" CONFIG_CAN_ARM_CPU_1176=y CONFIG_ARM_V6=y CONFIG_ARM_V6PLUS=y CONFIG_PF_BCM283X_RPIZW=y CONFIG_ABI_VF=y CONFIG_ARM_1176=y CONFIG_FPU=y CONFIG_LAZY_FPU=y CONFIG_ARM_1176_CACHE_ALIAS_FIX=y CONFIG_ARM_CPU_ERRATA=y In this configuration, I also selected things that appeared useful, such as FPU support and the fixes/errata, which might not have been appropriate. Finally, I built L4Re with the Raspberry Pi Model B selected as platform: CONFIG_CPU_ARM_ARMV6ZK=y CONFIG_CPU="armv6zk" CONFIG_PLATFORM_TYPE="rpi_b" CONFIG_CPU_ARMV6KPLUS=y CONFIG_CPU_ARMV6PLUS=y CONFIG_PLATFORM_TYPE_rpi_b=y Having built the "hello" example, deploying the bootstrap_hello.raw file to the microSD card as hello.raw, alongside the extra files that the Raspberry Pi's proprietary bootloader uses, with a config.txt file that indicates the kernel... kernel=hello.raw ...I see the payload starting over the UART connection, but it appears that the board keeps restarting, meaning that I just see the following over and over again: L4 Bootstrapper Build: #8 Mon Jun 10 19:44:17 CEST 2019, 7.4.0 Scanning up to 512 MB RAM, starting at offset 32MB Memory size is 512MB (00000000 - 1fffffff) RAM: 0000000000000000 - 000000001fffffff: 524288kB Total RAM: 512MB Scanning fiasco Scanning sigma0 Scanning moe Moving up to 5 modules behind 1100000 Instrumenting the boot_modules.cc file in the bootstrap package, I see that the Boot_modules::move_modules is not completed. Generally, modules appear to be identified, but some other operation is causing a problem that appears to cause the board to reset. I have also tried building Fiasco for first generation Raspberry Pi devices along with the FPU support and other special options. And I have tried building for the Pi Zero-W without these extra options. The result seems to be the same, so I suspect that the kernel plays no role here, particularly since the problem appears to lie in the bootstrap code. I wonder if others who have looked at Raspberry Pi hardware might be able to share their experiences. It is rather frustrating to have to piece together the details of building and deploying for hardware ostensibly supported by L4Re. Paul

On Tuesday 11. June 2019 00.08.59 Adam Lackorzynski wrote:

If it just reboots in the middle of a rather normal C function, then it's likely some instruction that has been generated there and not understood by the CPU. Could you maybe use u-boot as it shows a register dump when something like this happens, allowing to know where it branched away.

I think that introducing U-Boot may save me some time in the end, so I'll look into that. The Raspberry Pi stuff is rather alien to me, but I had hoped that there was an established recipe for getting things up and running. Thanks for the suggestion! Paul

On Tue Jun 11, 2019 at 02:00:00 +0200, Paul Boddie wrote:

On Tuesday 11. June 2019 00.18.43 Paul Boddie wrote:

...

On Tuesday 11. June 2019 00.08.59 Adam Lackorzynski wrote:

...

If it just reboots in the middle of a rather normal C function, then it's likely some instruction that has been generated there and not understood by the CPU. Could you maybe use u-boot as it shows a register dump when something like this happens, allowing to know where it branched away.

I think that introducing U-Boot may save me some time in the end, so I'll look into that. The Raspberry Pi stuff is rather alien to me, but I had hoped that there was an established recipe for getting things up and running.

Well, with config.txt updated with...

kernel=u-boot.bin

...I get a prompt on the UART and can tell U-Boot to load and start the payload, according to some instructions I found online:

https://ltekieli.com/buildroot-with-raspberry-pi-u-boot/

Here is the conversation:

----

U-Boot> mmc dev 0 switch to partitions #0, OK mmc0 is current device U-Boot> fatload mmc 0:1 ${kernel_addr_r} hello.raw 1209560 bytes read in 91 ms (12.7 MiB/s) U-Boot> bootz ${kernel_addr_r} Kernel image @ 0x080000 [ 0x1000000 - 0x11274d8 ]

Interesting way of doing it, i.e. good that our bootstrap image also looks like a Linux image. I'd typically do something like this using the uimage: fatload mmc 0:1 0x0ffffc0 hello.uimage bootm 0x0ffffc0 But in the end it should not make a difference.

Starting kernel ...

L4 Bootstrapper Build: #8 Mon Jun 10 19:44:17 CEST 2019, 7.4.0 Scanning up to 512 MB RAM, starting at offset 32MB Memory size is 512MB (00000000 - 1fffffff) RAM: 0000000000000000 - 000000001fffffff: 524288kB Total RAM: 512MB Scanning fiasco Scanning sigma0 Scanning moe Moving up to 5 modules behind 1100000

----

Unfortunately, nothing more is shown and there is no register dump. Maybe I have to compile in some kind of exception handler or something.

No, it's built-in. So my suggestion would be to narrow it down by putting some more printfs etc. in the code so see where it diverts. But understandably this is pretty cumbersome with the SD card. Could you put the hello.elf online so that I could have a look to check for the assumption with the wrong instruction? Adam

On Wed Jun 12, 2019 at 00:34:02 +0200, Paul Boddie wrote:

On Tuesday 11. June 2019 23.32.43 Adam Lackorzynski wrote:

...

On Tue Jun 11, 2019 at 02:00:00 +0200, Paul Boddie wrote:

...

Unfortunately, nothing more is shown and there is no register dump. Maybe I have to compile in some kind of exception handler or something.

No, it's built-in. So my suggestion would be to narrow it down by putting some more printfs etc. in the code so see where it diverts. But understandably this is pretty cumbersome with the SD card.

Indeed, but it was more cumbersome blitting patterns to the display on the Ben NanoNote to trace progress through the bootstrap code and kernel. :-)

Previously, I did generate debugging statements, and initially it seemed that the code didn't get beyond calc_modules_size, seemingly failing to loop over more than one module. However, splitting up the following statement...

s += l4_round_size(mbi->module(i).size(), p2align);

...and inserting other debugging statements did seem to persuade the code to proceed until move_module was called. Thus, output like this can be generated:

Counting 5 modules... Module 0... Size: 648904... Module 1... Size: 41840... Module 2... Size: 218336... Module 3... Size: 87208... Module 4... Size: 132812... Total: 1142784

In fact, this is from my most recent attempt at instrumenting the code in this way.

...

Could you put the hello.elf online so that I could have a look to check for the assumption with the wrong instruction?

Perhaps take a look at the following:

http://www.boddie.org.uk/downloads/armv6_hello.elf http://www.boddie.org.uk/downloads/armv6zk_hello.elf

I see a couple of floating point instructions and it would be interesting to know whether one of those triggers it. Sorry for not saying this earlier, but could you place a version with debugging symbols (of bootstrap) online, i.e. one where I could see where move_modules is. Thanks, Adam

I rebuilt the toolchain using the raspberrypi0_defconfig instead of manually selecting Raspberry Pi Zero in the configuration menu, just to make sure that there wasn't some important difference.

Then, I rebuilt Fiasco and L4Re. The ARMv6 and ARMv6zK payloads were generated from separate ARMv6 and ARMv6zK builds of L4Re.

The result is unchanged from before, meaning that either the generated code has no influence on the result - it does differ slightly, but mostly involves the usual reordering done when such settings are altered - or "bad" instructions are somehow being generated anyway.

Thanks for taking an interest in this! I imagine that I am probably doing something wrong, but hopefully my mistakes can be used to teach others what to do.

Hi, On Thu Jun 13, 2019 at 19:36:33 +0200, Paul Boddie wrote:

On Thursday 13. June 2019 17.48.59 Adam Lackorzynski wrote:

...

On Wed Jun 12, 2019 at 00:34:02 +0200, Paul Boddie wrote:

...

Perhaps take a look at the following:

http://www.boddie.org.uk/downloads/armv6_hello.elf http://www.boddie.org.uk/downloads/armv6zk_hello.elf

I see a couple of floating point instructions and it would be interesting to know whether one of those triggers it. Sorry for not saying this earlier, but could you place a version with debugging symbols (of bootstrap) online, i.e. one where I could see where move_modules is.

To do this, I ran the following command:

make O=mybuild E=hello uimage BOOTSTRAP_NO_STRIP=1

I had been looking in the general build configuration, but the "Strip binaries on install" (BID_STRIP_PROGS) option evidently only applies to bundled programs, not the bootstrap payload itself.

The above binaries have now been replaced with ones that include symbol information. Previously, I had been looking at the dump for boot_modules.o and didn't notice anything particularly unusual, but I must admit that I don't know the more exotic ARM instructions.

I do not see anything particular special either. The floating point instructions are in the libc/*printf, so are not affected here. I could only continue with a binary search style of looking where it breaks, with "while(1);" as printfs do seems to change too much as you already found out. Adam

Hi Paul, the cross compiler shipped with Debian silently produce code for armv7, even if you specify -march=armv6zk. It just ignores that. I also had to find this out the hard way. My only solution was to create a cross compiler myself and configure it with armv6 support. Here are (maybe incomplete) instructions for doing that: Note: gcc 6.4 has a bug which means it will not build on latest Debian (​https://github.com/crosstool-ng/crosstool-ng/issues/992). This is why we select the gcc 5.4 version below. Install all the requirements:: $ sudo apt-get install bison cvs flex gperf texinfo automake libtool html2man Download and unpack crosstool-ng: $ wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.23.0.tar.bz2 $ tar -xjvf crosstool-ng-1.23.0.tar.bz2 $ cd crosstool-ng-1.23.0 Configure and build it: $ ./configure --prefix=$PWD/../crosstool-ng $ make $ make install $ export PATH=$PATH:$PWD/../crosstool-ng $ cd .. $ mkdir raspi $ cd raspi $ ct-ng armv6-rpi-linux-gnueabi $ ct-ng menuconfig Now change some options which are still not right: Path and misc options: -> Render the toolchain read-only: OFF C-Library: -> Build and install locales: OFF C Compiler: -> gcc version: linaro-5.4-2017.01 Debug facilities: -> gdb: OFF Build it: $ ct-ng build The result will be stored in ${HOME}/x-tools/x-tools/armv6-rpi-linux-gnueabi. To build l4re/fiasco you need to refer to the new cross compiler: $ export CROSS_COMPILE=${HOME}/x-tools/x-tools/armv6-rpi-linux-gnueabi/bin/armv6-rpi-linux-gnueabi- Chris On 6/17/19 11:37 PM, Paul Boddie wrote:

On Monday 17. June 2019 00.08.27 Adam Lackorzynski wrote:

...

I do not see anything particular special either. The floating point instructions are in the libc/*printf, so are not affected here. I could only continue with a binary search style of looking where it breaks, with "while(1);" as printfs do seems to change too much as you already found out.

Well, I also tried with the Debian ARM cross-toolchains (GCC 7.4.0), just to see whether they might produce differently-behaving code. The boot process only gets this far with the generated code...

---- ## Booting kernel from Legacy Image at 00080000 ... Image Name: L4 Image #2 Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 1226456 Bytes = 1.2 MiB Load Address: 01000000 Entry Point: 01000000 Verifying Checksum ... OK Loading Kernel Image ... OK

Starting kernel ...

L4 Bootstrapper ----

However, I do wonder whether the compiler options might be inappropriate. I had assumed that some CPU-specific options may have been introduced into the compiler invocations, but the Raspberry Pi configuration options in L4Re don't seem to have any such effect (these presumably being in mk/platform).

Meanwhile, Fiasco introduces -mcpu=arm1176jzf-s as an option (in src/Makeconf.arm) thanks to the Raspberry Pi definitions (in src/kern/arm/bsp/bcm283x/Kconfig) setting CAN_ARM_CPU_1176 and thus ARM_1176 then being set (in src/kern/arm/Kconfig).

Presumably, the -march=armv6zk option set within L4Re (in mk/arch/Makeconf.arm) thanks to CPU_ARM_ARMV6ZK (in mk/arch/Kconfig.arm.inc) is supposed to cover the Broadcom 2835 in the Raspberry Pi Zero.

Looking for guidance on this, I found the following:

https://raspberrypi.stackexchange.com/questions/2046/which-cpu-flags-are-sui...

Here, the following options are suggested:

-march=armv6zk -mcpu=arm1176jzf-s -mfloat-abi=hard -mfpu=vfp

To investigate, I changed CARCHFLAGS_arm_armv6zk (in mk/arch/Makeconf.arm) to incorporate these extra options and rebuilt L4Re. The resulting code is different, with various tables containing somewhat different values, and with some small additions and relocations amongst the routines.

Since this exercise involved the Debian toolchain, the resulting output did not indicate any more progress through the bootstrapping process than before, but I do wonder if things might change more substantially by using these flags with the "clean" Buildroot toolchain I prepared earlier.

I guess trying to build everything as soft-float might also be worthwhile, but I do doubt that floating point instructions are the real problem here. The odd behaviour obtained by placing printf statements in the failing routines almost suggests some kind of initialisation or referencing issue, perhaps linker- related.

Paul

_______________________________________________ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

-- Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. Geschäftsführer: Dr.-Ing. Michael Hohmuth

On Monday 24. June 2019 20.03.26 Paul Boddie wrote:

[Crosstool-NG instructions]

Thanks for the instructions! As you may have seen, I did use Buildroot to make a comparable toolchain, but maybe I need to investigate whether the two different systems produce toolchains with differing characteristics. A few aspects come to mind.

A quick look at the latest Crosstool-NG seemed to indicate that the expected compiler/linker options would be set (in samples/armv6-rpi-linux- gnueabi/crosstool.config): CT_ARCH_CPU="arm1176jzf-s" CT_ARCH_SUFFIX="v6" CT_ARCH_FPU="vfp" This shouldn't be different from what Buildroot seems to set up (in configs/raspberrypi0_defconfig): BR2_arm1176jzf_s=y BR2_ARM_EABIHF=y I did wonder if Crosstool-NG might drop the floating point support and generate soft-float code in its toolchain output, but I guess it doesn't. Unfortunately, I didn't manage to finish building the toolchain yesterday (after ct-ng pegged my CPU and disk for several hours, helpfully assisted occasionally by Firefox and Akonadi/MySQL), and attempting to restart the build today caused it to throw away the build results from yesterday and start again. (Not exactly what you would expect from a Makefile-based system.)

GCC versions and regressions with regard to the kind of code they generate. If the Debian compiler ignores the architecture option due to an upstream issue, this might also explain why your Crosstool-NG toolchain (GCC 5.4) works but the toolchain produced by Buildroot (GCC 7.4) does not.

I still wonder about this, but only slightly. One thing I did notice is that the latest Crosstool-NG does not offer the Linaro compiler variant, although my impression is that the Linaro toolchain is largely obsolete.

Configuration differences: I would assume that the Buildroot compiler would actually work with the Raspberry Pi Zero given the presence of a suitable "defconfig" file, but maybe it is broken and nobody is testing it any more.

L4Re versions and regressions: presumably L4Re has worked on the original Raspberry Pi and on the Raspberry Pi Zero W, given that the latter appears in the Fiasco configuration menus. Maybe I should rewind to some earlier L4Re (and Fiasco) repository version and try the result.

I tried to do this, but version 76 of L4Re was the earliest that would cross- build from my MIPS-based computer. Versions 69 and 74 experienced the issues with "make B=mybuild" that I described recently, but I couldn't work around them within a reasonable amount of time.

I was also told about floating point instruction choices with regard to the Inferno toolchain output, these causing problems when someone ported Inferno to the Raspberry Pi. Obviously, the exact problems would be specific to Inferno, but I can imagine "bad" instructions leaking out from some kind of GCC regression.

One thing I did discover about this was the apparent need to enable the floating point unit explicitly. There are various articles mentioning this such as... http://cholla.mmto.org/arm/assembly/float.html Adding the mrc...mcr instruction block to the crt0.S file didn't help, however, making me think that this isn't completely related to floating point instructions.

Sadly, U-Boot doesn't seem to play along by setting a trap handler, but perhaps I can install one in the initialisation of the bootstrap module. Alternatively, I could build a soft-float toolchain and try and see if that makes any difference.

So I suppose that there are a few things to look at here, but I appreciate the suggestion of trying a different toolchain system, and maybe that's the first thing I will look at.

Well, I may get back to this, but what seemed like a interesting diversion has turned into a larger and less rewarding exercise. As I experienced with the MIPS-related parts of L4Re, there seems to be a gap between claimed and actual support for hardware that, in this case, may only be a matter of documentation. However, I rather feel that L4Re has a broader reproducibility problem. Paul