Fiasco.OC: sigma0 stucks in ipc-path
Dear kernel-hackers, I think I came along a problem in the IPC-path of Fiasco.OC sometimes in the past, but it was somehow hard to reproduce. But now I've an example that quite reliable triggers the issue. The symptom in the past, and in the concrete example was that our roottask in Genode (called core) was requesting memory from sigma0, either implicitly just by touching some memory-area, like a ROM-module loaded by the bootloader, or explicitly by using the sigma0-protocol, e.g. to request I/O memory for the framebuffer. After that the request was received by sigma0, and also processed, but the answer never reached the faulter/client in this case the core-pager thread. The typical picture in the kernel-debugger then looks like the following: http://pastebin.com/cFvn1NV2 sigma0 is marked as ready, but it still stucks in the ipc-syscall. By instrumenting the kernel I could break down the point where sigma0 never returns. It is when trying to establishing the actual mapping in 'Thread::transfer_msg_items' in file src/kern/thread-ipc.cc. The spot is the following: ``` cpu_lock.clear(); L4_error err = fpage_map(snd->space(), sfp, rcv->space(), L4_fpage(buf->d), item->b, &rl); cpu_lock.lock(); ``` After the cpu lock is given away, sigma0 never aquires it back successfully. It looks to me like a race-condition, nevertheless simply letting the lock being locked doesn't solved the problem ;-). Maybe given your insight knowledge, you're much faster in tracking the problem down to its root? I've to add, that the problem occurred on x86 as well as ARM, on different QEMU versions, as well as real hardware. I also could reproduce it with slightly older versions of Fiasco.OC, and in different development stages of Genode. Nevertheless, it was never that reliable to reproduce. The current example which reproduces it reliable at least with my QEMU (qemu-kvm-0.14.0) version. You can find in form of an ISO-image here: http://dl.dropbox.com/u/82567292/avplay.iso You can try the iso-image like the following: qemu -no-kvm -m 256 -soundhw all -serial mon:stdio -cdrom avplay.iso Or, if you've to build it on your own, here is my topic branch, including the avplay run-script that triggers the problem: https://github.com/skalk/genode/tree/fiasco.oc-ipc-issue To compile and run it yourself you've to do the following steps (after installing the genode toolchain from http://genode.org/download/tool-chain): git clone git@github.com:skalk/genode.git cd genode git checkout -b issue origin/fiasco.oc-ipc-issue git clone git@github.com:genodelabs/linux_drivers.git make -C base-foc prepare make -C libports prepare PKG="libav libc sdl zlib" tool/create_builddir foc_x86_32 BUILD_DIR=build sed -i "/#REPOSITORIES.*libports/s/#//" build/etc/build.conf sed -i "/#REPOSITORIES.*linux_drivers/s/#//" build/etc/build.conf make -C build run/avplay Thank you in advance & best regards! Stefan
Hi, On Thu May 31, 2012 at 18:30:07 +0200, Stefan kalkowski wrote:
I think I came along a problem in the IPC-path of Fiasco.OC sometimes in the past, but it was somehow hard to reproduce. But now I've an example that quite reliable triggers the issue.
The symptom in the past, and in the concrete example was that our roottask in Genode (called core) was requesting memory from sigma0, either implicitly just by touching some memory-area, like a ROM-module loaded by the bootloader, or explicitly by using the sigma0-protocol, e.g. to request I/O memory for the framebuffer. After that the request was received by sigma0, and also processed, but the answer never reached the faulter/client in this case the core-pager thread. The typical picture in the kernel-debugger then looks like the following:
I'm wondering how the higher-prio and ready 'pthread' thread relates to that. Is it always ready and doing something? Adam -- Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
Hi Adam, On 03.06.2012 16:36, Adam Lackorzynski wrote:
Hi,
On Thu May 31, 2012 at 18:30:07 +0200, Stefan kalkowski wrote:
I think I came along a problem in the IPC-path of Fiasco.OC sometimes in the past, but it was somehow hard to reproduce. But now I've an example that quite reliable triggers the issue.
The symptom in the past, and in the concrete example was that our roottask in Genode (called core) was requesting memory from sigma0, either implicitly just by touching some memory-area, like a ROM-module loaded by the bootloader, or explicitly by using the sigma0-protocol, e.g. to request I/O memory for the framebuffer. After that the request was received by sigma0, and also processed, but the answer never reached the faulter/client in this case the core-pager thread. The typical picture in the kernel-debugger then looks like the following:
I'm wondering how the higher-prio and ready 'pthread' thread relates to that. Is it always ready and doing something?
Yes, that's strange. The sigma0 thread is always marked as being ready in the kernel-debugger, but obviously isn't doing progress anymore. Nevertheless, although sigma0 is marked as ready, and has the highest priority, other thread's are still doing progress. For instance, two threads that are using the timer-service are still doing ping-pong IPC with it (can be observed, when enabling IPC-logging after sigma0 stucked). Regards Stefan
Hi Adam, sorry I got you wrong in the first place. I didn't realized, that sigma0 actually runs on a lower priority than any other thread in the system. Obviously the donated timeslice from the core-pager gets fully consumed, and after that it never gets scheduled again, because at any time another thread is active (which is a condition I actually build in with intention to trigger the "bug"). I raised the priority of sigma0 and then it worked. So it seams to be no problem in the Fiasco.OC kernel at all. Nevertheless, it raises the question, whether it would be more reasonable to execute sigma0 on a higher priority by default? Regards Stefan On 03.06.2012 17:54, Stefan kalkowski wrote:
Hi Adam,
On 03.06.2012 16:36, Adam Lackorzynski wrote:
Hi,
On Thu May 31, 2012 at 18:30:07 +0200, Stefan kalkowski wrote:
I think I came along a problem in the IPC-path of Fiasco.OC sometimes in the past, but it was somehow hard to reproduce. But now I've an example that quite reliable triggers the issue.
The symptom in the past, and in the concrete example was that our roottask in Genode (called core) was requesting memory from sigma0, either implicitly just by touching some memory-area, like a ROM-module loaded by the bootloader, or explicitly by using the sigma0-protocol, e.g. to request I/O memory for the framebuffer. After that the request was received by sigma0, and also processed, but the answer never reached the faulter/client in this case the core-pager thread. The typical picture in the kernel-debugger then looks like the following:
I'm wondering how the higher-prio and ready 'pthread' thread relates to that. Is it always ready and doing something?
Yes, that's strange. The sigma0 thread is always marked as being ready in the kernel-debugger, but obviously isn't doing progress anymore. Nevertheless, although sigma0 is marked as ready, and has the highest priority, other thread's are still doing progress. For instance, two threads that are using the timer-service are still doing ping-pong IPC with it (can be observed, when enabling IPC-logging after sigma0 stucked).
Regards Stefan
_______________________________________________ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
Hi, On Mon Jun 04, 2012 at 13:21:19 +0200, Stefan kalkowski wrote:
sorry I got you wrong in the first place. I didn't realized, that sigma0 actually runs on a lower priority than any other thread in the system. Obviously the donated timeslice from the core-pager gets fully consumed, and after that it never gets scheduled again, because at any time another thread is active (which is a condition I actually build in with intention to trigger the "bug").
I raised the priority of sigma0 and then it worked.
So it seams to be no problem in the Fiasco.OC kernel at all. Nevertheless, it raises the question, whether it would be more reasonable to execute sigma0 on a higher priority by default?
Yes, that's a good question. This cannot really happen within L4Re due to the eager memory allocation of moe, so it was never an issue there. I do not really have an opinion on that right now but I'll try to get one. Adam -- Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
Hello Adam, On Mon, Jun 04, 2012 at 09:17:44PM +0200, Adam Lackorzynski wrote:
I do not really have an opinion on that right now but I'll try to get one.
Does it help you finding your opinion, if I mention dynamic discovery and, therefore, mapping of MMIO regions during runtime? Regards -- Christian Helmuth Genode Labs http://www.genode-labs.com/ · http://genode.org/ · /ˈdʒiː.nəʊd/ Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Hi Adam,
Yes, that's a good question. This cannot really happen within L4Re due to the eager memory allocation of moe, so it was never an issue there.
I thought so too because Genode's core pre-allocates all memory from sigma0 at boot time. There are, however, rare cases where the sigma0 protocol is invoked anyway. For example, for handing out MMIO regions to user-level device drivers (roottask doesn't know about those physical regions at boot time) and handing out boot modules. Because the use of the sigma0 protocol at runtime is extremely rare, and the chance for the time slice running out while a sigma0 request is processed even more so, and the chance to have a fully saturated system even more more so, this problem triggered only sporadically. Circa once in a month, and only in highly complex/dynamic scenarios. We have observed it the first time more than 6 months ago. It took us until now to come up with a stable test case that pinpoints the problem.
I do not really have an opinion on that right now but I'll try to get one.
To me the problem looks like a school-book example for priority inversion, which just happened to got covered up pretty nicely by Fiasco's time-slice donation optimization. Do you have an alternative way to fix it other than boost the priority of sigma0? You say that L4Re does not use the sigma0 protocol at runtime. Is there a compelling reason for keeping this little guy called sigma0 lurking around in the system then? If not, getting rid of sigma0 (as done by OKL4 or NOVA, btw) would certainly be the cleanest way to fix our problem. .-) Norman -- Dr.-Ing. Norman Feske Genode Labs http://www.genode-labs.com · http://genode.org Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Hi, On Wed Jun 06, 2012 at 09:06:39 +0200, Norman Feske wrote:
Yes, that's a good question. This cannot really happen within L4Re due to the eager memory allocation of moe, so it was never an issue there.
I thought so too because Genode's core pre-allocates all memory from sigma0 at boot time. There are, however, rare cases where the sigma0 protocol is invoked anyway. For example, for handing out MMIO regions to user-level device drivers (roottask doesn't know about those physical regions at boot time) and handing out boot modules.
Because the use of the sigma0 protocol at runtime is extremely rare, and the chance for the time slice running out while a sigma0 request is processed even more so, and the chance to have a fully saturated system even more more so, this problem triggered only sporadically. Circa once in a month, and only in highly complex/dynamic scenarios. We have observed it the first time more than 6 months ago. It took us until now to come up with a stable test case that pinpoints the problem.
I do not really have an opinion on that right now but I'll try to get one.
To me the problem looks like a school-book example for priority inversion, which just happened to got covered up pretty nicely by Fiasco's time-slice donation optimization. Do you have an alternative way to fix it other than boost the priority of sigma0?
You say that L4Re does not use the sigma0 protocol at runtime. Is there
The issue with mmio is very valid, thanks for the heads-up. Boosting the prio seems the obvious thing to do. Adam -- Adam adam@os.inf.tu-dresden.de Lackorzynski http://os.inf.tu-dresden.de/~adam/
participants (4)
-
Adam Lackorzynski -
Christian Helmuth -
Norman Feske -
Stefan kalkowski