l4-hackers August 1999

l4-hackers@os.inf.tu-dresden.de

12 participants
19 discussions

Re: L4-Linux worked on 486!
by Michael Hohmuth 18 Aug '99

18 Aug '99

Edmund, good news: I think I have found the problem! edmundo(a)rano.demon.co.uk writes: > I've investigated the ready queue after hitting "irq still active" by > doing "p (class thread_t*)0xc0000000" and "p $.ready_next" repeatedly. > In each case the threads corresponding to irqs 5 and 14 seemed to be > ready. In one case, so was the thread corresponding to irq 0. > > Just to confirm: irq 5 = 0xc014a800, irq 14 = 0xc014f000 > > I think I'm dealing with irq 5 when the error occurs because the value > 0x20 is in eax and ebx. Yep. > So presumably I should investigate the kernel stack for irq thread 14, > which was apparently preempted in some mysterious fashion by an > interrupt thread of lower priority. I see there's a kernel_sp in > thread_t. [...] I had a very similar situation, and from looking at the stack of the preempted higher-priority thread, it could tell that the higher-priority thread was voluntarily switching to the lower-priority thread by calling schedule(). However, schedule() should not have switched to the lower-priority thread if the higher-priority thread was runnable... Anyway, a close look at the scheduler revealed a race condition where a high-priority thread could become runnable after it has decided to switch to a lower-priority one. I think I have eliminated that race now. Please try the latest version of thread.cc from CVS (>= 1.51). Thanks again for testing! Michael (Taking a mental note not to throw away my good old 486 so soon because slow machines reveal much more races...) -- hohmuth(a)innocent.com, hohmuth(a)inf.tu-dresden.de http://home.pages.de/~hohmuth/

1 0

Re: L4-Linux worked on 486!
by Michael Hohmuth 18 Aug '99

18 Aug '99

edmundo(a)rano.demon.co.uk writes: > So presumably I should investigate the kernel stack for irq thread 14, > which was apparently preempted in some mysterious fashion by an > interrupt thread of lower priority. I see there's a kernel_sp in > thread_t. Can anyone tell me how to get gdb to analyse that stack for > me, to save me picking out the values that look like program addresses > from the hex dump by hand? Gdb has a command "frame <frame-address>" which should make it possible to select a different stack frame. However, this feature seems to be broken for x86 (at least in Gdb 4.17). I have made a hack to Gdb which lets you use a command "frame <frame-address> <program-counter>"; this command selects a different frame, and you can use "up" and "down" to look at nested or parent frames (sorry, no "backtrace"!). This hack only seems to work if the current frame is valid, but at least it's something to start from. I've appended a patch below. It probably would be worthwile to develop a real fix for Gdb... > I haven't been able to contact os.inf.tu-dresden.de, The network should have been back up since Monday morning. > so I'm still using the same irq.c as before I went on holiday. It > seems to be 1.16 with Michael's patch that was sent to l4-hackers on > 1999-07-23 plus the following patch: I've applied your patch. Thanks! [ from a different email: ] > Also, has anyone other than me experienced the "irq still active" > problem? I have seen mention of a PIC bug in certain Intel 486s; it > would be nice to know for certain that this problem isn't specific to > just my hardware ... No, it's not: I've just been able to reproduce your problem. (This took a while because I first had to fix a different synchronization problem I ran into when I induced the test load you described -- `cksum' + heavy CVS usage on network-mounted disk.) I'll have a look at the problem (and the data you provided -- thanks for digging in!) and post again when I know more. Later, Michael -- hohmuth(a)innocent.com, hohmuth(a)inf.tu-dresden.de http://home.pages.de/~hohmuth/

1 0

"irq still active"
by edmundo＠rano.demon.co.uk 18 Aug '99

18 Aug '99

I've had another look at the kernel_sp for the irq threads after I hit "irq still active" (/linux22/arch/l4-i386/kernel/irq.c:1058). The active threads are: (thread_t *) 0xc0000000 (thread_t *) 0xc014a800 (thread_t *) 0xc014f000 The latter two correspond to tcb 5 0x15 and tcb 5 0x1e, irqs 5 and 14, which are NE2000 and ide0. I rather suspect that I have been barking up the wrong tree by looking at where kernel_sp points at. Although it looks like an execution stack, with a trail of frame pointers and program addresses adjacent to them, I can't get gdb to interpret it for me, and, suspiciously, the values of kernel_sp are equally spaced between the threads, as if all the alleged stacks are exactly the same size, which suggests that they don't have any interesting information for me. tcb 5 0x15 kernel_sp = 0xc014aee4 tcb 5 0x1e kernel_sp = 0xc014f6e4 etc Any other debugging suggestions? Also, has anyone other than me experienced the "irq still active" problem? I have seen mention of a PIC bug in certain Intel 486s; it would be nice to know for certain that this problem isn't specific to just my hardware ... Edmund

1 0

Re: L4-Linux worked on 486!
by Michael Hohmuth 16 Aug '99

16 Aug '99

Edmund GRIMLEY EVANS <edmundo(a)rano.demon.co.uk> writes: > From what you wrote, it sounds like the problem could be solved in > either of two ways: > > - make L4 priorities correspond to PIC priorities > - use a specific instead of a non-specific EOI That's correct. We went for the first solution (because AFAIK, the second one (specific EOI) is not allowed for the mode of the PIC we use -- the special fully nested mode --, or at least it not documented what happens when used in this mode). > Can either of these be implemented as a quick fix even if the long-term > solution is moving irq ack into the microkernel? I've done this already; what you've digged up is my implementation of the first solution: > I notice that linux22/arch/l4-i386/kernel/irq.c has: > > static const > /* prio for irq 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 */ > char irq_prio[] = {1, 0,15, 6, 5, 4, 3, 2,14,13,12,11,10, 9, 8, 7}; > > Can I change this? What to? > > My wild guess would be {15,14,13,4,3,2,1,0,12,11,10,9,8,7,6,5} ... > but there must be some reason for it being the way it is ... This array sorts the priority of the interrupt threads in the order of the hardware priorities. It reflects the priorities after they have been changed by programming the PIC in irq_init.cc: irq_t::init(). Jean already explained why this is the correct order: We have to make sure that IRQ8 (Fiasco's timer interrupt) has the highest priority (i.e., so that it always gets through even if other irqs are in service (i.e., haven't been acknowledged yet)), and that's why IRQ2, the cascade irq for the slave PIC (which hosts IRQ8) gets the highest priority on the master PIC. So, you shouldn't change this array. Now, to debug your problem (some irq's in-service flag is still set after we think we're acknowledged the irq), here are some tips: I suggest you try to find out if there are other irq threads which are in the ready queue and which have been preempted before they have acknowledged their interrupt. In an L4Linux setup, these threads are 5.10 to 5.20 (in hexadecimal task.lthread format, as understood by the built-in kernel debugger Jdb). Then we can try to understand why that thread has been preempted by walking up its kernel stack. Some random hints: In Jdb, you can use `r n' and `r p' to traverse the ready queue (mnemonics: ``ready next'', ``ready prev''). Look at a thread_t by typing something like `t 5.10'. For Gdb, .gdbinit has a macro which can be used like this: "tcb 5 0x10" prints thread's 5.10 thread_t. I've added a command `k p' to Jdb which prints out the state of the PICs. Maybe you can see something suspect there? You can switch back and forth between remote Gdb and Jdb by using "set jdb::use_nested = 0" in Gdb and by pressing `V s' in Jdb. (Of course, all this only works if you haven't completely disabled Jdb with the "-nojdb" command-line switch.) > > There was a long discussion about that and it looks like irq > > acknowledge will move to the micro kernel. > > Was this an e-mail discussion accessible to people like myself, or a > discussion in the coffee room in TU Dresden? An electronic coffee room meant for L4 implementors on various architectures (a private mailing list hosted by USNW). Do you think that more of the internal kernel-design-and-interface discussion should take place in public? If so, we might try to pursuade the other members of that other mailing list to move more of their discussions here. Michael -- hohmuth(a)innocent.com, hohmuth(a)inf.tu-dresden.de http://home.pages.de/~hohmuth/

2 1

Re: L4-Linux worked on 486!
by Michael Hohmuth 12 Aug '99

12 Aug '99

edmundo(a)rano.demon.co.uk writes: > Jochen Liedtke asked, and I think it's a crucial question: > > > are the interrupts constantly disabled from raising the interrupt > >until executing that code? > > Looking at the code, I think the answer is yes. (There is, however, > quite a lot of code between the interrupt happening and the > non-specific EOI being issued, and there are lots of macros and > conditional compilations, so I might be wrong. Unfortunately, you are wrong: Fiasco likes to enable interrupts (because it is a real-time kernel), and there is no guaranty at all that irqs will remain disabled until the EOI is issued (even if the irq thread enters the kernel with interrupts disabled). > Is it possible to inspect the state of the interrupt-disable flag > using remote gdb?) Yes, with "info registers" (or "i r" for short). But when you're in the debugger, it will be too late. > In which case I no longer understand what Jean Wolter wrote about > priority assignment to interrupt threads. If interrupts are > constantly disabled from raising the interrupt until the > non-specific EOI, how can the priorities matter? If the PIC can > "deliver" an interrupt with higher priority while interrupts are > disabled, It can't. > and if the non-specific EOI then acknowledges the highest-priority > interrupt that was delivered, It doesn't, in the sense of "deliver" you defined above. > as opposed to the hightest-priority interrupt that is being > serviced, then we are always in trouble if we issue a non-specific > EOI with interupts disabled. In this case, should we reenable > interrupts before doing the EOI? > > But I can't think why the PIC should have been designed this way. It isn't. Unspecific EOI acknowledges the latest irq that has been accepted by the CPU, and if the CPU is in "cli" mode, it will not accept an irq no matter what its hardware priority in the PIC is. So, given that the CPU operates in "cli" mode, hardware priorities don't matter. But it doesn't operate in "cli" mode at all times. And even if it would, there are problems with the irq threads' software priorities: Unlike L4/x86, Fiasco doesn't deliver an irq-IPC message to an interrupt thread if there is a different thread with a higher priority; instead, the irq ``sender'' is blocked and queued in the irq thread's sender queue. This can lead to a thread different from the irq thread being scheduled and, as a consequence, to interrupts being enabled in the CPU ("sti" mode). > [...] > To reconcile this, I am lead to conclude that my inspection of the > code is mistaken, interrupts are not disabled constantly from raising > the interrupt until issuing the EOI, and this is deliberately so, This is correct. > but somehow we are ending up in the wrong interrupt thread after a > higher-priority interrupts interrupts a lower-priority one. Yep. Maybe some event or IPC caused an lower-priority irq thread to be scheduled before a higher-priority irq thread could acknowledge its irq. > I'm really out of my depth here. Can anyone recommend a good book, or > other source of information, about x86 and PIC programming? (I don't > need an introduction to assembler programming, as I did a lot of 6502 > and 8080 as a child, but I don't know much about x86 specifics or > about interrupt details.) I don't know of any good book describing PIC programming. The best reference I could find was a specification of the ``legacy interrupt controller'' for the PowerPC CHRP platform (which probably emulates the PIC's interface for old x86-PC hardware). But this is only a specification and not a programming manual. Among other places, it's available in Chapter 4 of <URL: ftp://ftp.austin.ibm.com/pub/technology/spec/chrp/inwork/CHRP_IORef_1.0.pdf>. You might also want to try doing a websearch on "programmable interrupt controller programming" or somesuch; I just did a quick search, and <URL:http://www.geocities.com/SiliconValley/Bay/8302/interupt.htm> seems to be a nice intro, but it doesn't discuss the special fully nested mode Fiasco uses. A search on "special fully nested mode" brought up a description of a PCI bridge containing PIC functionality at <URL:http://www.intel.com/design/intarch/techinfo/430tx/ticTOC.htm>. Michael -- hohmuth(a)innocent.com, hohmuth(a)inf.tu-dresden.de http://home.pages.de/~hohmuth/

1 0

Re: L4-Linux worked on 486!
by Michael Hohmuth 12 Aug '99

12 Aug '99

edmundo(a)rano.demon.co.uk writes: > L4-Linux is running quite nicely for me now, but I can still crash it > by doing cksum /usr/bin/* over CVS. It dies after saying "irq still > active". (To get that message I'm using the patch that Michael Hohmuth > posted to the list on July 25.) > > I think I may have found a bug in the SANITY code in > linux22/arch/l4-i386/kernel/irq.c, by the way. [...] > This is what I did to linux22/arch/l4-i386/kernel/irq.c: > > In timer_irq_thread(), I moved > > #ifdef SANITY > irqs_in_progress &= ~(1 << TIMER_IRQ); > #endif > > from its present position just before the call to > l4_i386_ipc_receive() to just before the call to > execute_bottom_halves(). This got rid of the "irq active" messages > which I think are spurious. > > Sorry for the lack of patch but I've rather lost control of the > versions of that file ... My patch you mentioned already contains a similar change... Can you send my (in private email) the output of "cvs diff irq.c" (before you do "cvs update")? Anyway, I've now checked in my sanity-checking changes in linux22/arch/l4-i386/kernel/irq.c. I suggest you use that version for testing. > What I have been doing is telling GDB "symbol-file .../vmlinux" to > inspect what's happening in Linux, then "symbol-file .../kernel.image" > to go back to inspecting Fiasco. There's probably some way of defining > macros to make that switching a bit easier. I don't know if GDB could > use both sets of symbols at the same time. Me neither. > (This is a bit off topic, but does anyone know a better solution to > this one? I tell GDB "p spong" and get a value that is totally wrong. > This is because the variable has been optimised into a register. So to > find out its value I use "disassemble" and "info registers". Is there > a quicker way?) Unfortunately, no, AFAIK. Michael -- hohmuth(a)innocent.com, hohmuth(a)sax.de http://www.sax.de/~hohmuth/

1 0

L4-Linux worked on 486!
by suzukis＠file.phys.tohoku.ac.jp 05 Aug '99

05 Aug '99

Hi, Just I could boot L4-Linux and execute "ls" etc on 486, without rebooting. At present, it seems that Fiasco must be recent version, but the version of L4-Linux does not matter. Both of the previous snapshot (L4-Linux 2.0.21) and the anonymous CVS version (linux22 which I checked out July/26) are working. I remember, after the announcement of linux22 availability via CVS, I experienced several 486 rebooting caused by "ls". So I will continue to search the critical point. After several trying of remote debug, I think "kernel"-debugger is not so much powerful to detect how L4-Linux "server" breaks down. It's not easy for me to understand what happens in the server program from the system calls. Therefore, now I'm looking for a remote-debug-stub for server program on Fiasco. The document of the OSKit tells that the debug-stub in the OSKit is easy to re-use for user-spaced programs, I will try. But there's any working debug-stub? suzuki P.S. By the way, wait_for_keypress() (linux22/drivers/char/tty_io.c) of the latest linux22 works well? It is called when mounting the root filesystem on floppy, or on ramdisk loaded via floppy, like: printk(KERN_NOTICE "VFS: Insert root floppy and press ENTER\n"); wait_for_keypress(); (from linux/fs/super.c). In my testing, the latest linux22 is killed when it calls wait_for_keypress(). But the previous snapshot of L4-Linux (2.0.21) safely passes wait_for_keypress(). The message from the debugger (when linux22 is killed) is like this: grub> kernel=(fd0)/rmgr -nopentium -configfile -sigma0 [Multiboot-elf, <0x100000:0x20320:0x0>,<0x121320:0x418:0x26dac>,entry=0x100000] grub> module=(fd0)/main -nokdb -nojdb [Multiboot-module @ 0x149000, 0x24200 bytes] grub> module=(fd0)/sigma0 [Multiboot-module @ 0x16e000, 0xcb26 bytes] grub> module=(fd0)/rmgr.cfg [Multiboot-module @ 0x17b000, 0x56d bytes] grub> module=(fd0)/glinux.gz init=/bin/sh root=/dev/fd1 [Multiboot-module @ 0x17c000, 0xe428 bytes] RMGR: loading task (fd0)/glinux.gz init=/bin/sh root=/dev/fd1 from 0x17c000-0x260238 to [ 0x3ff000-0x4b2170 0x4b4000-0x4fe034 ] RMGR: starting task (fd0)/glinux.gz init=/bin/sh root=/dev/fd1 from 0x17c000-0x260238 at entry 0x3ff000 via trampoline page code 0x26114c [...snip...] VFS: Insert root floppy and press ENTER Dump of trap_state at 0xc0141fb4: EAX 00000000 EBX 00000001 ECX 004e1646 EDX 00000001 ESI 00000000 EDI 00000000 EBP 00000000 ESP 00d05f5c EIP 00000003 EFLAGS 00013a92 CS 0023 SS 002b DS 002b FS 002b GS 002b trapno 6, error 00000000, from user mode

3 7

Re: L4-Linux worked on 486!
by Jean Wolter 03 Aug '99

03 Aug '99

Edmund GRIMLEY EVANS <edmundo(a)rano.demon.co.uk> writes: > > Thanks for your reply. > > >From what you wrote, it sounds like the problem could be solved in > either of two ways: > > - make L4 priorities correspond to PIC priorities > - use a specific instead of a non-specific EOI > > Can either of these be implemented as a quick fix even if the long-term > solution is moving irq ack into the microkernel? > > I notice that linux22/arch/l4-i386/kernel/irq.c has: > > static const > /* prio for irq 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 */ > char irq_prio[] = {1, 0,15, 6, 5, 4, 3, 2,14,13,12,11,10, 9, 8, 7}; > > Can I change this? What to? It looks like Michael already changed that. The PIC is programmed with irq8 at the highest priority (thats the timer used by L4) and the threads have the same priority. So if you still have problems you should discuss this with Michael. He is currently not in Dresden but IIRC he will be back next week. Jean -- I get up each morning, gather my wits. Pick up the paper, read the obits. if I'm not there I know I'm not dead. So I eat a good breakfast and go back to bed. Peete Seeger

1 0

Re: L4-Linux worked on 486!
by Jean Wolter 03 Aug '99

03 Aug '99

edmundo(a)rano.demon.co.uk writes: > > > >It dies after saying "irq still active". > > >(To get that message I'm using the patch that Michael Hohmuth > > > posted to the list on July 25.) > > > I think I may have found a bug in the SANITY code in > > > linux22/arch/l4-i386/kernel/irq.c, by the way. See below. > > > > I remember, I've ever seen same message & following crash, > > thanks for the patch. > > Unfortunately I still don't understand why the irq is "still active". The problem is the following: Normally the pic only forwards one interrupt to the CPU and signals the next one after the current interrupt has been acknowledged. L4 uses a special PIC mode ("special fully nested mode" or something similar), that allows the pic to signal more than one interrupt to the CPU. It uses priorities to decide whether a newly arrived interrupt should be forwarded to the CPU. So it is possible that more than one interrupt is delivered to the CPU before the interrupt handler was able to acknowledge its own interrupt. If we use "unspecific end of interrupt" to acknowledge the interrupt we acknowledge the one with the highest priority. > > Here's the corresponding bit of code in irq.c. (The lines which aren't > indented as much as they should be came from Michael's debugging > patch.) > > > mask = 1 << (irq & 7); > if (irq < 8) > { > outb(inb(0x21) | mask, 0x21); /* block the irq */ > outb(0x20, 0x20); /* acknowledge the irq */ We use unspecific end of interrupt to acknowledge the irq. > > outb(0x0B, 0x20); > if (inb(0x20) & mask) > enter_kdebug("irq still active"); Now we read the "in service register" of the pic and find out that our interrupt is still active. Due to fiasco being preemptive and a wrong priority assignment to our interrupt threads it happened that a thread with a higher L4 priority which handled an irq with a lower pic priority accidently acknowledged the wrong interrupt. > If you look at the bottom of kernel/fiasco/src/irq.h you'll find an > unused function irq_ack() which does roughly the same thing, because > acknowleding the IRQ is something that ought to be done by L4/Fiasco, > but is at present done by Linux. > > So why do we get "irq still active" during heavy use of the network > card? Is there a PIC expert in the house? I hope the explanation above was clear enough :) > > Is it possible that the problem is caused by the interrupt not being > acknowledged quickly enough? If so, maybe I should move the ack from > Linux into Fiasco ... There was a long discussion about that and it looks like irq acknowledge will move to the micro kernel. Jean -- I get up each morning, gather my wits. Pick up the paper, read the obits. if I'm not there I know I'm not dead. So I eat a good breakfast and go back to bed. Peete Seeger

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

l4-hackers August 1999