Re: A question about the softirq implementation in DDE Linux26
Hi, On 10-2-27 上午9:10, Dirk Vogt wrote:
Hej, You're right. There is indeed a problem, which we haven't stumbled over yet. That is, because the irq handler runs at at a higher priority in our ddekit implementation. However on a multiprocessor machine this I don't understand. How can using a higher priority in the irq handler solve/avoid the problem? unless you mean running the softirq thread in a higher priority. Otherwise, the softirq thread can still be preempted and lose raised softirq as I mentioned in the previous letter. problem would indeed occur. We recognized the problem and are currently discussing solutions. One Solution would be to have one Soft-irq thread for each irq. You can expect a fix for this soon. If mach supports it Isn't it the current implementation? In DDE, each driver has one irq and one softirq thread? and you are using a single core machine a suggest to raise the prio of the irq handler thread in the ddekit as workaround. In my current porting, I already raise the priority of the irq handler, but it doesn't solve the problem.
I am thinking of using a lock to simulate cli/sti. When local_irq_enable() or local_irq_save() is called, we hold the lock; when local_irq_restore() or local_irq_disable(), we release the lock. We can even provide nested locking support if local_irq_enable() or local_irq_save() is allowed to be called multiple times in the same thread. Best regards, Zheng Da
On 10-2-27 上午9:39, Da Zheng wrote:
problem would indeed occur. We recognized the problem and are currently discussing solutions. One Solution would be to have one Soft-irq thread for each irq. You can expect a fix for this soon. If mach supports it Isn't it the current implementation? In DDE, each driver has one irq and one softirq thread? and you are using a single core machine a suggest to raise the prio of the irq handler thread in the ddekit as workaround. In my current porting, I already raise the priority of the irq handler, but it doesn't solve the problem. I forgot to mention I run DDE on a one-processor virtual machine.
Zheng Da
Hi, On Sat, Feb 27, 2010 at 09:39:19AM +0800, Da Zheng wrote:
On 10-2-27 上午9:10, Dirk Vogt wrote:
Hej, You're right. There is indeed a problem, which we haven't stumbled over yet. That is, because the irq handler runs at at a higher priority in our ddekit implementation. However on a multiprocessor machine this I don't understand. How can using a higher priority in the irq handler solve/avoid the problem? unless you mean running the softirq thread in a higher priority. Otherwise, the softirq thread can still be preempted and lose raised softirq as I mentioned in the previous letter.
You're right, priorities would not help on a multi-processor platform. But these scenarios are currently not supported anyway, so let's stay with one CPU. In this case DDE Linux works like the good old BSD4.4, in which the kernel execution contexts were assigned to priority levels. The interrupts on the higher levels and the user contexts on the lower ones. This scheme guarantees that interrupt contexts always finish their execution before other lower-priority contexts are executed. Linux adopted this in a way. Anyway, any approach that relies on priorities regarding DDE (or paravirtualization at large) on a microkernel caused problems in the past. Therefore, we discuss better options.
problem would indeed occur. We recognized the problem and are currently discussing solutions. One Solution would be to have one Soft-irq thread for each irq. You can expect a fix for this soon. If mach supports it Isn't it the current implementation? In DDE, each driver has one irq and one softirq thread?
That's not completely correct as DDE Linux does not prevent to run various drivers linked together in one protection domain. This may end up with multiple interrupt or softirq threads.
and you are using a single core machine a suggest to raise the prio of the irq handler thread in the ddekit as workaround. In my current porting, I already raise the priority of the irq handler, but it doesn't solve the problem.
I am thinking of using a lock to simulate cli/sti. When local_irq_enable() or local_irq_save() is called, we hold the lock; when local_irq_restore() or local_irq_disable(), we release the lock. We can even provide nested locking support if local_irq_enable() or local_irq_save() is allowed to be called multiple times in the same thread.
I think this addresses our issue in a way: Prevent handling of the just registered softirq until the interrupt handler returns. Afterwards, the registered softirq can be handled from the standard softirq thread as long as the queue manipulation is atomic. The issue is now near the top of our agenda for DDE. Greets -- Christian Helmuth Genode Labs http://www.genode-labs.com/ · http://genode.org/ Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Hi, On 10-2-28 下午8:57, Christian Helmuth wrote:
You're right. There is indeed a problem, which we haven't stumbled over yet. That is, because the irq handler runs at at a higher priority in our ddekit implementation. However on a multiprocessor machine this I don't understand. How can using a higher priority in the irq handler solve/avoid the problem? unless you mean running the softirq thread in a higher priority. Otherwise, the softirq thread can still be preempted and lose raised softirq as I mentioned in the previous letter.
You're right, priorities would not help on a multi-processor platform. But these scenarios are currently not supported anyway, so let's stay with one CPU. In this case DDE Linux works like the good old BSD4.4, in which the kernel execution contexts were assigned to priority levels. The interrupts on the higher levels and the user contexts on the lower ones. This scheme guarantees that interrupt contexts always finish their execution before other lower-priority contexts are executed. Linux adopted this in a way. I don't think it can work on one CPU either. The problem is caused when the softirq thread is preempted by the interrupt thread. When you set the interrupt thread a higher priority, it's more likely the softirq thread gets preempted. softirq queue manipulation must be atomic.
I am thinking of using a lock to simulate cli/sti. When local_irq_enable() or local_irq_save() is called, we hold the lock; when local_irq_restore() or local_irq_disable(), we release the lock. We can even provide nested locking support if local_irq_enable() or local_irq_save() is allowed to be called multiple times in the same thread.
I think this addresses our issue in a way: Prevent handling of the just registered softirq until the interrupt handler returns. What if the softirq thread is running when an interrupt comes? Since we cannot disable the hard irq, it is very likely to happen. Afterwards, the registered softirq can be handled from the standard softirq thread as long as the queue manipulation is atomic. The issue is now near the top of our agenda for DDE. If you only consider about handling the softirq queue, I think it's easy, simply using a lock to protect it explicitly, but it cannot solve the problem completely. There should be many race conditions in other places. I implement raw_local_irq_disable() and raw_local_irq_enable() with a lock, but there is still a serious problem. Because of spin_lock_irqsave, there can be dead locks. One example is in pcnet32_watchdog() and pcnet32_interrupt(). So I think we can just let spin_lock_irqsave() become spin_lock(). After all, the interrupt handling doesn't run in the real interrupt context any more, so I think there is no reason to do spin_lock_irqsave() or spin_lock_irq().
Best regards, Zheng Da
Hello, On Mon, Mar 01, 2010 at 10:45:44AM +0800, Da Zheng wrote:
I don't think it can work on one CPU either. The problem is caused when the softirq thread is preempted by the interrupt thread. When you set the interrupt thread a higher priority, it's more likely the softirq thread gets preempted. softirq queue manipulation must be atomic.
Right, that's what I wrote.
I think this addresses our issue in a way: Prevent handling of the just registered softirq until the interrupt handler returns. What if the softirq thread is running when an interrupt comes? Since we cannot disable the hard irq, it is very likely to happen. Afterwards, the registered softirq can be handled from the standard softirq thread as long as the queue manipulation is atomic. The issue is now near the top of our agenda for DDE. If you only consider about handling the softirq queue, I think it's easy, simply using a lock to protect it explicitly, but it cannot solve the problem completely. There should be many race conditions in other places. I implement raw_local_irq_disable() and raw_local_irq_enable() with a lock, but there is still a serious problem. Because of spin_lock_irqsave, there can be dead locks. One example is in pcnet32_watchdog() and pcnet32_interrupt(). So I think we can just let spin_lock_irqsave() become spin_lock(). After all, the interrupt handling doesn't run in the real interrupt context any more, so I think there is no reason to do spin_lock_irqsave() or spin_lock_irq().
Sounds like my idea behind the current scheme. Thanks for your input on this. Regards -- Christian Helmuth Genode Labs http://www.genode-labs.com/ · http://genode.org/ Genode Labs GmbH · Amtsgericht Dresden · HRB 28424 · Sitz Dresden Geschäftsführer: Dr.-Ing. Norman Feske, Christian Helmuth
Hi, On 10-3-1 下午5:58, Christian Helmuth wrote:
If you only consider about handling the softirq queue, I think it's easy, simply using a lock to protect it explicitly, but it cannot solve the problem completely. There should be many race conditions in other places. I implement raw_local_irq_disable() and raw_local_irq_enable() with a lock, but there is still a serious problem. Because of spin_lock_irqsave, there can be dead locks. One example is in pcnet32_watchdog() and pcnet32_interrupt(). So I think we can just let spin_lock_irqsave() become spin_lock(). After all, the interrupt handling doesn't run in the real interrupt context any more, so I think there is no reason to do spin_lock_irqsave() or spin_lock_irq().
Sounds like my idea behind the current scheme. Thanks for your input on this. I have one implementation and I sent it to the mailing list. Could you please check if it works? If possible, try it on your DDE. It works for my pcnet32 driver.
Best regards, Zheng Da
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I am thinking of using a lock to simulate cli/sti. When local_irq_enable() or local_irq_save() is called, we hold the lock; when local_irq_restore() or local_irq_disable(), we release the lock. We can even provide nested locking support if local_irq_enable() or local_irq_save() is allowed to be called multiple times in the same thread.
A lock does not work - been there, done that. The problem is that you can call local_irq_disable() several times and then only call local_irq_enable() only once to reset everything, so the semantics are different from recursive locks. L4Linux uses a threaded lock implementation for that reason, so we'll discuss reusing this idea. Bjoern -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEUEARECAAYFAkuLefsACgkQP5ijxgQLUNlp4ACYuM+qru4pw128Pg/2JE3KQX5Q uwCgh7RbluzXo4LWs4/QkQOHowlUuDw= =eJ/l -----END PGP SIGNATURE-----
Hi, On 10-3-1 下午4:25, Björn Döbel wrote:
I am thinking of using a lock to simulate cli/sti. When local_irq_enable() or local_irq_save() is called, we hold the lock; when local_irq_restore() or local_irq_disable(), we release the lock. We can even provide nested locking support if local_irq_enable() or local_irq_save() is allowed to be called multiple times in the same thread.
A lock does not work - been there, done that. The problem is that you can call local_irq_disable() several times and then only call local_irq_enable() only once to reset everything, so the semantics are different from recursive locks. I guess you have used locks, but somehow later removed the code, though I didn't know the reason.
Can't we simulate the same semantics with a lock? If the same thread already holds the lock, local_irq_disable() does nothing. My implementation for now works pretty well, at least for the pcnet32 driver. Maybe I miss something important. Here is my code: int raw_irqs_disabled_flags(unsigned long flags) { return ((int)flags > 0); } /* If it does lock operation successfully, return > 0. Otherwise, 0. */ static int nested_lock(ddekit_lock_t lock) { int do_lock = 0; if (ddekit_lock_try_lock(&lock)) { /* if we cannot lock */ /* check who hold the lock. */ if (_ddekit_lock_owner(&lock) != (int) ddekit_thread_myself()) { /* Someone else holds the lock, * or by the time I try to lock again, * the person has release the lock. */ ddekit_lock_lock(&lock); do_lock = 1; } /* If I hold the lock myself, I don't need to worry * the lock will be released somewhere before I do it. */ } else do_lock = 2; return do_lock; } /* Store the current flags state. * * This is done by returning the current refcnt. * * XXX: Up to now, flags was always 0 at this point and * I assume that this is always the case. Prove? */ unsigned long __raw_local_save_flags(void) { unsigned long flags; int do_lock = 0; if (cli_lock == NULL) ddekit_lock_init_unlocked(&cli_lock); /* It's important to do lock here. * Otherwise, a thread might not get correct flags. */ do_lock = nested_lock(cli_lock); flags = (unsigned long)atomic_read(&_refcnt); if (do_lock) ddekit_lock_unlock(&cli_lock); return flags; } /* Restore IRQ state. */ void raw_local_irq_restore(unsigned long flags) { Assert(cli_lock != NULL); atomic_set(&_refcnt, flags); if (flags == 0) ddekit_lock_unlock(&cli_lock); } /* Disable IRQs by grabbing the IRQ lock. */ void raw_local_irq_disable(void) { struct ddekit_thread *helder; int is_print = 0; if (cli_lock == NULL) ddekit_lock_init_unlocked(&cli_lock); nested_lock(cli_lock); atomic_inc(&_refcnt); } /* Unlock the IRQ lock until refcnt is 0. */ void raw_local_irq_enable(void) { Assert(cli_lock != NULL); atomic_set(&_refcnt, 0); ddekit_lock_unlock(&cli_lock); }
L4Linux uses a threaded lock implementation for that reason, so we'll discuss reusing this idea.
What is a threaded lock? Best regards, Zheng Da
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi,
Can't we simulate the same semantics with a lock? If the same thread already holds the lock, local_irq_disable() does nothing. My implementation for now works pretty well, at least for the pcnet32 driver. Maybe I miss something important. Here is my code: int raw_irqs_disabled_flags(unsigned long flags) { return ((int)flags > 0); }
/* If it does lock operation successfully, return > 0. Otherwise, 0. */ static int nested_lock(ddekit_lock_t lock) { int do_lock = 0;
if (ddekit_lock_try_lock(&lock)) { /* if we cannot lock */ /* check who hold the lock. */ if (_ddekit_lock_owner(&lock) != (int) ddekit_thread_myself()) { /* Someone else holds the lock, * or by the time I try to lock again, * the person has release the lock. */ ddekit_lock_lock(&lock); do_lock = 1; } /* If I hold the lock myself, I don't need to worry * the lock will be released somewhere before I do it. */ } else do_lock = 2;
return do_lock; }
/* Store the current flags state. * * This is done by returning the current refcnt. * * XXX: Up to now, flags was always 0 at this point and * I assume that this is always the case. Prove? */ unsigned long __raw_local_save_flags(void) { unsigned long flags; int do_lock = 0;
if (cli_lock == NULL) ddekit_lock_init_unlocked(&cli_lock); /* It's important to do lock here. * Otherwise, a thread might not get correct flags. */ do_lock = nested_lock(cli_lock); flags = (unsigned long)atomic_read(&_refcnt); if (do_lock) ddekit_lock_unlock(&cli_lock); return flags; }
/* Restore IRQ state. */ void raw_local_irq_restore(unsigned long flags) { Assert(cli_lock != NULL); atomic_set(&_refcnt, flags); if (flags == 0) ddekit_lock_unlock(&cli_lock); }
/* Disable IRQs by grabbing the IRQ lock. */ void raw_local_irq_disable(void) { struct ddekit_thread *helder; int is_print = 0;
if (cli_lock == NULL) ddekit_lock_init_unlocked(&cli_lock);
nested_lock(cli_lock); atomic_inc(&_refcnt); }
/* Unlock the IRQ lock until refcnt is 0. */ void raw_local_irq_enable(void) { Assert(cli_lock != NULL); atomic_set(&_refcnt, 0); ddekit_lock_unlock(&cli_lock); }
Compiles and runs for me right now. However, this solution does a global IRQ disable by using a global CLI/STI lock and therefore does not adhere to the DDE semantics of every thread running on its own virtual CPU. So, this may not be the final solution we'll end up with, but if it works for you, feel free to go ahead with it.
L4Linux uses a threaded lock implementation for that reason, so we'll discuss reusing this idea. What is a threaded lock?
It's a lock implementation that uses a dedicated thread to serialize lock accesses and implements waiting on a lock by blocking in an IPC to this thread. Bjoern -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuL5G4ACgkQP5ijxgQLUNkrvACdHNZA0ecT1Xsl0/IUvO30eTEt dbMAn0PMcLb1Xu5EjzYeKVsstUo+tiuA =ONEv -----END PGP SIGNATURE-----
Hi, On 10-3-1 下午11:59, Björn Döbel wrote:
Compiles and runs for me right now. However, this solution does a global IRQ disable by using a global CLI/STI lock and therefore does not adhere to the DDE semantics of every thread running on its own virtual CPU. So, this may not be the final solution we'll end up with, but if it works for you, feel free to go ahead with it. I don't know why DDE needs this kind of semantics. Does the implementation of DDE reply on it? or how does DDE reply on it?
L4Linux uses a threaded lock implementation for that reason, so we'll discuss reusing this idea. What is a threaded lock?
It's a lock implementation that uses a dedicated thread to serialize lock accesses and implements waiting on a lock by blocking in an IPC to this thread. Isn't it also like a global lock?
Best regards, Zheng Da
participants (3)
-
Björn Döbel -
Christian Helmuth -
Da Zheng