On Mon Dec 12, 2005 at 15:50:17 -0500, Julian Grizzard wrote:
to track the program counter and registers of all threads in the L4Linux task. Here's exactly what I would like to be able to do from a monitoring L4 task:
I'll throw in some comments.
-Obtain the PC/regs for the first thread (thread 0) for L4Linux just before the thread executes it's first instruction. It would be nice to make the thread wait for a signal before beginning execution.
The regs are useless is this case as they haven't been filled in with anything. Only ip and sp are set. Making the thread wait is possible by changing the startup method. The monitor must be the exception handler then.
-Obtain the PC/regs for any additional thread created or any thread moved for L4Linux (i.e. from a l4_thread_ex_regs call) just before that thread executes. Again, it would be nice to make that thread wait for a signal before beginning its execution.
If the threads are started the same way as above this should be possible.
-Randomly preemptively obtain the PC/regs for all threads of L4Linux.
Getting the ip is possible, getting all the regs is more complicated due to our multiple extry paths into the kernel and thus different stack frames.
-Obtain the PC/regs for all threads of L4Linux on demand.
What's the difference from above? Works the same way, I'd guess.
-Receive notification of thread exit for all threads of L4Linux.
Threads do not exit, they just go to sleep (forever). So a thread exit can only be a user level thing.
There are a few items that I do not yet fully understand that *may* make matters more difficult:
-The setup code executing in libloader.s.so.
-Trampoline code
There are a few assumptions I am willing to have for initial testing to make the development easier:
-Assume Task number of L4Linux is known by the microkernel (i.e. hard code it in).
I'd say that's a really minor issue you have there ;)
-Assume Task number of monitoring thread is known by the microkernel.
As a first crack, I have added some code to the microkernel to print out information on the L4Linux task, to see if that information would be good to transfer to the monitoring task. Below is a summary of things I have tried and results seen:
MODIFICATION #1 l4/kernel/fiasco/src/kern/context.cpp Context::schedule() Goal: Track current PC of L4Linux threads ======================= ... /* in the for loop */ Sys_task_new_frame *regs = sys_frame_cast<Sys_task_new_frame>(next_to_run->regs()); L4_uid id = regs->dst(); if (id.task() == 0xC) // hard coded task of L4Linux { unsigned int ip = next_to_run->regs()->ip(); unsigned int sp = next_to_run->regs()->sp(); printf("task: %d.%d @ ip: %08x sp: %08x\n", id.task(), id.lthread(), ip, sp); } ... ======================= Results: task: 12.3 @ ip: 004ccb0b sp: b00fff80 task: 12.3 @ ip: 004cd303 sp: b01fff7c task: 12.4 @ ip: 004cca9d sp: b00fff80 task: 12.5 @ ip: 00402ae6 sp: 00657f50 ======================= Notes: 004ccb0b - right after an 'int 0x30' in timer_irq_thread 004cd303 - right after an 'int 0x30' in irq_thread_hw 004cca9d - right after an 'int 0x30' in timer_irq_thread 00402ae6 - right after an 'int 0x30' in l4x_idle
This makes perfectly sense.
MODIFICATION #2 l4/kernel/fiasco/src/kern/thread-syscall.cpp Thread::sys_task_new() Goal: Track starting PC of main L4Linux thread (i.e. thread 0) ======================= ... /* near tail of for loop, just before return */ if (taskno == 0xC) // hard coded L4Linux task ID { printf("new task: %d @ ip: %08x sp: %08x\n", taskno, regs->ip(), regs->sp()); } ======================= Results: new task: 12 @ ip: 00015ea0 sp: 00008f6c ======================= Notes: 00015ea0 - part of the libloader.s.so code
This can be expected.
MODIFICATION #3 l4/kernel/fiasco/src/kern/thread-syscall.cpp Thread::sys_thread_ex_regs() Goal: Track PC entry of new L4Linux threads ======================= ... if (!dst->exists()) { unsigned int task_num = dst_id.task(); unsigned int lthread_num = regs->thread(); new_thread = true; check (new (dst_id) Thread (dst_task, dst_id, sched()->prio(),
mcp()) == dst); if (taskno == 0xC) { printf("new thread: %d.%d @ ip: %08x sp: %08x\n", task_num, lthread_num, regs->ip(), regs->sp()); }
} else { unsigned int task_num = dst_id.task(); unsigned int lthread_num = regs->thread(); if (taskno == 0xC) { printf("modified thread: %d.%d @ ip: %08x sp: %08x\n", task_num, lthread_num, regs->ip(), regs->sp()); }
} ... ======================= Results: modified thread: 12.0 @ ip: ffffffff sp: ffffffff modified thread: 12.0 @ ip: ffffffff sp: ffffffff new thread: 12.1 @ ip: 0001ac10 sp: afe00000 new thread: 12.2 @ ip: 0001ac10 sp: aff00000 modified thread: 12.2 @ ip: ffffffff sp: ffffffff new thread: 12.3 @ ip: 0001ac10 sp: 00657ff8 modified thread: 12.3 @ ip: ffffffff sp: ffffffff new thread: 12.4 @ ip: 0001ac10 sp: b0100000 modified thread: 12.3 @ ip: ffffffff sp: ffffffff new thread: 12.5 @ ip: 0001ac10 sp: b0200000 modified thread: 12.3 @ ip: ffffffff sp: ffffffff new thread: 12.6 @ ip: 0001ac10 sp: b0300000 new thread: 12.7 @ ip: 0001ac10 sp: b0400000 ======================= Notes: 0001ac10 - part of the libloader.s.so code
Yup.
ffffffff - not really useful. wrong data perhaps.
No, see the spec, when ip or sp are -1 then they're not changed, i.e. ex_regs is giving back values only.
QUESTIONS:
GENERAL QUESTIONS -Is there any easy to delay execution of newly created threads, both new task thread 0 and additional threads?
What one could do is put an exception handler task between the loader and l4linux and intercept interesting syscalls. This way you also get notifications for new threads. There are probably things which need to be enhanced/fixed/whatever in the kernel but theoretically this could work this way.
-Is there an easy way to randomly interrupt L4Linux threads to check their PC value?
If you monitor task runs L4Linux is not running (UP system) so you can just exregs them to query the ip.
-Do any L4Linux threads exit?
In the context of L4Linux this is possible, in the context of L4 it's not (see above).
MODIFICATION #1 -Seems like threads resume execution right after an interrupt.
int 0x30 is the ipc system call.
I don't see threads getting preempted in the middle of servicing a system call, for example. Any ideas why not?
Bad luck. I guess you'll need some threads with a higher prio doing some work to actually see this.
Further, is there one thread dedicated to handling system calls?
Yes, only one thread is handling Linux system calls.
-Is there any better way to track where all the L4Linux threads are? Maybe somewhere else besides the scheduler? Are any other classes/functions helpful to get this information?
What you are doing is reading the ip and sp from the return frame, you can basically do this from everywhere when you run on the behalf of some user thread. It's not limited to the schedule function.
MODIFICATION #2 -I believe this is working correctly. See anything wrong here?
Possible.
MODIFICATION #3 -I would like to catch all instances of creating new threads for L4Linux. Will this current approach work, or is it not correct?
Maybe you want to better hook into Thread::initialize so you get task_new and threads in one go.
If my implementation is correct: -Why are L4Linux threads getting modified? -Any idea why my ip and sp values are 0xffffffff for modified threads?
Again, see the spec. Querying pager/preempter is done with -1 in ip and sp, so this can be one cause of the things you're seeing.
-Looks like all new threads start at the same place. How would they distinguish from one another in terms of functionality? Is that encoded in their stack (e.g. when they issue a 'ret', they'll return to different locations or maybe arguments to their function are on the stack?)
The thread lib is doing this for us. All threads are executing some setup code before they're branched to their actual function (see l4/pkg/thread/lib/src/create.c).
Adam