Track Program_Counter(PC)/regs of L4Linux Task

[Julian Grizzard]

All,

Okay, so let me back up a minute from trying to understand all of these
items at the same time.  What I would like to understand first is what
are the main threads for the L4Linux server.  Based on your feedback and
some further analysis, here is what I believe are the main threads:

Thread 0 - l4env initialization, then sleeps
Thread 1 - ??
Thread 2 - runs "main" to get the L4Linux server going, then sleeps
Thread 3 - runs "l4env_linux_startup" function, then sleeps
until told that L4Linux is exiting
Thread 4 - Timer interrupt thread, handles the periodic timer for linux
   the entry functions for thread 4 are:
   timer_irq_thread --- the main handler
   l4lx_irq_timer_startup --- gets timer_irq_thread started
   l4lx_irq_timer_shutdown
   l4lx_irq_timer_enable
   l4lx_irq_timer_disable
   l4lx_irq_timer_ack
   l4lx_irq_timer_end
Additional threads >= 5, one for each IRQ (virtual not implemented yet)
   the entry functions for these threads are
   irq_dev_thread --- the main handler
   l4lx_irq_dev_startup_hw  --- gets irq_dev_thread started
   l4lx_irq_dev_shutdown_hw
   l4lx_irq_dev_enable_hw
   l4lx_irq_dev_disable_hw
   l4lx_irq_dev_ack_hw
   l4lx_irq_dev_end_hw
   l4lx_irq_dev_startup_virt
   l4lx_irq_dev_shutdown_virt
   l4lx_irq_dev_enable_virt
   l4lx_irq_dev_disable_virt
   l4lx_irq_dev_ack_virt
   l4lx_irq_dev_end_virt

Anything wrong with that description?  What does thread 1 do?  I can't
seem to track it down.  Also, which thread is responsible for handling
system calls?  Is that what thread 1 does maybe?  Which L4Linux function
does the system call handler function start?

Thanks!

Julian


Adam Lackorzynski wrote:
> On Mon Dec 12, 2005 at 15:50:17 -0500, Julian Grizzard wrote:
> 
>>to track the program counter and registers of all threads in the L4Linux
>>task.  Here's exactly what I would like to be able to do from a
>>monitoring L4 task:
> 
> 
> I'll throw in some comments.
>  
> 
>>-Obtain the PC/regs for the first thread (thread 0) for L4Linux just
>>before the thread executes it's first instruction.  It would be nice to
>>make the thread wait for a signal before beginning execution.
> 
> 
> The regs are useless is this case as they haven't been filled in with
> anything. Only ip and sp are set. Making the thread wait is possible by
> changing the startup method. The monitor must be the exception handler
> then.
> 
> 
>>-Obtain the PC/regs for any additional thread created or any thread
>>moved for L4Linux (i.e. from a l4_thread_ex_regs call) just before that
>>thread executes.  Again, it would be nice to make that thread wait for a
>>signal before beginning its execution.
> 
> 
> If the threads are started the same way as above this should be
> possible.
> 
> 
>>-Randomly preemptively obtain the PC/regs for all threads of L4Linux.
> 
> 
> Getting the ip is possible, getting all the regs is more complicated due
> to our multiple extry paths into the kernel and thus different stack
> frames.
> 
> 
>>-Obtain the PC/regs for all threads of L4Linux on demand.
> 
> 
> What's the difference from above? Works the same way, I'd guess.
> 
> 
>>-Receive notification of thread exit for all threads of L4Linux.
> 
> 
> Threads do not exit, they just go to sleep (forever). So a thread exit
> can only be a user level thing.
> 
> 
>>There are a few items that I do not yet fully understand that *may* make
>>matters more difficult:
>>
>>-The setup code executing in libloader.s.so.
>>
>>-Trampoline code
>>
>>There are a few assumptions I am willing to have for initial testing to
>>make the development easier:
>>
>>-Assume Task number of L4Linux is known by the microkernel (i.e. hard
>>code it in).
> 
> 
> I'd say that's a really minor issue you have there ;)
> 
> 
>>-Assume Task number of monitoring thread is known by the microkernel.
>>
>>As a first crack, I have added some code to the microkernel to print out
>>information on the L4Linux task, to see if that information would be
>>good to transfer to the monitoring task.  Below is a summary of things I
>>have tried and results seen:
>>
>>MODIFICATION #1
>>l4/kernel/fiasco/src/kern/context.cpp
>>Context::schedule()
>>Goal: Track current PC of L4Linux threads
>>=======================
>>...
>>/* in the for loop */
>>Sys_task_new_frame *regs =
>>sys_frame_cast<Sys_task_new_frame>(next_to_run->regs());
>>L4_uid id = regs->dst();
>>if (id.task() == 0xC) // hard coded task of L4Linux
>>  {
>>    unsigned int ip = next_to_run->regs()->ip();
>>    unsigned int sp = next_to_run->regs()->sp();
>>    printf("task: %d.%d @ ip: %08x sp: %08x\n", id.task(), id.lthread(),
>>            ip, sp);
>>  }
>>...
>>=======================
>>Results:
>>task: 12.3 @ ip: 004ccb0b sp: b00fff80
>>task: 12.3 @ ip: 004cd303 sp: b01fff7c
>>task: 12.4 @ ip: 004cca9d sp: b00fff80
>>task: 12.5 @ ip: 00402ae6 sp: 00657f50
>>=======================
>>Notes:
>>004ccb0b - right after an 'int 0x30' in timer_irq_thread
>>004cd303 - right after an 'int 0x30' in irq_thread_hw
>>004cca9d - right after an 'int 0x30' in timer_irq_thread
>>00402ae6 - right after an 'int 0x30' in l4x_idle
> 
> 
> This makes perfectly sense.
>  
> 
>>MODIFICATION #2
>>l4/kernel/fiasco/src/kern/thread-syscall.cpp
>>Thread::sys_task_new()
>>Goal: Track starting PC of main L4Linux thread (i.e. thread 0)
>>=======================
>>...
>>/* near tail of for loop, just before return */
>>if (taskno == 0xC) // hard coded L4Linux task ID
>>  {
>>    printf("new task: %d @ ip: %08x sp: %08x\n",
>>            taskno, regs->ip(), regs->sp());
>>  }
>>=======================
>>Results:
>>new task: 12 @ ip: 00015ea0 sp: 00008f6c
>>=======================
>>Notes:
>>00015ea0 - part of the libloader.s.so code
> 
> 
> This can be expected.
> 
> 
>>MODIFICATION #3
>>l4/kernel/fiasco/src/kern/thread-syscall.cpp
>>Thread::sys_thread_ex_regs()
>>Goal: Track PC entry of new L4Linux threads
>>=======================
>>...
>>if (!dst->exists())
>>  {
>>    unsigned int task_num = dst_id.task();
>>    unsigned int lthread_num = regs->thread();
>>    new_thread = true;
>>    check (new (dst_id) Thread (dst_task, dst_id, sched()->prio(),
>>
>>           mcp()) == dst);
>>    if (taskno == 0xC)
>>      {
>>        printf("new thread: %d.%d @ ip: %08x sp: %08x\n",
>>               task_num, lthread_num, regs->ip(), regs->sp());
>>      }
>>  }
>>else
>>  {
>>    unsigned int task_num = dst_id.task();
>>    unsigned int lthread_num = regs->thread();
>>    if (taskno == 0xC)
>>      {
>>        printf("modified thread: %d.%d @ ip: %08x sp: %08x\n",
>>               task_num, lthread_num, regs->ip(), regs->sp());
>>      }
>>
>>  }
>>...
>>=======================
>>Results:
>>modified thread: 12.0 @ ip: ffffffff sp: ffffffff
>>modified thread: 12.0 @ ip: ffffffff sp: ffffffff
>>new thread: 12.1 @ ip: 0001ac10 sp: afe00000
>>new thread: 12.2 @ ip: 0001ac10 sp: aff00000
>>modified thread: 12.2 @ ip: ffffffff sp: ffffffff
>>new thread: 12.3 @ ip: 0001ac10 sp: 00657ff8
>>modified thread: 12.3 @ ip: ffffffff sp: ffffffff
>>new thread: 12.4 @ ip: 0001ac10 sp: b0100000
>>modified thread: 12.3 @ ip: ffffffff sp: ffffffff
>>new thread: 12.5 @ ip: 0001ac10 sp: b0200000
>>modified thread: 12.3 @ ip: ffffffff sp: ffffffff
>>new thread: 12.6 @ ip: 0001ac10 sp: b0300000
>>new thread: 12.7 @ ip: 0001ac10 sp: b0400000
>>=======================
>>Notes:
>>0001ac10 - part of the libloader.s.so code
> 
> 
> Yup.
> 
> 
>>ffffffff - not really useful.  wrong data perhaps.
> 
> 
> No, see the spec, when ip or sp are -1 then they're not changed, i.e.
> ex_regs is giving back values only.
> 
> 
>>QUESTIONS:
>>
>>GENERAL QUESTIONS
>>-Is there any easy to delay execution of newly created threads, both new
>>task thread 0 and additional threads?
> 
> 
> What one could do is put an exception handler task between the loader
> and l4linux and intercept interesting syscalls. This way you also get
> notifications for new threads. There are probably things which need to
> be enhanced/fixed/whatever in the kernel but theoretically this could
> work this way.
> 
> 
>>-Is there an easy way to randomly interrupt L4Linux threads to check
>>their PC value?
> 
> 
> If you monitor task runs L4Linux is not running (UP system) so you can
> just exregs them to query the ip.
> 
> 
>>-Do any L4Linux threads exit?
> 
> 
> In the context of L4Linux this is possible, in the context of L4 it's
> not (see above).
> 
> 
>>MODIFICATION #1
>>-Seems like threads resume execution right after an interrupt.
> 
> 
> int 0x30 is the ipc system call.
> 
> 
>>I don't
>>see threads getting preempted in the middle of servicing a system call,
>>for example.  Any ideas why not?
> 
> 
> Bad luck. I guess you'll need some threads with a higher prio doing
> some work to actually see this.
> 
> 
>>Further, is there one thread dedicated
>>to handling system calls?
> 
> 
> Yes, only one thread is handling Linux system calls.
> 
> 
>>-Is there any better way to track where all the L4Linux threads are?
>>Maybe somewhere else besides the scheduler?  Are any other
>>classes/functions helpful to get this information?
> 
> 
> What you are doing is reading the ip and sp from the return frame, you
> can basically do this from everywhere when you run on the behalf of some
> user thread. It's not limited to the schedule function.
> 
> 
>>MODIFICATION #2
>>-I believe this is working correctly.  See anything wrong here?
> 
> 
> Possible.
> 
> 
>>MODIFICATION #3
>>-I would like to catch all instances of creating new threads for
>>L4Linux.  Will this current approach work, or is it not correct?
> 
> 
> Maybe you want to better hook into Thread::initialize so you get
> task_new and threads in one go.
> 
> 
>>If my implementation is correct:
>>-Why are L4Linux threads getting modified?
>>-Any idea why my ip and sp values are 0xffffffff for modified threads?
> 
> 
> Again, see the spec. Querying pager/preempter is done with -1 in ip and
> sp, so this can be one cause of the things you're seeing.
> 
> 
>>-Looks like all new threads start at the same place.  How would they
>>distinguish from one another in terms of functionality?  Is that encoded
>>in their stack (e.g. when they issue a 'ret', they'll return to
>>different locations or maybe arguments to their function are on the stack?)
> 
> 
> The thread lib is doing this for us. All threads are executing some
> setup code before they're branched to their actual function (see
> l4/pkg/thread/lib/src/create.c).
> 
> 
> 
> Adam