Hello,
I have been working through reimplementing the mechanisms involved with
launching programs in L4Re, managing to initiate programs apparently
successfully under QEMU emulating amd64 and x86, but I have been less
successful getting them to work under QEMU emulating a MIPS system (MIPS
Malta).
My arrangement of threads within tasks should be fairly similar to what L4Re
normally employs to host and run programs, and I wrote it up here:
"Gradual Explorations of Filesystems, Paging and L4Re"
https://blogs.fsfe.org/pboddie/?p=2540
The pictures at the end summarise what has been done. I have a region mapper
running in a thread in the same task as the program to be launched. The region
mapper handles page faults and interacts with dataspaces on the program's
behalf. Faults caused by the region mapper are handled by the initiating task.
What I seem to have avoided previously but now experience is that after the
region mapper has been started, it is able to service page faults, but after a
certain point in the execution of the program of interest, the region mapper
itself experiences an exception in the __pthread_cleanup_push_defer function
within uClib's pthreads implementation, specifically at this line:
buffer->__canceltype = THREAD_GETMEM(self, p_canceltype);
Digging into this, I found that self is apparently zero. Since self is
obtained from a call to thread_self which in turn obtains the user[0] member
of the thread control register structure (providing the thread descriptor
address), I can only conclude that either this member has not been initialised
or that it has been overwritten by the other thread as it started up.
Investigating uClibc startup, I notice that pthread initialisation does indeed
involve a traversal of UTCB areas, zeroing user[0] in the supposedly free UTCB
blocks. And with the region mapper having failed, the main program will also
fail fairly quickly, with an exception typically occurring in the
__pthread_initialize_minimal function, coincidentally just after this zeroing
has occurred, just before a call to __libc_setup_tls.
I have been looking for hints as to how this work is done in L4Re normally.
Given that the region mapper and the program being run are normal programs
that are executed via their conventional entry points, this resulting in the
initialisation of the C library and threading for each payload, it seems that
some measures must be in place to prevent this initialisation from causing
conflicts in the UTCB area.
For instance, I see that in the file...
pkg/l4re-core/l4re_kernel/server/src/main.cc
... the __libc_alloc_initial_tls function is redefined to presumably override
the weakly referenced version in uClibc, but if this were invoked it would
probably cause uClibc to fail during initialisation, at least unless the zero
page is going to be used (but it would need allocation somewhere else), so
maybe it doesn't override anything after all. In any case, this function would
only change some aspects of how __libc_setup_tls would work, so maybe it is a
distraction.
I did wonder if there was something specific about MIPS that would cause this
failure, investigating the user-local register (ULR) used to hold UTCB-related
information, but getting the UTCB address via the ULR would appear to work:
otherwise, the region mapper would probably not start up. I also wondered
about program initialisation issues, since I did experience problems on actual
MIPS hardware related to various registers not being set up correctly, but the
generated code seems fairly comprehensive in this regard these days (and maybe
it was all along but I encountered a confounding phenomenon).
Does anyone have any familiarity with how this actually works and might be
able to indicate what I am doing wrong?
As always, thanks for any help that can be offered!
Paul
Hello,
I finally got round to experimenting with L4Re again, but in attempting to
investigate task creation, I seem to have some difficulties understanding the
mechanism by which tasks are typically created and how the l4_task_map
function might be used in the process.
After looking at lots of different files in the L4Re distribution, my
understanding of the basic mechanism is as follows:
1. Some memory is reserved for the UTCB of a new task, perhaps using the
l4re_ma_alloc_align function (or equivalent) to obtain a dataspace.
2. A task is created using l4_factory_create_task, indicating the UTCB
flexpage, with this being defined as...
l4_factory_create_task(l4re_env()->factory, new_task,
l4_fpage(utcb_start, utcb_log2size, L4_FPAGE_RW))
3. A thread is created using l4_factory_create_thread.
l4_factory_create_thread(l4re_env()->factory, new_thread)
4. The thread attributes are set using the l4_thread_control API.
5. The l4_thread_ex_regs function is used to set the instruction pointer
(program counter) and stack pointer of the thread.
6. The l4_scheduler_run_thread function is used to initiate the thread.
The expectation is that the thread will immediately fault because there is no
memory mapped at the instruction pointer location. However, it seems to me
that it should be possible to use l4_task_map to make a memory region
available within the task's address space, although I don't ever see this
function used in L4Re for anything.
(The C++ API makes it difficult to perform ad-hoc searches for such low-level
primitives, in my view, so perhaps I am missing use of the equivalent
methods.)
Tentatively, I would imagine that something like this might work:
l4_task_map(new_task, L4RE_THIS_TASK_CAP,
l4_fpage(program_start, program_log2size, L4_FPAGE_RX),
task_program_start)
Here, the program payload would be loaded into the creating task at
program_start, but the new task would be receiving the payload at
task_program_start, with the configured instruction pointer location occurring
within the receive window (after task_program_start, in other words).
There are, of course, many other considerations around creating tasks, which I
have noted from looking at the different packages (libloader, l4re_kernel,
moe, ned), and I am aware that a few other things need to be done to start a
task such as...
* Defining capability selectors and mapping appropriate capabilities to the
new task.
* Creating a stack for the task and populating it with arguments and
environment information.
* Defining a suitable pager and exception handler, with this usually being
provided by the l4re binary, as I understand it.
Also, when actually dealing with program loading generally, I realise that the
ELF binary needs to be interpreted and the appropriate regions associated with
different parts of memory, this typically being handled by the region mapper/
manager in L4Re. And there is also the matter of dynamic library loading.
But here, I am just attempting to establish the basic mechanism when a task
starts up. Unfortunately, the only discussion I found was this (after some
initial discussion about a related topic):
http://os.inf.tu-dresden.de/pipermail/l4-hackers/2014/015366.html
There are various examples in Subversion (maybe somewhere in the Git
repositories, too) that create tasks or threads, but I don't find them
particularly helpful, apparently being oriented towards very specific
applications. A previous example was referenced in the above thread for the
older L4Env system (or maybe an even earlier system):
http://os.inf.tu-dresden.de/pipermail/l4-hackers/2000/000384.html
As for why I would be wondering about such things - a question inevitably
asked in the first thread referenced above - I firstly want to be able to
understand the mechanism involved, but I also want to be able to integrate
work I have been doing on file paging into task creation.
Although I can probably do this by customising the "app model" normally used
by the different loaders, it seems that I would need to construct an
alternative l4re binary, which is rather cumbersome and perhaps a weakness of
the abstractions that are provided, these being rather oriented towards
obtaining dataspaces via the namespace API which I don't want to have to
support in my filesystem.
In any case, I wonder if there are any resources that describe the use of
l4_task_map and the details of the program environment within tasks.
Paul
