Adam,
Thanks for the reply!
On Monday, 11 April 2022 01:02:37 CEST Adam Lackorzynski wrote:
Hi Paul,
On Sun Apr 10, 2022 at 18:58:10 +0200, Paul Boddie wrote:
I finally got round to experimenting with L4Re again, but in attempting to investigate task creation, I seem to have some difficulties understanding the mechanism by which tasks are typically created and how the l4_task_map function might be used in the process.
After looking at lots of different files in the L4Re distribution, my understanding of the basic mechanism is as follows:
- Some memory is reserved for the UTCB of a new task, perhaps using the
l4re_ma_alloc_align function (or equivalent) to obtain a dataspace.
No, for UTCBs there's a dedicated call l4_task_add_ku_mem in case one needs more UTCB memory than has been initially created with l4_factory_create_task().
OK, I did see that function being used, too, but I also found plenty of other things in my perusal of the different files. Obviously, being able to extend the UTCB memory is an important consideration.
- A task is created using l4_factory_create_task, indicating the UTCB
flexpage, with this being defined as...
l4_factory_create_task(l4re_env()->factory, new_task,
l4_fpage(utcb_start, utcb_log2size, L4_FPAGE_RW))
Yes. Here the flexpage defines where memory usable for UTCBs shall be created.
Right. I see that the factory function actually sends the flexpage in the IPC call (using l4_factory_create_add_fpage_u), thus mapping it in the task. I find it hard to follow where this message is actually handled (I presume that Moe acts as the factory) or what the factory actually does with the flexpage, but I presume that it ultimately causes it to be mapped in the new task.
[Thread creation and initiation]
The expectation is that the thread will immediately fault because there is no memory mapped at the instruction pointer location. However, it seems to me that it should be possible to use l4_task_map to make a memory region available within the task's address space, although I don't ever see this function used in L4Re for anything.
(The C++ API makes it difficult to perform ad-hoc searches for such low-level primitives, in my view, so perhaps I am missing use of the equivalent methods.)
Indeed. L4::Task::map is used, for example to map some initial capabilities, and typically not memory.
Yes, I see a lot of these map operations operating on capabilities. For example:
pkg/l4re-core/libloader/include/remote_app_model
However, I wonder about the "chicken and egg" situation in new tasks. It seems to me that the way things work is that a new task in L4Re is typically populated with the l4re binary containing the region mapper/manager (RM). This seems to be initiated here (in launch_loader):
pkg/l4re-core/ned/server/src/lua_exec.cc
This RM is then able to handle the page fault when an attempt is made to load and run a new program. But one cannot rely on the RM when it isn't already installed in a task, so there must be a way of mapping it into the new task so that it can be present. I assumed that using l4_task_map might be one way of doing so.
Otherwise, I thought that perhaps an existing task could provide a kind of RM to act as the new task's pager in the bootstrapping phase, so that page faults would be directed towards the existing task's RM and mappings established to get the new task's RM up and running. However, in that case, since the usual IPC traffic between RM and dataspaces does not involve sending flexpages to the new task (and thus implicitly mapping them in the task, as I understand it), it seems that the existing task's RM would also need to explicitly map flexpages in the new task, again using something like l4_task_map.
I think I understand the usual mechanism between a task's RM and dataspaces, at least enough to have implemented paging with dataspaces myself, but I don't follow what is actually being done here.
Tentatively, I would imagine that something like this might work: l4_task_map(new_task, L4RE_THIS_TASK_CAP,
l4_fpage(program_start, program_log2size, L4_FPAGE_RX), task_program_start)
Here, the program payload would be loaded into the creating task at program_start, but the new task would be receiving the payload at task_program_start, with the configured instruction pointer location occurring within the receive window (after task_program_start, in other words).
Yes, this would work.
As far as I have seen, the "send base" with l4_task_map is effectively defining the location where the flexpage is mapped, since the receive window is "the whole address space" of the destination task according to the documentation. Some testing with existing tasks appears to confirm this, but then I also seem to experience issues with memory coherency or something resembling it, meaning that I write to several pages, map the memory to another task, and yet the mapped region does not reflect the mapped memory (and it is indeed mapped, since I can neglect to map it and get a page fault).
I just spent some time mapping memory by sending flexpages from one task to another, defining the receive window using the recipient's buffer registers so that the mappings are established by the kernel (again, as I understand how this actually ends up working), and this does establish the mappings, but I still see a lack of coherency. Maybe I am just failing to call the appropriate functions to let the recipient see all of the changed memory contents, however.
[...]
In any case, I wonder if there are any resources that describe the use of l4_task_map and the details of the program environment within tasks.
l4_task_map() has documentation: https://l4re.org/doc/group__l4__task__api.html#ga8ed2ff7ba204de7c01311c2241 2a2063 and is a direct API to the kernel for mapping resources, defined by l4sys. At this level, there is not really a definition of how a program environment looks like. However, as Fiasco needs to supply its initial programs some capabilities, those are defined (https://l4re.org/doc/group__l4__cap__api.html#gaa7801b63edba351bad9ea802643 2b5c4). What moe and ned do, is similar, but not necessarily the same, as they provide a more powerful interface to this (https://l4re.org/doc/group__api__l4re__env.html) and also provide all the functionality normal programs enjoy, like argument lists, environment variables, etc.
I've been spending plenty of time looking at the documentation. However, I really feel that the fundamentals of the system are often not readily documented, at least in such reference documentation. So, I've also spent time looking at teaching materials related to L4Re and Fiasco, some of which are helpful, but they obviously do not go into much detail.
That leaves the code, which is not always very easy to follow. Part of the reason I just decided to implement my own IPC library and interface description language was that the IPC support in L4Re is incoherent, with different approaches used in different places, and sometimes convoluted, also not relating obviously to the low-level libraries, thus making it difficult to identify common areas of functionality.
This might just sound like me complaining, but I also have some concerns about being able to verify the behaviour of some of the code. For example, I recently found that my dataspace implementation was getting requests from a region mapper/manager with an opcode of 0x100000000, which doesn't make any sense to me at all, given that the dataspace interface code in L4Re implicitly defines opcodes that are all likely to be very small integers. At first I obviously blamed my own code, but then I found that in the IPC call implementation found here...
pk/l4re-core/l4sys/include/cxx/ipc_iface
...if I explicitly cleared the first message register before this statement...
int send_bytes = Args::template write_op<Do_in_data>(mrs->mr, 0, Mr_bytes, Opt::Opcode, a...);
...then the opcode was produced as expected again. I suppose what I am trying to communicate is that some of the organisation of the code is not conducive to inspection, nor does it readily reflect the mechanisms involved. Although the availability of software to do a task arguably diminishes the need to be familiar with what the software does, that arrangement only works if the software is usable, extensible and does what it is supposed to.
In case you might be wondering why I am doing any of this (as I sometimes do myself), I am attempting to integrate a filesystem into L4Re, but this also leads to the matter of running programs from the filesystem. And so, it becomes interesting to try and create tasks and populate them with those programs.
Paul