Hi Paul,

On 5/23/23 00:51, Paul Boddie wrote:

On Sunday, 23 April 2023 10:09:52 CEST Jakub Jermář wrote:

...

The idea is that you have a dataspace and a DMA space. In order to obtain a DMA address that you can then use to program your I/O device, you need to map a portion of the dataspace into the DMA space. After use you unmap it from the DMA space.

See for example the implementation of the Inout buffer in our nvme-driver:

https://github.com/kernkonzept/nvme-driver/blob/master/server/src/

inout_buffer.h#L46

I also looked at the main.cc file to see how I might obtain a DMA space and assign it to a DMA domain. Obtaining a DMA space appears to involve invoking a factory create operation using the L4RE_PROTO_DMA_SPACE protocol.

Here, I found a difference between the L4Re C and C++ libraries, with the latter referencing a "user_factory" but defining it as the memory allocator, so I just used the mem_alloc member of the environment structure in C for the factory.

In C++ mem_alloc and user_factory are nearly the same. Mem_alloc extends the factory protocol with allocation for dataspaces. At the moment, this is effectively a convenience wrapper, as both are using the same capability index and thus talk to the same server. The reason for the user_factory name is to be able to write less confusing code, e.g. where one would use mem_alloc to create some non-memory objects.

With a DMA space, it should supposedly be possible to specify ~0U (or -1, I guess) as the domain identifier when assigning the space to a domain, but this does not work and yields the following warning:

IO | vbus vbus does not support a global DMA domain

This along with an error indicating "No such object found".

The main.cc file in the nvme-driver performs a search for a DMA domain resource, but replicating this myself produces a few issues. First of all, a search from the root device doesn't manage to find a resource offering a DMA domain if I traverse devices and their resources.

However, I see that the hw_root_bus.cc file in pkg/io/io/server/src adds root resources for a DMA domain, amongst other things, and if I also investigate the resources of the root device itself, then I do find such resources. Strangely, these resources appear to have an identifier of zero after they have been added to the bus, even though they get set up with expected identifiers such as "DMAD".

Meanwhile, in the main.cc file in pkg/io/io/server/src, a DMA domain resource is explicitly created when no iommu capability is provided to the server. This actually appears in the dump of the devices and resources at the system bus level, as produced by Io, but not within the vbus.

So, I think I need some suggestions as to how I can actually obtain a domain identifier and maybe how a domain should be made available in the .io file. Sorry if I failed to understand what to do, but none of this is particularly obvious to me.

The path you are on sounds correct. A reason for a missing DMA domain resource on the vbus is (usually) the lack of a DMA capable device on said vbus.

https://github.com/kernkonzept/manifest/wiki/NVMeWithLinux#vbus-configuratio...

shows a vbus configuration file that places all PCI/storage devices on the vbus. I presume your MIPS board lacks PCI, so you have to write the entries for System_bus and the Vbus yourself.

The helpful sections in another tutorial are: https://github.com/kernkonzept/manifest/wiki/HwPassThrough#ios-config-file https://github.com/kernkonzept/manifest/wiki/HwPassThrough#vbus-configuratio...

I found your mercurial instance for the l4re topic but didn't find any io & vbus configuration. Can you point me to your config files? And/Or send along the Io output with maximum verbosity (-vvvvvvv)?

Cheers, Philipp

On Tuesday, 23 May 2023 17:32:16 CEST Paul Boddie wrote:

But the DMA domain is still not found when assigning the space. I did wonder whether I might need to create a device to hold the DMA domain resource, but my attempt to implement this failed:

for resource in hw:resources() do print("resource name=" .. resource:id()) if resource:id() == 1145130308 then device = Io.Vi.System_bus() device:add_resource(resource) bus:add_child(device) end end

Well, this proved to be the wrong track, as I imagined, but the following helped me onto the right track:

Here, Io crashes...

IO | L4Re[rm]: unhandled read page fault at 0x31 pc=0x102aba0 IO | L4Re: rom/io: Unhandled exception: PC=0x102aba0 PFA=0x30 LdrFlgs=0x0

...which is the following line in Dma_domain::add_to_group:

if (!_v_domain && !g->_set)

I imagine that I am making this harder than it should be, though.

I already suspected that I needed a DMA domain inside an actual device, as opposed to the system bus, so that the code in Io would be able to find the resource. Obviously, the attempt above did not achieve this, indicating that there is more involved in the initialisation than simply "copying" a resource into a suitable location.

I then discovered that the Hw_device_DF_dma_supported property flag would allow a search using the vbus to yield a DMA domain resource with a valid identifier and for Io to find this by itself when assigning a DMA space to a domain. Although the SoC I am using has no notion of device-specific DMA domains, conceptually it makes sense to associate DMA with the peripheral I am trying to configure, so I introduced the following to my LCD controller device:

LCD = Hw.Device(function() Property.hid = "jz4780-lcd"; -- compatible = {"mips,jz4780-lcd"}; Resource.regs = Res.mmio(0x13050000, 0x130517ff); Resource.irq = Res.irq({31, 31}, Io.Resource.Irq_type_level_high); Property.flags = Io.Hw_device_DF_dma_supported; end);

However, one counterintuitive aspect of assigning a DMA space involves the DMA domain identifier. As suggested by the nvme-driver code, although a DMA domain has a resource identifier of 0x44414d44 ("DMAD"), the assign operation actually wants the start address associated with the resource. This turns out to be zero in my case.

I still wasn't completely done, however. The map operation was failing with what tends to be described in the error messages as a "bad, unknown runtime error". Eventually, I tracked this down to the _get_ds function found in pkg/ l4re-core/moe/server/src/dma_space.cc which has some L4Re::chksys invocations. These invocations are quite unhelpful, obscuring the actual error, which was due to the received DMA space capability not having write permission.

In fact, I see that the nvme-driver code for invoking the operation uses the make_cap_rw method on a L4::Ipc::Cap instance, whereas those using the C library have to set the rights bits the old-fashioned way. I don't think I have needed to do this for anything before, so this caught me out.

In any case, with all this tidied up, I finally got hold of those physical addresses again, and thankfully the rest of my existing code still worked. I do wonder about where I might put that Hw_device_DF_dma_supported property, but I imagine that even if it were replicated on all appropriate devices, it would not matter which device such a global domain would be acquired from.

Anyway, I hope this counts as some kind of feedback about how these frameworks are experienced by outsiders such as myself. I will now aim to pick up where I left off a few weeks ago...

Thanks for the help given!

Paul