Kernel allocated memory not visible in L4Re virtual device despite correct physical RAM mapping
Hi l4-hackers, I am trying to pass a kernel-allocated memory buffer to a virtual device, but I am running into an issue. I would appreciate any help or suggestions. 1.Kernel side(driver) In my platform driver, I allocate memory using devm_kzalloc, write a string into it, and pass its physical address to the device via an MMIO register: …… void* buf = devm_kzalloc(&pdev->dev, 0x200, GFP_KERNEL); if (!buf) return -ENOMEM; phys_addr_t pa = virt_to_phys(buf); char *str = "hi stephen"; memcpy(buf, str, strlen(str) + 1); wmb(); writeq((u64)pa, hb->base + 0x00); …… My understanding is: allocate kernel memory → write data → pass physical address to the device. 2. Memory layout I observed that the allocated physical address falls within the RAM regions configured via bootargs : 0x66200000 - 0x6fffffff So my assumption was that if the virtual device maps this physical memory region, it should be able to access the data written by the kernel. 3. Virtual device side (L4Re) On the virtual device side, I map the physical memory using L4Re RM attach: void write(unsigned reg, char /*size*/, l4_uint64_t value, unsigned) { if (reg == 0x00) { addr = value; auto d = L4Re::Env::env()->get_cap<L4Re::Dataspace>("lram"); l4_addr_t vaddr; l4_addr_t base = 0x66200000; l4_addr_t size = 0x09E00000; int r = L4Re::Env::env()->rm()->attach( &vaddr, size, L4Re::Rm::F::Search_addr | L4Re::Rm::F::RW | L4Re::Rm::F::Cache_uncached, L4::Ipc::make_cap_rw(d), 0, L4_PAGESHIFT); if (r) { printf("can't attach hb memory\n"); return; } char *dst = reinterpret_cast<char *>(vaddr + (addr - base)); for (int i = 0; i < 20; i++) { printf("%02x ", (unsigned char)dst[i]); } printf("\n"); printf("get msg for guest: %s\n", dst); } } 4.Problem However, I cannot read the string written by the kernel. The memory content I get appears to be uninitialized or unrelated data. 5.Question I can confirm that the lram dataspace capability does correctly correspond to the physical memory region 0x66200000 - 0x6fffffff, and this region is valid and usable RAM on the system. I am unsure where the issue lies: (1) Is this approach (kernel allocation → pass physical address → direct mapping in virtual device) actually valid? (2) Could this be a cache coherence / DMA issue? (3) Or is there something incorrect in my L4Re RM attach usage? Any insights would be greatly appreciated.
Hi Stephen, the memory in Linux is mapped cached, while in your L4Re side, you mapped it uncached. Did you try mapping it cached on the L4Re side? Or you do a cache clean on the str string in Linux. BR, Adam On Mon Apr 13, 2026 at 14:53:00 +0800, stephen.yang wrote:
Hi l4-hackers, I am trying to pass a kernel-allocated memory buffer to a virtual device, but I am running into an issue. I would appreciate any help or suggestions.
1.Kernel side(driver) In my platform driver, I allocate memory using devm_kzalloc, write a string into it, and pass its physical address to the device via an MMIO register: ¡¡ void* buf = devm_kzalloc(&pdev->dev, 0x200, GFP_KERNEL); if (!buf) return -ENOMEM; phys_addr_t pa = virt_to_phys(buf); char *str = "hi stephen"; memcpy(buf, str, strlen(str) + 1); wmb(); writeq((u64)pa, hb->base + 0x00); ¡¡
My understanding is: allocate kernel memory ¡ú write data ¡ú pass physical address to the device.
2. Memory layout I observed that the allocated physical address falls within the RAM regions configured via bootargs : 0x66200000 - 0x6fffffff So my assumption was that if the virtual device maps this physical memory region, it should be able to access the data written by the kernel.
3. Virtual device side (L4Re) On the virtual device side, I map the physical memory using L4Re RM attach: void write(unsigned reg, char /*size*/, l4_uint64_t value, unsigned) { if (reg == 0x00) { addr = value;
auto d = L4Re::Env::env()->get_cap<L4Re::Dataspace>("lram");
l4_addr_t vaddr; l4_addr_t base = 0x66200000; l4_addr_t size = 0x09E00000; int r = L4Re::Env::env()->rm()->attach( &vaddr, size, L4Re::Rm::F::Search_addr | L4Re::Rm::F::RW | L4Re::Rm::F::Cache_uncached, L4::Ipc::make_cap_rw(d), 0, L4_PAGESHIFT); if (r) { printf("can't attach hb memory\n"); return; } char *dst = reinterpret_cast<char *>(vaddr + (addr - base)); for (int i = 0; i < 20; i++) { printf("%02x ", (unsigned char)dst[i]); } printf("\n"); printf("get msg for guest: %s\n", dst); } }
4.Problem However, I cannot read the string written by the kernel. The memory content I get appears to be uninitialized or unrelated data.
5.Question I can confirm that the lram dataspace capability does correctly correspond to the physical memory region 0x66200000 - 0x6fffffff, and this region is valid and usable RAM on the system.
I am unsure where the issue lies: (1) Is this approach (kernel allocation ¡ú pass physical address ¡ú direct mapping in virtual device) actually valid? (2) Could this be a cache coherence / DMA issue? (3) Or is there something incorrect in my L4Re RM attach usage?
Any insights would be greatly appreciated.
_______________________________________________ l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de
Hi Adam, Thank you very much for your reply and suggestions. I have tried making changes according to your recommendations, but they did not seem to take effect. However, I have discovered something interesting: when disabling forced mapping and using the method I mentioned in my previous email, the virtual device is able to read the content that the kernel writes to that address range. As described in the documentation： -i, --ram-identity-mapping When set, the option forces the guest RAM to be mapped to its corresponding host-physical addresses. Flag. True if provided. Additionally, I found that even with forced mapping enabled, the physical address seen by the guest is different from that seen by the host, with an offset between them.The host sees the address as 0x66200000, while the guest sees it as 0x66400000.Based on the code mentioned earlier, this issue can be resolved simply by accounting for this offset during the mapping process. BR, Stephen
Hi Stephen, great it is working for you now. All the addresses in a virtualization setup can be indeed confusing. With a virtual device using memory of the VM, the virtual device needs to know about the guest physical mapping, or better the mapping of the dataspace into the VMs guest physical space. -i can make this a bit easier but it's really just a workaround. Adam On Sat Apr 18, 2026 at 14:22:16 +0800, stephen.yang wrote:
Hi Adam, Thank you very much for your reply and suggestions. I have tried making changes according to your recommendations, but they did not seem to take effect. However, I have discovered something interesting: when disabling forced mapping and using the method I mentioned in my previous email, the virtual device is able to read the content that the kernel writes to that address range. As described in the documentation£º -i, --ram-identity-mapping When set, the option forces the guest RAM to be mapped to its corresponding host-physical addresses. Flag. True if provided.
Additionally, I found that even with forced mapping enabled, the physical address seen by the guest is different from that seen by the host, with an offset between them.The host sees the address as 0x66200000, while the guest sees it as 0x66400000.Based on the code mentioned earlier, this issue can be resolved simply by accounting for this offset during the mapping process. BR, Stephen
Hi Adam, Your point is well taken. It is not feasible to disable forced mapping simply based on this example. First, this address offset cannot be obtained directly, as no relevant interface is provided. The same physical memory region perceived by the guest and the host actually corresponds to different addresses on each side. Second, even if this offset were accessible, it would allow the virtual device to access the entire system memory space, which poses significant security risks to the system. Therefore, the approach shown in this example is not viable in practice. Nevertheless, this discussion has deepened my understanding of the L4Re framework and virtualization technologies. Thank you again for your reply. BR，Stephen At 2026-04-22 12:59:29, "Adam Lackorzynski" <adam@l4re.org> wrote:
Hi Stephen,
great it is working for you now. All the addresses in a virtualization setup can be indeed confusing. With a virtual device using memory of the VM, the virtual device needs to know about the guest physical mapping, or better the mapping of the dataspace into the VMs guest physical space. -i can make this a bit easier but it's really just a workaround.
Adam
On Sat Apr 18, 2026 at 14:22:16 +0800, stephen.yang wrote:
Hi Adam, Thank you very much for your reply and suggestions. I have tried making changes according to your recommendations, but they did not seem to take effect. However, I have discovered something interesting: when disabling forced mapping and using the method I mentioned in my previous email, the virtual device is able to read the content that the kernel writes to that address range. As described in the documentation£º -i, --ram-identity-mapping When set, the option forces the guest RAM to be mapped to its corresponding host-physical addresses. Flag. True if provided.
Additionally, I found that even with forced mapping enabled, the physical address seen by the guest is different from that seen by the host, with an offset between them.The host sees the address as 0x66200000, while the guest sees it as 0x66400000.Based on the code mentioned earlier, this issue can be resolved simply by accounting for this offset during the mapping process. BR, Stephen
Hi Stephen, not sure my understanding is right, you want a small piece of the guest's memory to be shared with another L4Re program? One way to do this is to give the VM multiple dataspaces as its memory and share one of those dataspaces with the application. uvmm supports giving multiple dataspaces to a VM. Inside the VM the device tree tells where the memory is. The driver inside the VM needs to be aware of that split and act accordingly. Sharing just a part of a dataspace is indeed not possible as no such functionality exists. BR, Adam On Sun Apr 26, 2026 at 13:18:05 +0800, stephen.yang wrote:
Hi Adam, Your point is well taken. It is not feasible to disable forced mapping simply based on this example. First, this address offset cannot be obtained directly, as no relevant interface is provided. The same physical memory region perceived by the guest and the host actually corresponds to different addresses on each side. Second, even if this offset were accessible, it would allow the virtual device to access the entire system memory space, which poses significant security risks to the system. Therefore, the approach shown in this example is not viable in practice. Nevertheless, this discussion has deepened my understanding of the L4Re framework and virtualization technologies. Thank you again for your reply.
BR，Stephen
At 2026-04-22 12:59:29, "Adam Lackorzynski" <adam@l4re.org> wrote:
Hi Stephen,
great it is working for you now. All the addresses in a virtualization setup can be indeed confusing. With a virtual device using memory of the VM, the virtual device needs to know about the guest physical mapping, or better the mapping of the dataspace into the VMs guest physical space. -i can make this a bit easier but it's really just a workaround.
Adam
On Sat Apr 18, 2026 at 14:22:16 +0800, stephen.yang wrote:
Hi Adam, Thank you very much for your reply and suggestions. I have tried making changes according to your recommendations, but they did not seem to take effect. However, I have discovered something interesting: when disabling forced mapping and using the method I mentioned in my previous email, the virtual device is able to read the content that the kernel writes to that address range. As described in the documentation£º -i, --ram-identity-mapping When set, the option forces the guest RAM to be mapped to its corresponding host-physical addresses. Flag. True if provided.
Additionally, I found that even with forced mapping enabled, the physical address seen by the guest is different from that seen by the host, with an offset between them.The host sees the address as 0x66200000, while the guest sees it as 0x66400000.Based on the code mentioned earlier, this issue can be resolved simply by accounting for this offset during the mapping process. BR, Stephen
Hi Adam, Sorry for the delayed reply. You are exactly correct. I have fully understood your solution of assigning multiple dataspaces to the VM and sharing one of them with the application, and I have already carried out some work on this basis. My original intention was to try sharing just part of a single dataspace. And as verified in my earlier tests, this is indeed not feasible due to lack of such capability. Thanks a lot for your valuable answer. BR, Stephen ---- Replied Message ---- | From | Adam Lackorzynski<adam@l4re.org> | | Date | 05/04/2026 12:53 | | To | stephen.yang<yy18513676366@163.com> | | Cc | l4-hackers<l4-hackers@os.inf.tu-dresden.de> | | Subject | Re: Kernel allocated memory not visible in L4Re virtual device despite correct physical RAM mapping | Hi Stephen, not sure my understanding is right, you want a small piece of the guest's memory to be shared with another L4Re program? One way to do this is to give the VM multiple dataspaces as its memory and share one of those dataspaces with the application. uvmm supports giving multiple dataspaces to a VM. Inside the VM the device tree tells where the memory is. The driver inside the VM needs to be aware of that split and act accordingly. Sharing just a part of a dataspace is indeed not possible as no such functionality exists. BR, Adam On Sun Apr 26, 2026 at 13:18:05 +0800, stephen.yang wrote:
Hi Adam, Your point is well taken. It is not feasible to disable forced mapping simply based on this example. First, this address offset cannot be obtained directly, as no relevant interface is provided. The same physical memory region perceived by the guest and the host actually corresponds to different addresses on each side. Second, even if this offset were accessible, it would allow the virtual device to access the entire system memory space, which poses significant security risks to the system. Therefore, the approach shown in this example is not viable in practice. Nevertheless, this discussion has deepened my understanding of the L4Re framework and virtualization technologies. Thank you again for your reply.
BR，Stephen
At 2026-04-22 12:59:29, "Adam Lackorzynski" <adam@l4re.org> wrote:
Hi Stephen,
great it is working for you now. All the addresses in a virtualization setup can be indeed confusing. With a virtual device using memory of the VM, the virtual device needs to know about the guest physical mapping, or better the mapping of the dataspace into the VMs guest physical space. -i can make this a bit easier but it's really just a workaround.
Adam
On Sat Apr 18, 2026 at 14:22:16 +0800, stephen.yang wrote:
Hi Adam, Thank you very much for your reply and suggestions. I have tried making changes according to your recommendations, but they did not seem to take effect. However, I have discovered something interesting: when disabling forced mapping and using the method I mentioned in my previous email, the virtual device is able to read the content that the kernel writes to that address range. As described in the documentation£º -i, --ram-identity-mapping When set, the option forces the guest RAM to be mapped to its corresponding host-physical addresses. Flag. True if provided.
Additionally, I found that even with forced mapping enabled, the physical address seen by the guest is different from that seen by the host, with an offset between them.The host sees the address as 0x66200000, while the guest sees it as 0x66400000.Based on the code mentioned earlier, this issue can be resolved simply by accounting for this offset during the mapping process. BR, Stephen
l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de
participants (2)
-
Adam Lackorzynski -
stephen.yang