Many Thanks Phillip It's clear now and the issue had been resolved 😊
On Tue, Nov 5, 2024 at 12:40 PM Philipp Eppelt < philipp.eppelt@kernkonzept.com> wrote:
Hi Mohamed,
Am 05.11.24 um 09:52 schrieb Mohamed Dawod:
Hi Philipp, Thanks for the clarification I changed the prop and cpus parameters for the two VMs to be like VM1 : prio=1, cpus=0x2, VM2 : prio=100, cpus=0xD,
Why did you chose a priority of 100? Be aware, that the kernel implements priority based round robin. Meaning the scheduler selects all threads with the highest priority that are runnable on a specific core and among these implements round robin scheduling. Only if these threads yield/block the lower priority levels are considered / get computation time.
Vm2 is configured to run on cores 0, 2, and 3. In the setup you sent last time, core 0 also runs ned, moe, io, cons and virtio switch. A prio of 100 usually means that your VM runs with higher priority than the services it depends upon (namely: io, cons, virtio switch).
If a thread of VM2 now requests a service of e.g. io, and core 0 of your VM runs, this request is delayed until core 0 of said VM yields/blocks. This is indeed a race condition and care must be taken when assigning priorities.
You can have a look into the kernel debugger JDB to see the priority assignments. If JDB is configured in the kernel config press `ESC` to enter the debugger and then press `lp` to see the list of threads. (press `h` for help) The non self-explanatory column names in full are:
- pr: priority
- sp: address space
- wait: ID of the thread this thread waits for
- to: IPC timeout
All threads of one application / task share have the same address space number. If you want to see the uvmm instances named in this list, pass jdb=1 to start_vm().
To change the priority of a service, assign a new scheduler proxy in the ned script. E.g. my entry for cons looks like this:
L4.default_loader:start( { scheduler = vmm.new_sched(0x40), log = L4.Env.log, caps = { cons = vmm.loader.log_fab:svr(), jdb = L4.Env.jdb }
}, "rom/cons -k -a");Cheers, Philipp
I think that the slow boot issue has been fixed.
But the hanging issue still persists. About 90% of running trials fail because of hanging one of the VMs! Sometimes, the VM hangs while Linux booting and sometimes it hangs after
booting
and logging successfully while using the VM!
The hanging behaviour seems like there are unresolved race conditions
causing
deadlock. Although it hangs, my host machine is working intensively and my laptop
fans
work loudly! My laptop has 8 cores CPU and 32 GB RAM.
What could be the problem?
On Mon, Nov 4, 2024 at 1:04 PM Philipp Eppelt <
philipp.eppelt@kernkonzept.com
mailto:philipp.eppelt@kernkonzept.com> wrote:
Hi Mohamed, a colleague just informed me about a recently fixed cause for a slowboot with
multiple VMs. Try this commit for uvmm, it's not part of the 2024-08 snapshot.https://github.com/kernkonzept/uvmm/commit/8c6b3080d69e9e2c82211388ba641241f... < https://github.com/kernkonzept/uvmm/commit/8c6b3080d69e9e2c82211388ba641241f...
Cheers, Philipp Am 04.11.24 um 11:36 schrieb Philipp Eppelt: > Hi Mohamed, > > > Am 04.11.24 um 06:26 schrieb Mohamed Dawod: >> Hi Philipp, >> >> I'm already using *start_vm()* and setting */cpu/* parameter tothe
values you >> mentioned, but it is still working randomly (sometimes works andsometimes
>> hangs or one of the VM hangs). >> >> Please find attached my ned script : > A note on the script: > You are using "console=hvc0" which is the virtio-console device.This is
> initialized only late in the linux boot process. For earlyconsole output
I use > "console=ttyS0 earlyprintk=serial,ttyS0" in the bootargs string. > > Since you are using hvc0, I assume you have changed > uvmm/configs/dts/virt-pc.dts, and enabled the virtio_uart node. I recommend to > also change the uart8250 node to include the l4vmm,vcon_cap line.Then
you also > need to provide the capability named "uart" to the uvmm. You cando this
via the > ext_caps parameter. For example: > > ext_caps = { uart = vmm.loader.log_fab:create(L4.Proto.Log,"uart") }
> > Change the "uart" string in the create() function to somethingyou like to
> distinguish the two uvmms. > >> >> Also I can not understand the effect of the */prio/* parameter! >> I have changed it among different values but nothing changed! >> What is the effect of the */prio/* parameter and when/how can Iuse it ?
> > I assume you are using the 2024-08 snapshot. In this version,both, the
prio and > cpus parameters are necessary to take effect. (A later versionmakes this
more > user friendly see [1].) > > In more detail: To assign apps like the uvmm to specific cores,L4Re uses
> scheduling proxies, which ensure the application scheduled ontopof the
> scheduling proxy have only access to the resources managed by theproxy.
Here > resources means cores and priority range. > > If given prio and cpus parameter, start_vm() creates a schedulingproxy with
> these parameters and limiting priority range to prio+10. > This means: > vm1 runs on cores 0xc with a priority range of [255, 255](min/max).
> vm2 runs on cores 0x3 with a priority range of [3, 13]. > > 3 and 255 are the interesting numbers here: The base priority forl4re
apps is 2 > and the given prio parameter just adds to this, so 2+1=3. 2+12345is 255,
> because 255 is the maximum priority level. > > This can be source of the slowdown behavior you are observing,since this
> priority level is the same as for services vm1 depends upon. > My recommendation would be prio=2 for vm1. > > Cheers, > Philipp > > > p.s. I haven't forgotten about your arm64 PCI MSI question, Ijust need some
> time to set this up myself to be able to give a good answer. > > > [1]https://github.com/kernkonzept/uvmm/blob/master/configs/vmm.lua#L40
<https://github.com/kernkonzept/uvmm/blob/master/configs/vmm.lua#L40 > > >> >> Thanks in advance, >> Regards >> >> On Thu, Oct 31, 2024 at 8:47 PM Philipp Eppelt <philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com>
>> <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com>>> wrote: >> >> Hi Mohamed, >> >> Am 31.10.24 um 14:30 schrieb Mohamed Dawod: >> > Thanks Philipp, >> > >> > Multiple CPUs worked for virt-arm64 machine >> Yippie! >> >> > I tried to launch 2 linux VMs on top of L4 using uvmm andassign
2 CPUs >> to one >> > of the two VMs and another 2 CPUs to the other one. >> > I noticed that the linux booting process becomes slowerand as more
>> CPUs are >> > added to qemu with -smp option and passed to the VMs asmore as
the VM >> booting >> > becomes more slower! >> > Also VMs become working randomly (sometimes they work and sometimes they >> hang or >> > one of them hangs) > >> > Why does this strange behaviour happen when using uvmmand 2
Linux VMs ? >> To make this easier please show me your ned script startingthe VMs.
>> >> If you use start_vm() please note that the `cpus` parametertakes a
>> bitmap. So >> make sure to start VM1 with `cpus=0x3` and VM2 with`cpus=0xc` to place
>> them on >> separate cores of a four core platform (e.g. QEMU with -smp4).
>> >> Cheers, >> Philipp >> >> > >> > Thanks, >> > Regards >> > >> > On Wed, Oct 30, 2024 at 8:46 PM Philipp Eppelt >> <philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com> <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com>>
>> > <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com> >> <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com>>>> wrote: >> > >> > Hi Mohamed, >> > >> > Am 29.10.24 um 10:43 schrieb Mohamed Dawod: >> > > Hello, >> > > I'm trying to provide multiple CPUs for a linux VMon top
of L4. >> > > I'm using the qemu virt machine and building foraarch64. so I
>> used *-smp* >> > > option to provide more CPUs. >> > > >> > > $ qemu-system-aarch64 -Mvirt,virtualization=true -cpu
>> cortex-a57 >> > -smp 4 -m >> > > 1024 -kernel ....etc.... >> > I'm not sure which gic version qemu uses. Please trysetting it
>> explicitly to >> > with gic-version=3 argument: `-M >> virt,virtualization=true,gic-version=3` >> > >> > > >> > > Unfortunately, This didn't work. I tried to addmore CPU
device >> nodes to >> > the dts >> > > file *virt-arm_virt-64.dts *but it also didn'twork. >
>> > > I think that it's because of the provided interrupt-controller with >> > > *virt-arm_virt-64.dts* in /l4/pkg/uvmm/conf/dts/which
mentioned >> that it >> > > supports only one CPU. >> > > >> > > icsoc { >> > > compatible = "simple-bus"; >> > > #address-cells = <2>; >> > > #size-cells = <2>; >> > > ranges; >> > > >> > > /* Uvmm will adapt the compatiblestring
depending >> on the >> > present gic >> > > * version. It expects reg entriesthat provide
>> enough space >> > for the >> > > * Cpu/Dist interface for gicv2 (atleast 0x1000,
>> 0x1000) or the >> > > * Dist/Redist interface for gicv3(0x10000,
0x20000 * >> > number of cpus). >> > >> > I'm not an expert for ARM64, but judging from theline above I'd
>> say you >> > have to >> > increase the size of the second reg entry. Forexample for
four cores: >> > reg = <0 0x40000 0 0x10000>, >> > <0 0x50000 0 0x80000>; >> > >> > You should be able to just use the github version ofthis
file, it >> has a gic >> > node that is configured for 32 cores and comes withfour CPU
nodes. >> > >> >>https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64... < https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64... < https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64... < https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64... < https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64... < https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64... < https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64... < https://github.com/kernkonzept/uvmm/blob/master/configs/dts/virt-arm_virt-64...
>> > >> > >> > > * *The entries provided here supportany
gicv2 setup >> or a >> > gicv3 setup >> > > * with one Cpu.* >> > > */ >> > > gic: interrupt-controller { >> > > compatible = "arm,gic-400", "arm,cortex-a15-gic", >> > > "arm,cortex-a9-gic"; >> > > #interrupt-cells = <3>; >> > > #address-cells = <0>; >> > > interrupt-controller; >> > > reg = <0 0x40000 0 0x10000>, >> > > <0 0x50000 0 0x20000>; >> > > }; >> > > }; >> > > >> > > >> > > My question now, is there any workaround to support multiple CPUs >> for virt >> > > machine on arm64 ? >> > >> > Multiple CPUs should work. For SMP there are a coupleof
things to >> consider: >> > - QEMU: -smp parameter >> > - Kernel configuration for SMP and the number ofmaximum cores
>> > - The DTS defines the maximum number of cores for theuvmm
will set >> up. So >> > adding CPU device nodes is the correct path. >> > - The ned script defines the number of coresavailable at
runtime to >> uvmm. No >> > cpu parameter in the start_vm({}) call means the VMgets
access to >> all cpus. >> > - Linux must of course also support SMP, but that'svery
likely not >> the problem >> > here ;-) >> > >> > I hope this sheds some light. >> > >> > Cheers, >> > Philipp >> > >> > -- >> > philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com> <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com>>
>> <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com> >> <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com>>> - >> > Tel. 0351-41 883 221 >> > http://www.kernkonzept.com <http://www.kernkonzept.com> <http://www.kernkonzept.com <http://www.kernkonzept.com>> >> <http://www.kernkonzept.com <http://www.kernkonzept.com> <http://www.kernkonzept.com <http://www.kernkonzept.com>>> >> > >> > Kernkonzept GmbH. Sitz: Dresden. AmtsgerichtDresden, HRB
31129. >> > Geschäftsführer: Dr.-Ing. Michael Hohmuth >> > _______________________________________________ >> > l4-hackers mailing list --l4-hackers@os.inf.tu-dresden.de
<mailto:l4-hackers@os.inf.tu-dresden.de> >> <mailto:l4-hackers@os.inf.tu-dresden.de <mailto:l4-hackers@os.inf.tu-dresden.de>> >> > <mailto:l4-hackers@os.inf.tu-dresden.de <mailto:l4-hackers@os.inf.tu-dresden.de> >> <mailto:l4-hackers@os.inf.tu-dresden.de <mailto:l4-hackers@os.inf.tu-dresden.de>>> >> > To unsubscribe send an email to l4-hackers-leave@os.inf.tu-dresden.de <mailto:l4-hackers-leave@os.inf.tu-dresden.de> >> <mailto:l4-hackers-leave@os.inf.tu-dresden.de <mailto:l4-hackers-leave@os.inf.tu-dresden.de>> >> > <mailto:l4-hackers-leave@os.inf.tu-dresden.de <mailto:l4-hackers-leave@os.inf.tu-dresden.de> >> <mailto:l4-hackers-leave@os.inf.tu-dresden.de <mailto:l4-hackers-leave@os.inf.tu-dresden.de>>> >> > >> > >> > >> > *Driving Innovation! Visit our website www.avelabs.com <http://www.avelabs.com> >> <http://www.avelabs.com <http://www.avelabs.com>> >> > <http://www.avelabs.com/ <http://www.avelabs.com/> <http://www.avelabs.com/ <http://www.avelabs.com/>>>*, to readAvelabs
>> Confidentiality Notice, follow this >> > link: http://www.avelabs.com/email/disclaimer.html <http://www.avelabs.com/email/disclaimer.html> >> <http://www.avelabs.com/email/disclaimer.html <http://www.avelabs.com/email/disclaimer.html>> >> > <http://www.avelabs.com/email/disclaimer.html <http://www.avelabs.com/email/disclaimer.html> >> <http://www.avelabs.com/email/disclaimer.html <http://www.avelabs.com/email/disclaimer.html>>> >> >> -- philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com> >> <mailto:philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com>> - >> Tel. 0351-41 883 221 >> http://www.kernkonzept.com <http://www.kernkonzept.com> <http://www.kernkonzept.com <http://www.kernkonzept.com>> >> >> Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB
>> Geschäftsführer: Dr.-Ing. Michael Hohmuth >> >> >> >> *Driving Innovation! Visit our website www.avelabs.com <http://www.avelabs.com> >> <http://www.avelabs.com/ <http://www.avelabs.com/>>*, to readAvelabs
Confidentiality Notice, follow >> this link: http://www.avelabs.com/email/disclaimer.html <http://www.avelabs.com/email/disclaimer.html> >> <http://www.avelabs.com/email/disclaimer.html <http://www.avelabs.com/email/disclaimer.html>> > > > _______________________________________________ > l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de <mailto:l4-hackers@os.inf.tu-dresden.de> > To unsubscribe send an email tol4-hackers-leave@os.inf.tu-dresden.de
<mailto:l4-hackers-leave@os.inf.tu-dresden.de> -- philipp.eppelt@kernkonzept.com <mailto:philipp.eppelt@kernkonzept.com> -
Tel. 0351-41 883 221 http://www.kernkonzept.com <http://www.kernkonzept.com> Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. Geschäftsführer: Dr.-Ing. Michael Hohmuth _______________________________________________ l4-hackers mailing list -- l4-hackers@os.inf.tu-dresden.de <mailto:l4-hackers@os.inf.tu-dresden.de> To unsubscribe send an email tol4-hackers-leave@os.inf.tu-dresden.de
<mailto:l4-hackers-leave@os.inf.tu-dresden.de>*Driving Innovation! Visit our website www.avelabs.com http://www.avelabs.com/*, to read Avelabs Confidentiality Notice,
follow this
link: http://www.avelabs.com/email/disclaimer.html http://www.avelabs.com/email/disclaimer.html
-- philipp.eppelt@kernkonzept.com - Tel. 0351-41 883 221 http://www.kernkonzept.com
Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. Geschäftsführer: Dr.-Ing. Michael Hohmuth