Hi,
I've been looking for a way to run some sort of ping-pong example where each task runs on either the same or different CPUs.
I've been unable to see if moe's config files allow to start tasks pinned to a specific CPU, and after running the "thread_migrate" example, looks like only one CPU (CPU00) is always used:
$ make -C obj/l4/amd64/ qemu E=thread_migrate QEMU_OPTIONS="-serial stdio -vnc ::59000 -smp 2" [...] Not using serial hack in slow timer handler. CPU[0]: AuthenticAMD (6:2:3:0)[00000623] Model: QEMU Virtual CPU version 1.5.1 at 3092MHz
255/512 Entry I TLB (4K pages) 255 Entry I TLB (4M pages) 255/512 Entry D TLB (4K pages) 255 Entry D TLB (4M pages) 64 KB L1 I Cache (2-way associative, 64 bytes per line) 64 KB L1 D Cache (2-way associative, 64 bytes per line) 512 KB L2 U Cache (8-way associative, 64 bytes per line)
Freeing init code/data: 20480 bytes (5 pages) [...] Ned says: Hi World! Ned: loading file: 'rom/thread_migrate.cfg' migrate | Found 4 CPUs. migrate | Created 12 threads. migrate | Migrated Thread00 -> CPU00 migrate | Migrated Thread01 -> CPU00 migrate | Migrated Thread02 -> CPU00 migrate | Migrated Thread03 -> CPU00 migrate | Migrated Thread04 -> CPU00 migrate | Migrated Thread05 -> CPU00 migrate | Migrated Thread06 -> CPU00 migrate | Migrated Thread07 -> CPU00 migrate | Migrated Thread08 -> CPU00 migrate | Migrated Thread09 -> CPU00 migrate | Migrated Thread10 -> CPU00 migrate | Migrated Thread11 -> CPU00 migrate | Migrated Thread00 -> CPU00 migrate | Migrated Thread01 -> CPU00 migrate | Migrated Thread02 -> CPU00 migrate | Migrated Thread03 -> CPU00 migrate | Migrated Thread04 -> CPU00 migrate | Migrated Thread05 -> CPU00 migrate | Migrated Thread06 -> CPU00 migrate | Migrated Thread07 -> CPU00 migrate | Migrated Thread08 -> CPU00 migrate | Migrated Thread09 -> CPU00 migrate | Migrated Thread10 -> CPU00 migrate | Migrated Thread11 -> CPU00 [... same output forever ...]
I've modified "src/l4/conf/modules.list" to add the "thread_migrate" test, and I'm using l4re-snapshot-2013060718, with multiprocessing (with a maximum of 4) enabled on fiasco.
Looks like the benchmark always detects (L4::Scheduler::info) the number of maximum CPUs configured in fiasco, but not the number of CPUs provided by QEMU (only CPU 0 is online).
Am I missing some step to set all CPUs online? Is it possible to have MOE start tasks pinned to a specific CPU?
Thanks a lot, Lluis
On Mon Aug 05, 2013 at 21:50:55 +0300, Lluís Vilanova wrote:
I've been unable to see if moe's config files allow to start tasks pinned to a specific CPU, and after running the "thread_migrate" example, looks like only one CPU (CPU00) is always used:
$ make -C obj/l4/amd64/ qemu E=thread_migrate QEMU_OPTIONS="-serial stdio -vnc ::59000 -smp 2" [...] Not using serial hack in slow timer handler. CPU[0]: AuthenticAMD (6:2:3:0)[00000623] Model: QEMU Virtual CPU version 1.5.1 at 3092MHz
255/512 Entry I TLB (4K pages) 255 Entry I TLB (4M pages) 255/512 Entry D TLB (4K pages) 255 Entry D TLB (4M pages) 64 KB L1 I Cache (2-way associative, 64 bytes per line) 64 KB L1 D Cache (2-way associative, 64 bytes per line) 512 KB L2 U Cache (8-way associative, 64 bytes per line)
Freeing init code/data: 20480 bytes (5 pages) [...] Ned says: Hi World! Ned: loading file: 'rom/thread_migrate.cfg' migrate | Found 4 CPUs. migrate | Created 12 threads. migrate | Migrated Thread00 -> CPU00 migrate | Migrated Thread01 -> CPU00
I've modified "src/l4/conf/modules.list" to add the "thread_migrate" test, and I'm using l4re-snapshot-2013060718, with multiprocessing (with a maximum of 4) enabled on fiasco.
Looks like the benchmark always detects (L4::Scheduler::info) the number of maximum CPUs configured in fiasco, but not the number of CPUs provided by QEMU (only CPU 0 is online).
That it finds 4 is on purpose because more CPUs could come up. Does it work giving Qemu '-cpu core2duo'?
Am I missing some step to set all CPUs online? Is it possible to have MOE start tasks pinned to a specific CPU?
See the (commented out) "scheduler = ... " settings in, e.g., x86-fb.cfg and give a fourth parameter, which is a CPU mask. You can check in the kernel debugger on which CPU a thread is (via lp command).
Adam
Adam Lackorzynski writes:
On Mon Aug 05, 2013 at 21:50:55 +0300, Lluís Vilanova wrote:
I've been unable to see if moe's config files allow to start tasks pinned to a specific CPU, and after running the "thread_migrate" example, looks like only one CPU (CPU00) is always used:
$ make -C obj/l4/amd64/ qemu E=thread_migrate QEMU_OPTIONS="-serial stdio -vnc ::59000 -smp 2" [...] Not using serial hack in slow timer handler. CPU[0]: AuthenticAMD (6:2:3:0)[00000623] Model: QEMU Virtual CPU version 1.5.1 at 3092MHz
255/512 Entry I TLB (4K pages) 255 Entry I TLB (4M pages) 255/512 Entry D TLB (4K pages) 255 Entry D TLB (4M pages) 64 KB L1 I Cache (2-way associative, 64 bytes per line) 64 KB L1 D Cache (2-way associative, 64 bytes per line) 512 KB L2 U Cache (8-way associative, 64 bytes per line)
Freeing init code/data: 20480 bytes (5 pages) [...] Ned says: Hi World! Ned: loading file: 'rom/thread_migrate.cfg' migrate | Found 4 CPUs. migrate | Created 12 threads. migrate | Migrated Thread00 -> CPU00 migrate | Migrated Thread01 -> CPU00
I've modified "src/l4/conf/modules.list" to add the "thread_migrate" test, and I'm using l4re-snapshot-2013060718, with multiprocessing (with a maximum of 4) enabled on fiasco.
Looks like the benchmark always detects (L4::Scheduler::info) the number of maximum CPUs configured in fiasco, but not the number of CPUs provided by QEMU (only CPU 0 is online).
That it finds 4 is on purpose because more CPUs could come up. Does it work giving Qemu '-cpu core2duo'?
Yes, looks like making QEMU look like a somewhat more realistic CPU does no choke Fiasco's ability to detect the number of CPUs.
Still, L4::Scheduler::info says there's as many CPUs as the maximum number configured into Fiasco (not the number offered by QEMU). I'm just not sure if this should be considered a bug or a feature.
Am I missing some step to set all CPUs online? Is it possible to have MOE start tasks pinned to a specific CPU?
See the (commented out) "scheduler = ... " settings in, e.g., x86-fb.cfg and give a fourth parameter, which is a CPU mask. You can check in the kernel debugger on which CPU a thread is (via lp command).
Works like a charm, thanks.
Lluis
On Tue Aug 06, 2013 at 14:55:33 +0300, Lluís Vilanova wrote:
Adam Lackorzynski writes:
On Mon Aug 05, 2013 at 21:50:55 +0300, Lluís Vilanova wrote:
I've been unable to see if moe's config files allow to start tasks pinned to a specific CPU, and after running the "thread_migrate" example, looks like only one CPU (CPU00) is always used:
$ make -C obj/l4/amd64/ qemu E=thread_migrate QEMU_OPTIONS="-serial stdio -vnc ::59000 -smp 2" [...] Not using serial hack in slow timer handler. CPU[0]: AuthenticAMD (6:2:3:0)[00000623] Model: QEMU Virtual CPU version 1.5.1 at 3092MHz
255/512 Entry I TLB (4K pages) 255 Entry I TLB (4M pages) 255/512 Entry D TLB (4K pages) 255 Entry D TLB (4M pages) 64 KB L1 I Cache (2-way associative, 64 bytes per line) 64 KB L1 D Cache (2-way associative, 64 bytes per line) 512 KB L2 U Cache (8-way associative, 64 bytes per line)
Freeing init code/data: 20480 bytes (5 pages) [...] Ned says: Hi World! Ned: loading file: 'rom/thread_migrate.cfg' migrate | Found 4 CPUs. migrate | Created 12 threads. migrate | Migrated Thread00 -> CPU00 migrate | Migrated Thread01 -> CPU00
I've modified "src/l4/conf/modules.list" to add the "thread_migrate" test, and I'm using l4re-snapshot-2013060718, with multiprocessing (with a maximum of 4) enabled on fiasco.
Looks like the benchmark always detects (L4::Scheduler::info) the number of maximum CPUs configured in fiasco, but not the number of CPUs provided by QEMU (only CPU 0 is online).
That it finds 4 is on purpose because more CPUs could come up. Does it work giving Qemu '-cpu core2duo'?
Yes, looks like making QEMU look like a somewhat more realistic CPU does no choke Fiasco's ability to detect the number of CPUs.
The problem is some unfortunate handling of the startup sequence in Qemu when using the AMD model (can't remember details right now).
Still, L4::Scheduler::info says there's as many CPUs as the maximum number configured into Fiasco (not the number offered by QEMU). I'm just not sure if this should be considered a bug or a feature.
Fiasco does not know what the maximum number of CPUs on that particular machine will ever be, so it just says what it can support. Also, one has to wait a bit until all CPUs are up because Fiasco does not wait to start sigma0+moe until all CPUs are up (e.g. see migrate example).
Adam
Adam Lackorzynski writes: [...]
Still, L4::Scheduler::info says there's as many CPUs as the maximum number configured into Fiasco (not the number offered by QEMU). I'm just not sure if this should be considered a bug or a feature.
Fiasco does not know what the maximum number of CPUs on that particular machine will ever be, so it just says what it can support. Also, one has to wait a bit until all CPUs are up because Fiasco does not wait to start sigma0+moe until all CPUs are up (e.g. see migrate example).
Sorry, I just realized that what I misunderstood is the output of the migrate example, not the description of L4::Scheduler::info:
[...] static l4_umword_t cpu_map, cpu_nrs; [...] static int check_cpus(void) { l4_sched_cpu_set_t cs = l4_sched_cpu_set(0, 0);
if (l4_error(L4Re::Env::env()->scheduler()->info(&cpu_nrs, &cs)) < 0) return 1;
cpu_map = cs.map;
printf("Found %ld CPUs.\n", cpu_nrs); [...]
Maybe changing it to "Supports up to %ld CPUs" would be clearer, or instead (or in addition) calculating the number of detected/online CPUs.
Thanks, Lluis
On Tue Aug 06, 2013 at 23:47:58 +0300, Lluís Vilanova wrote:
Adam Lackorzynski writes: [...]
Still, L4::Scheduler::info says there's as many CPUs as the maximum number configured into Fiasco (not the number offered by QEMU). I'm just not sure if this should be considered a bug or a feature.
Fiasco does not know what the maximum number of CPUs on that particular machine will ever be, so it just says what it can support. Also, one has to wait a bit until all CPUs are up because Fiasco does not wait to start sigma0+moe until all CPUs are up (e.g. see migrate example).
Sorry, I just realized that what I misunderstood is the output of the migrate example, not the description of L4::Scheduler::info:
Maybe changing it to "Supports up to %ld CPUs" would be clearer, or instead (or in addition) calculating the number of detected/online CPUs.
True, I reworded it.
Adam
l4-hackers@os.inf.tu-dresden.de