[evcelica]

I have a dual 16 core machine (XEON 6226R), and am solving on 12 cores. When I go into Windows 10 task manager, and look at CPU Utilization, it looks like CFX is only using one of the processors. Is there any setting to change this, and wouldn't it be more efficient to use both? I don't know if this is a CFX thing or hardware thing, so forgive me for posting this in both forums.

NOTE: I am running CFX Distributed using IBMMPI to another identical machine. The other machine exhibits the same behavior as well.
I just repeated this using 18 cores per machine, and CPU usage shows 100% of the second Numa Node, and just a little bit of the first. So It prioritizes the 2nd numa node instead of distributing evenly, or cycling through all cores.

Testing IntelMPI local parallel on 12 cores: It splits it to about 8 cores on Numa Node 0 and 4 cores on Numa Node 1. Better than IBM MPI, but still not perfectly balanced.

[Opaque]

Have you checked the -numa option in cfx5solve ?

cfx5solve -numa auto -def ...

[evcelica]

Thanks for the reply Opaque
No, I have not. Where would that be done? Sorry for my ignorance.

I always just submit my job using the solver manager.

[Opaque]

You can check the help on the cfx5solve command

cfx5solve -help

or in the SM, go to the Solver Tab in the Define Run panel and set the "Solver Arguments" to -numa auto

[evcelica]

OK, I will try that....Nope, No difference.

I did a bit of reading, and I believe in my BIOS I have memory interleaving Enabled, which I believe disables NUMA? I will change that and see what happens.
Are there general recommendations for whether I should activate or disactivate interleaving (which i understand is the opposite of NUMA?)

So I will try disabling that and see what happens. Unfortunately I'm working remote, so I'll have to go in and make this change.

[evcelica]

I did a bit more testing, and the processors are not always 100% unbalanced like it looked before. It looks like it changes a little every now and then to a little more balanced, then a little less, then back @ 100% Load to only one CPU.

I checked my BIOS and it looks like I don't have the option to disable memory interleaving. Only options for interleaving are AUTO or 1-way, 2-way, etc. So perhaps I'm not understanding this or other NUMA settings on this machine, if that is such a thing. Motherboard is an ASUS C621E SAGE.
Actually I believe I may have been mistaken about "Memory interleaving" being the opposite of NUMA. I think that was actually "Node interleaving" per this source: https://frankdenneman.nl/2010/12/28/...le-or-disable/

INTEL MPI balances the load better than IBM, and it also shows better performance than IBM MPI on my machine: Actually, same performance at low core counts, but it scales better at higher core counts. So it seems more balanced CPU loading does indicate better scaling, but there might be more to it than that of course.
I'm just not sure where to attempt to make the change if in fact I can control this:
Hardware limitation of the motherboard?
BIOS?
Windows Setting?
CFX?
IBM MPI setting? I think I do remember reading something years ago about IBM-MPI and cyclic cpu loading or pinning. I'll search for that again.

I'll see if other programs (ANSYS mechanical) exhibit the same unbalanced CPU loading.
Do others see an unbalanced load with CFX on their 2 socket systems? Using INTEL MPI? Using IBM MPI?
I'd appreciate it if anyone could take a look. For Windows, open task manager, go to performance, then right click the graphs and click change to: NUMA nodes.
Thank you!
Erik

[arunantony]

Hi Evcelica,

I came across your post in a thread and am curious about whether you managed to solve the load balancing issue. I'm encountering a similar problem here with two servers, each having 72 cores. For some reason, the load distribution is not even, or there might be other underlying issues.

I would greatly appreciate it if you could share how you addressed this situation.

Kind regards,
Arun A