[haiteng]

I have a workstation with two CPUs and each CPU has 18 cores and 36 threads, so the machine has a total of 36 cores and 72 threads. When I run ANSYS Fluent using 32 process on my Windows 10 64-bit system by clicking "Fluent launcher -> Processing Options -> Parallel (Local Machine) ->32 Processes", the case can be allocated to 32 threads on one CPU, as observed from Windows Task Manager. However, when I type 64 on the "Processes" tab in Fluent launcher as described before, the case is still just allocated to 32 threads on a single CPU. Could anyone tell me why does this happen and how to solve the problem?

BTW, I am totally fresh to parallel computing and still confused about the relation between "compute node" and number of CPUs. Seen from Windows Task Manager (snapshot as attached), my workstation has two so-called "Node" and each node has 36 CPUs. Thus I guess the "Node" here refers to the fact that my workstation has two distinct Intel Xeon CPUs, and "CPU" here actually refers to the fact that each Intel CPU has 36 threads? How could I confirm the number of compute node and its correspondences with CPUs on my workstation?

[BlnPhoenix]

Hi Henry,

in your screenshots it says 2 sockets and as you wrote every socket has 18 physical cores. I would recommend to switch off Virtualisation (aka virtual doubling of cores) because this shows no fundamental improvement in most CFD related situations (at least that's what i've read on this forum) and makes things a bit more confussing.
Regarding the problem with 64 Cores: Do you have a licence for 64 cores?

Usally if you have two CPUs sitting on the same workstation it should be no problem to use all cores under Fluent.

[LuckyTran]

I don't know what information is in your loaded screenshot except processor information.


You have two sockets (two Xeon Gold 6139's), each with 18 thingies that are hyperthreaded to 36. This means you should be able to run Fluent with up to 72 Fluent processes. Which thingies the Fluent processes are assigned to are handled by the job scheduler. If you have basic Windows, you have no control over this. If you are using a windows job scheduler (HPC Windows version) then it's a little different.


Any error messages and so on? You could just be out of licenses or something. I don't know. Give us some info.

[andrews]

I recommend to turn off the function of hyperthreading on your workstation BIOS. this is not effective heavy calculation like Ansys fluent.
this is can make the CPU allocation problems.

[haiteng]

Originally Posted by BlnPhoenix

Hi Henry,

in your screenshots it says 2 sockets and as you wrote every socket has 18 physical cores. I would recommend to switch of Virtualisation (aka virtual doubling of cores) because this shows no fundamental improvement in most CFD related situations (at least that's what i've read on this forum) and makes things a bit more confussing.
Regarding the problem with 64 Cores: Do you have a licence for 64 cores?

Usally if you have two CPUs sitting on the same workstation it should be no problem to use all cores under Fluent.

I need to check with IT about the license limit... But I highly suspect this is the case: we only have license for 32 processes.
Under this limitation, it seems the best option is to turn off hyperthreading in BIOS? May I ask how hyperthreading affects FLUENT performance? Is there any comparison between, say, 16 physical cores with hyperthreading on versus off for FLUENT parallel computing?

[BlnPhoenix]

Originally Posted by haiteng

I need to check with IT about the license limit... But I highly suspect this is the case: we only have license for 32 processes.
Under this limitation, it seems the best option is to turn off hyperthreading in BIOS? May I ask how hyperthreading affects FLUENT performance? Is there any comparison between, say, 16 physical cores with hyperthreading on versus off for FLUENT parallel computing?

Yes one Thread is definitly slower than one core. When you have limited licences Hyperthreading is not a good option.
There has been some discussion on here. You will find information on this subject. Regards

[LuckyTran]

Hyperthreading isn't necessarily your problem if all you have is a licensing problem preventing you from running 36 processes. But Hyperthreading is bad because:


1) licenses are extremely limited and most people are not luxurious enough to have licenses just laying around. You/someone is paying big bucks for them. They're also usually shared. You using more licenses than needed is screwing someone else over.
2) It is obsolete on newer systems, but bad job schedulers would lead to less than ideal scenarios where the physical cores would not be saturated. Instead, one logical and one virtual core thread being assigned to the same physical core, slowing down the entire simulation immensely (by pretty much 50%). If you can avoid this scenario, it's not that bad. And this was only an issue in the earliest days of multi-core. Job schedulers are smart now and can distinguish which threads are running on which cores.
3) computation speed already does not scale linearly with number of Fluent processes and increased overhead means more memory (RAM) is needed.

[haiteng]

Originally Posted by BlnPhoenix

Hi Henry,

in your screenshots it says 2 sockets and as you wrote every socket has 18 physical cores. I would recommend to switch off Virtualisation (aka virtual doubling of cores) because this shows no fundamental improvement in most CFD related situations (at least that's what i've read on this forum) and makes things a bit more confussing.
Regarding the problem with 64 Cores: Do you have a licence for 64 cores?

Usally if you have two CPUs sitting on the same workstation it should be no problem to use all cores under Fluent.

If I have two CPU sockets (each socket has 18 physical cores) and use them to run 32 processes in Fluent, how does the two CPU sockets exchange data? If data exchange through CPU sockets are through network, does that mean I have to be connected to Ethernet, or perhaps WiFi?

[haiteng]

Originally Posted by LuckyTran

Hyperthreading isn't necessarily your problem if all you have is a licensing problem preventing you from running 36 processes. But Hyperthreading is bad because:


1) licenses are extremely limited and most people are not luxurious enough to have licenses just laying around. You/someone is paying big bucks for them. They're also usually shared. You using more licenses than needed is screwing someone else over.
2) It is obsolete on newer systems, but bad job schedulers would lead to less than ideal scenarios where the physical cores would not be saturated. Instead, one logical and one virtual core thread being assigned to the same physical core, slowing down the entire simulation immensely (by pretty much 50%). If you can avoid this scenario, it's not that bad. And this was only an issue in the earliest days of multi-core. Job schedulers are smart now and can distinguish which threads are running on which cores.
3) computation speed already does not scale linearly with number of Fluent processes and increased overhead means more memory (RAM) is needed.

Thanks! I have decided to switch OFF hyperthreading for FLUENT.
May I ask in what scenarios are hyperthreading be beneficial? For scientific computing (such as running Fluent or CFD in parallel), is hyperthread bad in general?

[LuckyTran]

Hyperthreading accelerates CPU execution, it doesn't increase bandwidth.

Hyperthreading is great when the problem is scalable and you can have multiple threads that can execute independently from one another for a long time. A nice check is whether asynchronous cloud-computing is applicable. If the problem can be computed in a cloud-like way, hyperthreading tends to be great. Most computing fields fall into this category... but not CFD.

In CFD we partition different parts of the mesh onto each process, but then there is shared boundary data that must be passed between processes. And this happens predictably... every iteration needs to be synchronized. The calculation cannot continue until this update finishes. This part is bandwidth limited.

Originally Posted by haiteng

If I have two CPU sockets (each socket has 18 physical cores) and use them to run 32 processes in Fluent, how does the two CPU sockets exchange data? If data exchange through CPU sockets are through network, does that mean I have to be connected to Ethernet, or perhaps WiFi?

The sockets are connected through interconnects. In old days we called this the front-side-bus, then it evolved into something called the QPI (QuickPath interconnect). Nowadays it's become the UltraPath Interconnect (UPI). AMD will have its own names for similar technology. The interconnects are high-speed (low latency) land-bridges that are much faster than what you can achieve with Ethernet. As far as our conversation is concerned, communication between sockets isn't much different than communication between cores within each socket (because all the info is being held in RAM, hence front-side-bus).

[haiteng]

If the parallel license is limited to 32 processes, does that mean I can use a maximum of 16 physical CPU cores with hyperthreading on? Or in another word, Is the virtual CPU (turned on by hyperthreading) treated the same as one physical CPU in terms of occupying the process limit by ANSYS license?

[LuckyTran]

Originally Posted by haiteng

If the parallel license is limited to 32 processes, does that mean I can use a maximum of 16 physical CPU cores with hyperthreading on?

No. You can use all 18 physical cores if you do it right. If you have 32 parallel licenses (presumably) and 18x2=36 physical CPU cores, you don't have enough licenses to use all 36 physical cores. You should run 32 processes. Windows will prioritize these threads to the physical cores.

Originally Posted by haiteng

Or in another word, Is the virtual CPU (turned on by hyperthreading) treated the same as one physical CPU in terms of occupying the process limit by ANSYS license?

This is actually a different question but yes. Ansys doesn't distinguish whether the CPU's these run on are physical or virtual hyperthreaded ones and counts them the same. Heck Ansys doesn't even care how many number of cores you even have. Ansys launches however many threads you specify in # of processes. If you specify 1000, it will launch 100 threads (and ask for 100 licenses) even if you have a single core CPU.

[Khan_CFD]

Originally Posted by LuckyTran

I don't know what information is in your loaded screenshot except processor information.


You have two sockets (two Xeon Gold 6139's), each with 18 thingies that are hyperthreaded to 36. This means you should be able to run Fluent with up to 72 Fluent processes. Which thingies the Fluent processes are assigned to are handled by the job scheduler. If you have basic Windows, you have no control over this. If you are using a windows job scheduler (HPC Windows version) then it's a little different.


Any error messages and so on? You could just be out of licenses or something. I don't know. Give us some info.

Hello sir,
I have opened 'setup' in a Fluent file in Ansys workbench 18.1, with 20 cores (I have a workstation with 2 sockets and 16 cores in each socket). We have institute-wide license for ansys academic research. I was able to successfully run the simulation and save the solution and exit. Next time, once when I opened the same file, an error message came as, "Unable to spawn node: license not available.
ANSYS LICENSE MANAGER ERROR:Not enough Fluent - HPC licenses 20/16.
Hit return to exit." This did not happened before when I tried to open this same file.
This problem did not arise when opening those fluent files which were setup with 16 cores or less.



1. Can you please point out what the problem could be and how to solve it?
2. Is it possible to open this fluent file with just 8 cores (for example) to see results, even though it was started with 20 cores when setting the problem in it?
Thank you.

[vinerm]

A case is neither parallel nor serial. It's the Fluent session that is parallel or serial. You can open any case on any number of cores or in serial, provided the machine is capable.

Since you might not be the only user at your institute, expected number of licenses may not be available at all times.

[Khan_CFD]

Originally Posted by vinerm

A case is neither parallel nor serial. It's the Fluent session that is parallel or serial. You can open any case on any number of cores or in serial, provided the machine is capable.

Since you might not be the only user at your institute, expected number of licenses may not be available at all times.

Thanks a lot Vinerm for your answer.


1. I am able to immediately open another file which uses less than or equal to 16 cores. I am also able to open two files simultaneously that use 20 cores cumulatively. But, not able to open a single Fluent file (whether new one or already solved one) that has 20 cores.


So, does a license availability issue restricts number of cores per simulation?


2. Yes, it is possible to open 'setup' with any number of cores but then solved data in 'solution' will be lost. Is there a way to just transfer the solved data from a previous Fluent file into a new Fluent file, in ansys workbench. I am not uploading job script to the cluster but creating multiple fluent files in a workbench project, in my own workstation.


Thanks a lot for your response.

[vinerm]

I didn't realize you could be using Fluent within WB. For that, right click on Solution, select Properties, and then change number of cores. A better alternative is to open Fluent stand-alone, i.e., outside WB and then read case and data files.

Licensing does not restrict number of cores per simulation, however, each Fluent license has four inbuilt cores. So, to run two simulations with 10 core each, you need 12 hpc licenses, however, to run one simulation in 20 cores, you require 16 hpc licenses.

[Khan_CFD]

Originally Posted by vinerm

I didn't realize you could be using Fluent within WB. For that, right click on Solution, select Properties, and then change number of cores. A better alternative is to open Fluent stand-alone, i.e., outside WB and then read case and data files.

Licensing does not restrict number of cores per simulation, however, each Fluent license has four inbuilt cores. So, to run two simulations with 10 core each, you need 12 hpc licenses, however, to run one simulation in 20 cores, you require 16 hpc licenses.

Your information was very helpful.


I right clicked on solution and went to properties but the cores there were not accessible to be modified. However, I went to setup and I could change number of processors from properties of setup. The number of processors in solution then changed automatically...


Thanks Vinerm..!