[Anna Tian]

I'm currently using CFX 15.0 to run my structured grids single phase CFD simulations. My workstation has 4 CPUs and 32 cores in total. I just found that if I use 32 cores for a 0.4 million grids simulation, it will be calculated even slower than the case I put 16 cores to calculate it.


I'm wondering what are the rules to select proper cores number. Is there a function to corelate the most proper cores number and the simulate grids number?

[ghorrocks]

There is no general rule. The effectiveness of parallelisation is different for every simulation. But as the partitions get smaller the effectiveness reduces and at some point more partitions will run slower.

Also keep in mind:

1) In distributed parallel, that as you get up to around 32 partitions or more that ethernet is unlikely to be adequate. You will probably need a high speed, low latency interconnect like infiniband.

2) In local parallel, many architectures have internal data bottlenecks. For instance a little while ago the front side bus which connected the CPU to the memory in a intel CPU system was a major bottleneck and prevented large numbers of cores being used effectively. So there is a maximum number of cores a single system can use effectively as well.

[Anna Tian]

Originally Posted by ghorrocks

There is no generalrule. The effectiveness of parallelisation is different for every simulation.But as the partitions get smaller the effectiveness reduces and at some pointmore partitions will run slower.

Thanks for your answer, ghorrocks. Even though there is no general rule to relate the cores number I shall use, people shall still need a rough approximation rule as a basic indication. Could you let me know your experience? For example, how many cores would you use if the simulation has 0.4 million or 1 million and the CFX license is not a constraint?

[monkey1]

According to ANSYS support there IS an "optimum" number of cores. They say 250.000 Elements / core gives the best efficiency (newest Info I got was 500.000 Elements / core is also ok). Meaning a mesh with 400.000 Elements would require 2 cores.
Using more Elements per core slows down the computational speed because every core has too much to do and using less cells slows down the speed because the inter core communication takes too much time compared to the solving speed.

[Anna Tian]

Originally Posted by monkey1

According to ANSYS support there IS an "optimum" number of cores. They say 250.000 Elements / core gives the best efficiency (newest Info I got was 500.000 Elements / core is also ok). Meaning a mesh with 400.000 Elements would require 2 cores.
Using more Elements per core slows down the computational speed because every core has too much to do and using less cells slows down the speed because the inter core communication takes too much time compared to the solving speed.

I see, from your post, that the cores number doesn't increase with elements number linearly. Then if I have simulation of 5 million elements, how many cores shall I use to run the simulation?

I think there must be someone, institute or company did some tests to give quite detailed cores number usage indications for single phase flow. Could anyone share the test results? It is quite strange that Ansys didn't have any words on this issue in its tutorial. I think Ansys should do these tests and publish the results to give users a basic indication.

[monkey1]

Ansys did these Tests, and when having an Ansys course they tell you about this.
And as I said it is Number of elemets / 250.000 = Number of cores
So for example a 5 Mio Elements simulation wouldirdeally require 5.000.000/250.000=20 Cores!
If you get a decimal value like for your 0.4 Mio Elements / 250.000= 1.6 Cores then just do a round off to the next full number, meaning 2 cores.

For me it worked quit well until now.

[Veronique Pe]

Hello,

Thanks for the answer, I had the same question.
But, the number of elements / core is independant of the simulation? single phase or multi phase? transient or steady ? turbulence model of zero equation or two equation or STT? it's just the convergence time will be impact by the simulation choice?

Thanks,

Véronique

[ghorrocks]

The thing which slow parallel simulations down is the inter-partition communications and coupling. So all the options you describe are just additional equations to solve, so the inter-partition communications will just increase at approximately the same rate as the simulation time - the result being that the speed up factor will be pretty much unchanged.

Things which really affect parallel efficiency are models which interact poorly with multiple partitions. I can only really think of three models which definitely require care on parallel:
* Monte Carlo radiation models (they result in a lot of extra inter-partition communication and can seriously degrade parallel performance.)
* Free surface models (if the free surface is nearly colinear with a partition boundary you will have convergence difficulties)
* Compressible flow with shocks (if the shock is nearly colinear with a partition boundary you will have convergence difficulties)

[Anna Tian]

Originally Posted by Veronique Pe

Hello,

Thanks for the answer, I had the same question.
But, the number of elements / core is independant of the simulation? single phase or multi phase? transient or steady ? turbulence model of zero equation or two equation or STT? it's just the convergence time will be impact by the simulation choice?

Thanks,

Véronique

Yes. It depends. It even much more significantly depends on your hardware. This is a very basic rough rule.