Parallel vs Serial Run-Solver manager

rohit_turbo · March 4, 2016, 05:10

Dear all,
I work on a DELL Inspiron T7810 with 128GB Ram and Intel Xeon 8 core processor. I am working my project in ANSYS CFX. My model contains about 14.5 L elements.
A very basic question off course which I would like to put up here is that, what difference would take place in results if I run all my simulations in Intel mpi local parallel with about 15 partitions and If I run the same simulation in serial run.
How do I know what amount of partitions is correct enough for the run.
(For e.g. 10,20 or what is the partition number maximum that can be given to run with accurate solutions)
Although it runs faster than a serial run , but I have a small hitch whether it will give me the same results as serial, since I lack time here to recheck and re simulate it, I seek help.
I know a very basic question to be put up, but due to the time constraint If somebody could be kind enough to explain the details I would be really glad.

Thanks in advance.
-Rohit

flotus1 · March 4, 2016, 05:41

You will have to trust the software developers at Ansys that the results are the same no matter how many partitions were used. This assumption is not too far-fetched, checking this is part of very basic software v&v. Of course you could ask them for documentation.
When it comes to the maximum speedup, using more partitions than cores available is usually a bad idea. In your case, 8 or 7 partitions (in case you want to do other stuff with your workstation while it is solving) should hit the sweetspot.

rohit_turbo · March 4, 2016, 08:20

Thanks flotus1, I usually use my workstation to only simulate my runs, so simultaneously I run about 8-10 runs at a time some with even a partition of 10 runs and some with a serial run.The system rarely hangs but I have one more grudge that how should I be making the fullest potential usage of my workstation.Even when I give 10 runs at a time and simulate it the task manager alsways shows the performance usage of the system at about 70-80% and the ram usage is never beyond 35GB.Am I doing the right way to find out answers or my system can work better than this!!.Thanks again
Rohit

-Maxim- · March 4, 2016, 08:53

As far as I know, the RAM usage corresponds mostly to the size of your mesh. Also, transient simulations use more RAM in my experience.
I wouldn't worry about maxing out the RAM too much since the CPU is usually the bottleneck (running at 100%). As flotus said, I would recommend using 6-8 partitions. When you work with several partitions, they always have some overlap ('areas' that are calculated by two partitions next to each other). So more partitions actually slow your simulation down!
I would not run 10 simulations at the same time (how many licences do you have?!

). Just write a batch and then let CFX solve your cases one by one...

flotus1 · March 4, 2016, 09:01

with 10 serial runs at a time and task manager showing only 60-70% CPU usage you obviously have SMT turned on (intel refers to it as hyperthreading) which gives you 16 virtual cores.
It is usually advised to turn SMT off for CFD workstations because it has no benefit in these workloads and can even decrease overall performance.
If some of the 10 simulations you have running simultaneously are multi-threaded (I did not quite understand if this was the case here) and the overall CPU usage is 60-70% there is quite obviously a bottleneck other than CPU performance. Usually it is memory bandwidth and the communication overhead introduced by mpi.

So my advice for optimal performance would be:
Turn off SMT in the bios.
If you have 8 or more simulations you can run simultaneously, run a maximum of 8 at a same time, all of them in serial.
If you have less than 8 simulations to run simultaneously, distribute the number of partitions so it does not exceed the number of cores available. E.g.
1 simulation - 8 partitions
2 simulations 4 partitions each... you get the picture

rohit_turbo · March 4, 2016, 09:34

Thank you both of you!! That kind of relieves my pressure of rerunning and checking whether the results are correct or not!!I think I shall take into account what flotus said!!6-8 partitions per run is good I believe and it does run faster as compared to the serial one. I was thinking of increasing the partitions more than 16!! I have run with 16 partitions and it does give results in 1/4th of the time what serial gives and one more thing I always had a doubt about was why cant we run more than 10 runs simultaneously??Is it a barrier that Ansys created or my system isnt accepting??

March 4, 2016, 05:10	Parallel vs Serial Run-Solver manager	#1
rohit_turbo New Member Rohit S Iyer Join Date: Oct 2015 Posts: 16 Rep Power: 11	Dear all, I work on a DELL Inspiron T7810 with 128GB Ram and Intel Xeon 8 core processor. I am working my project in ANSYS CFX. My model contains about 14.5 L elements. A very basic question off course which I would like to put up here is that, what difference would take place in results if I run all my simulations in Intel mpi local parallel with about 15 partitions and If I run the same simulation in serial run. How do I know what amount of partitions is correct enough for the run. (For e.g. 10,20 or what is the partition number maximum that can be given to run with accurate solutions) Although it runs faster than a serial run , but I have a small hitch whether it will give me the same results as serial, since I lack time here to recheck and re simulate it, I seek help. I know a very basic question to be put up, but due to the time constraint If somebody could be kind enough to explain the details I would be really glad. Thanks in advance. -Rohit

March 4, 2016, 08:53		#4
-Maxim- Senior Member Maxim Join Date: Aug 2015 Location: Germany Posts: 413 Rep Power: 13	As far as I know, the RAM usage corresponds mostly to the size of your mesh. Also, transient simulations use more RAM in my experience. I wouldn't worry about maxing out the RAM too much since the CPU is usually the bottleneck (running at 100%). As flotus said, I would recommend using 6-8 partitions. When you work with several partitions, they always have some overlap ('areas' that are calculated by two partitions next to each other). So more partitions actually slow your simulation down! I would not run 10 simulations at the same time (how many licences do you have?!). Just write a batch and then let CFX solve your cases one by one... rohit_turbo likes this.

March 4, 2016, 09:01		#5
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,428 Rep Power: 49	with 10 serial runs at a time and task manager showing only 60-70% CPU usage you obviously have SMT turned on (intel refers to it as hyperthreading) which gives you 16 virtual cores. It is usually advised to turn SMT off for CFD workstations because it has no benefit in these workloads and can even decrease overall performance. If some of the 10 simulations you have running simultaneously are multi-threaded (I did not quite understand if this was the case here) and the overall CPU usage is 60-70% there is quite obviously a bottleneck other than CPU performance. Usually it is memory bandwidth and the communication overhead introduced by mpi. So my advice for optimal performance would be: Turn off SMT in the bios. If you have 8 or more simulations you can run simultaneously, run a maximum of 8 at a same time, all of them in serial. If you have less than 8 simulations to run simultaneously, distribute the number of partitions so it does not exceed the number of cores available. E.g. 1 simulation - 8 partitions 2 simulations 4 partitions each... you get the picture -Maxim- and rohit_turbo like this.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
problem during mpi in server: expected Scalar, found on line 0 the word 'nan'	muth	OpenFOAM Running, Solving & CFD	3	August 27, 2018 05:18
Some questions about a multi region case run in parallel	zfaraday	OpenFOAM Running, Solving & CFD	5	February 23, 2017 11:25
parallel run is slower than serial run (pimpleFoam) !!!	mechy	OpenFOAM	18	August 17, 2016 18:19
simpleFoam in parallel issue	plucas	OpenFOAM Running, Solving & CFD	3	July 17, 2013 12:30
Serial run OK parallel one fails	r2d2	OpenFOAM Running, Solving & CFD	2	November 16, 2005 13:44

March 4, 2016, 05:41		#2
flotus1 Super Moderator Alex Join Date: Jun 2012 Location: Germany Posts: 3,428 Rep Power: 49	You will have to trust the software developers at Ansys that the results are the same no matter how many partitions were used. This assumption is not too far-fetched, checking this is part of very basic software v&v. Of course you could ask them for documentation. When it comes to the maximum speedup, using more partitions than cores available is usually a bad idea. In your case, 8 or 7 partitions (in case you want to do other stuff with your workstation while it is solving) should hit the sweetspot.

March 4, 2016, 08:20		#3
rohit_turbo New Member Rohit S Iyer Join Date: Oct 2015 Posts: 16 Rep Power: 11	Thanks flotus1, I usually use my workstation to only simulate my runs, so simultaneously I run about 8-10 runs at a time some with even a partition of 10 runs and some with a serial run.The system rarely hangs but I have one more grudge that how should I be making the fullest potential usage of my workstation.Even when I give 10 runs at a time and simulate it the task manager alsways shows the performance usage of the system at about 70-80% and the ram usage is never beyond 35GB.Am I doing the right way to find out answers or my system can work better than this!!.Thanks again Rohit

March 4, 2016, 09:34		#6
rohit_turbo New Member Rohit S Iyer Join Date: Oct 2015 Posts: 16 Rep Power: 11	Thank you both of you!! That kind of relieves my pressure of rerunning and checking whether the results are correct or not!!I think I shall take into account what flotus said!!6-8 partitions per run is good I believe and it does run faster as compared to the serial one. I was thinking of increasing the partitions more than 16!! I have run with 16 partitions and it does give results in 1/4th of the time what serial gives and one more thing I always had a doubt about was why cant we run more than 10 runs simultaneously??Is it a barrier that Ansys created or my system isnt accepting??