Cluster setup for FSI

Lance · October 27, 2009, 12:06

Hi,
Im trying to get our linux cluster to understand that it needs to run the Ansys mechanical part of my 2-way FSI on eight cores, and not just one core as it does now.

My start script looks like:

#SBATCH -N5 -n40

deffile=deffile.def
inpfile=inpfile.dat
inifile=restartfile.res
name=my_name

cfx5solve -def $deffile -ansys-input $inpfile -ansys-arguments "-np 8" -name $name -initial-file $inifile -start-method "HP MPI Distributed Parallel" -par-dist $(hostlist -e -s, -a'*8' $SLURM_NODELIST)

which makes the fluid solver run on 5 nodes with 40 cores, and the "-np 8" makes the solid solver to run on 1 node with 8 cores with shared memory (I assume??). But the solid solver's performance is really slow and Im suspecting that it only runs on one core.

Im running the simulation sequentially, starting with solid solver. Fluid model is pure hex mesh with ~3M cells, solid mesh is pure hex with ~0.4M cells and the memory of each node is 32 GB so I guess that that shouldnt be a problem. There are no large displacements in the beginning of the simulation and so far I've only managed to do 2 time steps, which took 15 WC-hours...

1) Is there a way to see if the solid solver only runs on one core? There's nothing in the ANSYS.stdout file that indicates that, except the poor/slow performance.

2) Could someone give an example of their own 2-way fsi start script?

Any suggestion/input would be very helpful!

regards
Lance

stumpy · October 27, 2009, 14:48

Near the top of the ANSYS.stdout file it would indicate it was using 8 processors. If you don't see anything, then it's likely running seriel (you can run "top" to see how many processors are in use when ANSYS is solving). Are you using v11 or v12? In v12 "-np 8" should work OK. In v11 this doesn't work (bug). You need to use
' -np 8'
exactly as shown. Note the space after the first single quote.

Lance · October 28, 2009, 03:33

Im using V12 and in the top of ANSYS.stdout is says:
8 PARALLEL CPUS REQUESTED.
Thats all the info I get about wether it runs serial or parallel. I guess it could be that the solver is in parallel mode but there's something else thats wrong...? I have HPC license btw.

There's an example in xmgr.pdf that shows how to start a multi-field run and only launching CFX and telling it to communicate with the already running mechanical solver. Is there anyone that uses that start method? How do I get the mechanical solver to start and wait for input from CFX?

/L

Lance · October 29, 2009, 10:30

Ok, a small update:
I started Ansys alone on one node with

cfx5solve -def $deffile -ansys-input $inpfile -mfx-run-mode "Start ANSYS only" -ansys-arguments "-np 8"

and then started CFX on another node and connected it to Ansys with

cfx5solve -def $deffile -initial-file $inifile -mfx-run-mode "Start CFX only" -start-method "HP MPI Distributed Parallel" -par-dist $(hostlist -e -s, -a'*8' $SLURM_NODELIST) -cplg-host #####@n### -cplg-slave CFX

which almost worked out fine. The simulation starts, and CFX runs on 8 cores with 100 % CPU load each, but Ansys still runs on one node and with 800% CPU load!

So I guess that Ansys makes 8 partitions on one core instead of 8 partitions on 8 cores... and I have no idea on how to fix this.

Any suggestions?

Lance

stumpy · October 29, 2009, 14:07

Ansys can only run in local parallel for FSI. So your requesting, and getting, 8 partitions. Therefore you'll need 8 cores on that machine. I think I've seen the 800% thing before - this might just be a quirk of how the parallel process is reported. If it's running faster than in serial then everything might be working OK.

Lance · October 30, 2009, 03:13

I do have 8 cores on that machine, it's two quad-cores connected on one compute node. That should work for shared memory I guess?

The thing is that the parallel simulation is running as fast as the serial (or as slow I should say), so Im still thinking that the 800% CPU load is the error here...

stumpy · October 30, 2009, 10:15

Yes, that sounds fine for shared memory. From what's you've said I suspect that ANSYS is indeed running on 8 cores. The message in the stdout file and the 800% CPU load is consistent with what I've seen, except that I get some speed up. Double-check that no other processes are running on those nodes too. So I think the question is why aren't you getting any speed up. I would run in serial, then on 2 partitions and compare. Perhaps performance is dropping off with 8 partitions due to overhead? Make sure you are only comparing the time ANSYS is taking. You could also try a structual-only run to see if this gets speed up. If not, then I'm no structural expect, so I would contact ANSYS support.

anand32 · June 28, 2018, 17:35

Quote:

Originally Posted by stumpy

Ansys can only run in local parallel for FSI. So your requesting, and getting, 8 partitions. Therefore you'll need 8 cores on that machine. I think I've seen the 800% thing before - this might just be a quirk of how the parallel process is reported. If it's running faster than in serial then everything might be working OK.

IS what you said true? "Ansys can only run in local parallel in FSI".
Could you cite a reference for this information?
It seems that I have been trying to run ANSYS FSI on distributed parallel on cluster but have not made any headway. If what you said is indeed correct then perhaps I should try other options, other than distributed parallel on cluster, I mean.
Please verify.
Thanks
Anand
Learn ICEM -CFD from scratch. Click on:
https://www.youtube.com/channel/UCdP...cBI4D2-ErOUuxQ

October 27, 2009, 12:06	Cluster setup for FSI	#1
Lance Senior Member Lance Join Date: Mar 2009 Posts: 669 Rep Power: 22	Hi, Im trying to get our linux cluster to understand that it needs to run the Ansys mechanical part of my 2-way FSI on eight cores, and not just one core as it does now. My start script looks like: #SBATCH -N5 -n40 deffile=deffile.def inpfile=inpfile.dat inifile=restartfile.res name=my_name cfx5solve -def $deffile -ansys-input $inpfile -ansys-arguments "-np 8" -name $name -initial-file $inifile -start-method "HP MPI Distributed Parallel" -par-dist $(hostlist -e -s, -a'*8' $SLURM_NODELIST) which makes the fluid solver run on 5 nodes with 40 cores, and the "-np 8" makes the solid solver to run on 1 node with 8 cores with shared memory (I assume??). But the solid solver's performance is really slow and Im suspecting that it only runs on one core. Im running the simulation sequentially, starting with solid solver. Fluid model is pure hex mesh with ~3M cells, solid mesh is pure hex with ~0.4M cells and the memory of each node is 32 GB so I guess that that shouldnt be a problem. There are no large displacements in the beginning of the simulation and so far I've only managed to do 2 time steps, which took 15 WC-hours... 1) Is there a way to see if the solid solver only runs on one core? There's nothing in the ANSYS.stdout file that indicates that, except the poor/slow performance. 2) Could someone give an example of their own 2-way fsi start script? Any suggestion/input would be very helpful! regards Lance

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
setup windows cluster	radiatn	STAR-CD	2	March 30, 2009 14:52
setup vof test problem	Fang Jin	FLUENT	1	June 14, 2005 09:27
Linux Cluster Performance with a bi-processor PC	M.	FLUENT	1	April 22, 2005 10:25
Linux Cluster Setup Problems	Bob	CFX	1	October 3, 2002 19:08
To Jonas: Linux Cluster	Mark Render	Main CFD Forum	1	June 20, 2002 05:19

October 27, 2009, 14:48		#2
stumpy Senior Member Join Date: Apr 2009 Posts: 531 Rep Power: 21	Near the top of the ANSYS.stdout file it would indicate it was using 8 processors. If you don't see anything, then it's likely running seriel (you can run "top" to see how many processors are in use when ANSYS is solving). Are you using v11 or v12? In v12 "-np 8" should work OK. In v11 this doesn't work (bug). You need to use ' -np 8' exactly as shown. Note the space after the first single quote.

October 28, 2009, 03:33		#3
Lance Senior Member Lance Join Date: Mar 2009 Posts: 669 Rep Power: 22	Im using V12 and in the top of ANSYS.stdout is says: 8 PARALLEL CPUS REQUESTED. Thats all the info I get about wether it runs serial or parallel. I guess it could be that the solver is in parallel mode but there's something else thats wrong...? I have HPC license btw. There's an example in xmgr.pdf that shows how to start a multi-field run and only launching CFX and telling it to communicate with the already running mechanical solver. Is there anyone that uses that start method? How do I get the mechanical solver to start and wait for input from CFX? /L

October 29, 2009, 10:30		#4
Lance Senior Member Lance Join Date: Mar 2009 Posts: 669 Rep Power: 22	Ok, a small update: I started Ansys alone on one node with cfx5solve -def $deffile -ansys-input $inpfile -mfx-run-mode "Start ANSYS only" -ansys-arguments "-np 8" and then started CFX on another node and connected it to Ansys with cfx5solve -def $deffile -initial-file $inifile -mfx-run-mode "Start CFX only" -start-method "HP MPI Distributed Parallel" -par-dist $(hostlist -e -s, -a'*8' $SLURM_NODELIST) -cplg-host #####@n### -cplg-slave CFX which almost worked out fine. The simulation starts, and CFX runs on 8 cores with 100 % CPU load each, but Ansys still runs on one node and with 800% CPU load! So I guess that Ansys makes 8 partitions on one core instead of 8 partitions on 8 cores... and I have no idea on how to fix this. Any suggestions? Lance

October 29, 2009, 14:07		#5
stumpy Senior Member Join Date: Apr 2009 Posts: 531 Rep Power: 21	Ansys can only run in local parallel for FSI. So your requesting, and getting, 8 partitions. Therefore you'll need 8 cores on that machine. I think I've seen the 800% thing before - this might just be a quirk of how the parallel process is reported. If it's running faster than in serial then everything might be working OK.

October 30, 2009, 03:13		#6
Lance Senior Member Lance Join Date: Mar 2009 Posts: 669 Rep Power: 22	I do have 8 cores on that machine, it's two quad-cores connected on one compute node. That should work for shared memory I guess? The thing is that the parallel simulation is running as fast as the serial (or as slow I should say), so Im still thinking that the 800% CPU load is the error here...

October 30, 2009, 10:15		#7
stumpy Senior Member Join Date: Apr 2009 Posts: 531 Rep Power: 21	Yes, that sounds fine for shared memory. From what's you've said I suspect that ANSYS is indeed running on 8 cores. The message in the stdout file and the 800% CPU load is consistent with what I've seen, except that I get some speed up. Double-check that no other processes are running on those nodes too. So I think the question is why aren't you getting any speed up. I would run in serial, then on 2 partitions and compare. Perhaps performance is dropping off with 8 partitions due to overhead? Make sure you are only comparing the time ANSYS is taking. You could also try a structual-only run to see if this gets speed up. If not, then I'm no structural expect, so I would contact ANSYS support.