Parallel run crashing

matejfor · December 4, 2008, 09:15

Dear all,
I have a problem with a simulation crashing in parallel.

solver: oodles (OF 1.5.x)
what works: simulation runs in single, runs in parallel on local quad core

what is the problem: distributed parallel using mpich (tha latest) and SGE for distribution on dual and quad core LINUX cluster crashes after several timesteps. Not dependent on saving data.

crash happens with different solvers and geometries. Other codes using the same mpich has no problem. The crash is not a problem of convergence or solver, but a mpich communication between the different nodes.

Has anyone been facing the same problem? Any extra tweaking of mpich for openfoam?

any hint is welcomed.
thanks
matej

liuch · May 10, 2009, 21:05

Dear Matej,

We're facing the same problem on our 8-core xeon linux cluster. MPICH works very well on one single CPU but fails when >1 CPU. It seems a sys. problem & we're working with the Sys Admin try to solve it. Thanks for sharing.

Best regards,
Chun-Ho

matejfor · May 11, 2009, 03:09

Hi,

how we have solved the problem was to run it on a one machine, meanwhile our admin upgraded the kernels on the nodes and the problem somehow disappeared. Now we can run distributed, but with a very small effectivity, but it seems to be a network related problem.

matej

liuch · May 11, 2009, 03:18

Hi Matej,

Thanks for your reply. Yes, I pretty sure it's system related & directly connected to the network/MPI setting. We have another set of OpenFOAM using OpenMPI on dual-processor, quad-core machine but we never have such "SAE" problem.

Our sys. adm. is currently busy with some othe stuff. We'll work together later this week.

Best regards,
Chun-Ho

December 4, 2008, 09:15	Dear all, I have a problem w	#1
matejfor Senior Member matej forman Join Date: Mar 2009 Location: Brno, Czech Republic Posts: 182 Rep Power: 17	Dear all, I have a problem with a simulation crashing in parallel. solver: oodles (OF 1.5.x) what works: simulation runs in single, runs in parallel on local quad core what is the problem: distributed parallel using mpich (tha latest) and SGE for distribution on dual and quad core LINUX cluster crashes after several timesteps. Not dependent on saving data. crash happens with different solvers and geometries. Other codes using the same mpich has no problem. The crash is not a problem of convergence or solver, but a mpich communication between the different nodes. Has anyone been facing the same problem? Any extra tweaking of mpich for openfoam? any hint is welcomed. thanks matej

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
CFX Mesh crashing	Blade	CFX	3	December 30, 2008 11:30
Solver crashing with error	kasim	CFX	6	February 11, 2008 07:17
Solver crashing with error	Usman	CFX	2	January 24, 2008 07:59
Ngeom Crashing on Trimmed / Hexahedral Mesh	Fr Ted Crilly	Siemens	2	October 13, 2005 08:08
Crashing GAMBIT with large .dbs filesize	OPK	FLUENT	3	April 15, 2005 09:35

May 10, 2009, 21:05		#2
liuch New Member Chun-Ho Liu Join Date: May 2009 Location: Hong Kong Posts: 2 Rep Power: 0	Dear Matej, We're facing the same problem on our 8-core xeon linux cluster. MPICH works very well on one single CPU but fails when >1 CPU. It seems a sys. problem & we're working with the Sys Admin try to solve it. Thanks for sharing. Best regards, Chun-Ho

May 11, 2009, 03:09		#3
matejfor Senior Member matej forman Join Date: Mar 2009 Location: Brno, Czech Republic Posts: 182 Rep Power: 17	Hi, how we have solved the problem was to run it on a one machine, meanwhile our admin upgraded the kernels on the nodes and the problem somehow disappeared. Now we can run distributed, but with a very small effectivity, but it seems to be a network related problem. matej

May 11, 2009, 03:18		#4
liuch New Member Chun-Ho Liu Join Date: May 2009 Location: Hong Kong Posts: 2 Rep Power: 0	Hi Matej, Thanks for your reply. Yes, I pretty sure it's system related & directly connected to the network/MPI setting. We have another set of OpenFOAM using OpenMPI on dual-processor, quad-core machine but we never have such "SAE" problem. Our sys. adm. is currently busy with some othe stuff. We'll work together later this week. Best regards, Chun-Ho