Iterations stop with no reason

JeffreyStein · July 22, 2013, 06:41

Good morning all,

I am currently using Fluent v14.5.0 on a Linux cluster to run small cases on about 8 threads.
My current case involves a turbulent and reactive flow which I solve in a transient mode, and with a PDF model to compute the rates of reaction that I implemented as a (big) UDF.
I spent long hours debugging the UDF itself using gdb and numerous fprintf, and in the end it was functional - at least, I was able to launch the calculation under fluent and have it run a few timesteps without any problems.

But ... not more than a few because I actually noticed that fluent stops iterating after a random number of iterations (of the order of 3000 which is nothing given I use a timestep of 10^-7s with 65 iterations per timestep and am supposed to run that simulation for about 2 ms real time), but still displays the box where it says "Calculating the solution" and does not display any error message, whether Fluent or UDF built-in.

I have looked everywhere, checked my code but I am definitely going crazy on this one !!
I would appreciate any advice, lead or help you guys could provide me !

Thx !

Jeff

vasava · July 22, 2013, 06:51

You can try two things: 1. Different MPI type (to check if there is something wrong with MPI). 2. Different domain discretization method e.g. simple, scotch.

JeffreyStein · July 22, 2013, 07:42

Hi vasava,

Thank you for your answer.
1/ Unfortunately, I do not have the possibility to change the MPI as I am working in a company.
2/ I tried changing the solver method, but again the same effect ... Fluent stops iterating after a while, even though everything seems to be working nominally.

vasava · July 22, 2013, 08:29

Does your case work fine in serial mode?

JeffreyStein · July 22, 2013, 08:38

I cannot really say. The problem is it takes hours to make one iteration if I am running on 1 processor only, hence I never reached the moment where it 'usually' stops iterating.

Because what could have been the problem according to you ?

vasava · July 22, 2013, 09:28

You could use coarser mesh once to check if your setup is fine.

It is also possible that your case is way too large for the memory available in the cluster. Sometimes issues related to communication between nodes can also cause such problems. You should consult the administrator of the cluster or someone who is expert with linux.

JeffreyStein · July 22, 2013, 09:59

Concerning memory, I must admit I never really knew how to make a good estimate of what the case would need. Any tricks or back of the envelope calculations that could be done to obtain a rough estimate ?

For this case, I based my approach for the number of threads on the number of cells : I have got about 75000 cells (yup, small thingy), so 8 threads - that is legit according to my more-experienced-colleagues.
On the other hand, for the memory, I could not do anything but guess : the PDF technique I implemented implies the transport of 8 extra scalars, and the UDF has to go through slightly complex mathematical operations. So I chose to run the case with 2 Go for each thread, so 16 Go total. Any comments on that ?

vasava · July 23, 2013, 03:42

You can try using more nodes. But I am not optimistic.

I am no expert in the cluster and linux matters so you are pretty much on your own

. As I mentioned that communication between the nodes could also be one issue.

I still suggest that you test your case with a coarse mesh first. This will ensure that the case setup is fine, UDF is ok and it might pin-point the problem area much quicker.

July 22, 2013, 06:41	Iterations stop with no reason	#1
JeffreyStein New Member Thibault Bridel Join Date: Jul 2013 Posts: 4 Rep Power: 13	Good morning all, I am currently using Fluent v14.5.0 on a Linux cluster to run small cases on about 8 threads. My current case involves a turbulent and reactive flow which I solve in a transient mode, and with a PDF model to compute the rates of reaction that I implemented as a (big) UDF. I spent long hours debugging the UDF itself using gdb and numerous fprintf, and in the end it was functional - at least, I was able to launch the calculation under fluent and have it run a few timesteps without any problems. But ... not more than a few because I actually noticed that fluent stops iterating after a random number of iterations (of the order of 3000 which is nothing given I use a timestep of 10^-7s with 65 iterations per timestep and am supposed to run that simulation for about 2 ms real time), but still displays the box where it says "Calculating the solution" and does not display any error message, whether Fluent or UDF built-in. I have looked everywhere, checked my code but I am definitely going crazy on this one !! I would appreciate any advice, lead or help you guys could provide me ! Thx ! Jeff

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Extrusion with OpenFoam problem No. Iterations 0	Lord Kelvin	OpenFOAM Running, Solving & CFD	8	March 28, 2016 12:08
Moving mesh	Niklas Wikstrom (Wikstrom)	OpenFOAM Running, Solving & CFD	122	June 15, 2014 07:20
pimpleFoam: turbulence->correct(); is not executed when using residualControl	hfs	OpenFOAM Running, Solving & CFD	3	October 29, 2013 09:35
calculation stops after few time steps	sivakumar	OpenFOAM Running, Solving & CFD	7	March 17, 2013 07:37
Orifice Plate with a fully developed flow - Problems with convergence	jonmec	OpenFOAM Running, Solving & CFD	3	July 28, 2011 06:24

July 22, 2013, 06:51		#2
vasava Senior Member Paritosh Vasava Join Date: Oct 2012 Location: Lappeenranta, Finland Posts: 732 Rep Power: 23	You can try two things: 1. Different MPI type (to check if there is something wrong with MPI). 2. Different domain discretization method e.g. simple, scotch.

July 22, 2013, 07:42		#3
JeffreyStein New Member Thibault Bridel Join Date: Jul 2013 Posts: 4 Rep Power: 13	Hi vasava, Thank you for your answer. 1/ Unfortunately, I do not have the possibility to change the MPI as I am working in a company. 2/ I tried changing the solver method, but again the same effect ... Fluent stops iterating after a while, even though everything seems to be working nominally.

July 22, 2013, 08:29		#4
vasava Senior Member Paritosh Vasava Join Date: Oct 2012 Location: Lappeenranta, Finland Posts: 732 Rep Power: 23	Does your case work fine in serial mode?

July 22, 2013, 08:38		#5
JeffreyStein New Member Thibault Bridel Join Date: Jul 2013 Posts: 4 Rep Power: 13	I cannot really say. The problem is it takes hours to make one iteration if I am running on 1 processor only, hence I never reached the moment where it 'usually' stops iterating. Because what could have been the problem according to you ?

July 22, 2013, 09:28		#6
vasava Senior Member Paritosh Vasava Join Date: Oct 2012 Location: Lappeenranta, Finland Posts: 732 Rep Power: 23	You could use coarser mesh once to check if your setup is fine. It is also possible that your case is way too large for the memory available in the cluster. Sometimes issues related to communication between nodes can also cause such problems. You should consult the administrator of the cluster or someone who is expert with linux.

July 22, 2013, 09:59		#7
JeffreyStein New Member Thibault Bridel Join Date: Jul 2013 Posts: 4 Rep Power: 13	Concerning memory, I must admit I never really knew how to make a good estimate of what the case would need. Any tricks or back of the envelope calculations that could be done to obtain a rough estimate ? For this case, I based my approach for the number of threads on the number of cells : I have got about 75000 cells (yup, small thingy), so 8 threads - that is legit according to my more-experienced-colleagues. On the other hand, for the memory, I could not do anything but guess : the PDF technique I implemented implies the transport of 8 extra scalars, and the UDF has to go through slightly complex mathematical operations. So I chose to run the case with 2 Go for each thread, so 16 Go total. Any comments on that ?

July 23, 2013, 03:42		#8
vasava Senior Member Paritosh Vasava Join Date: Oct 2012 Location: Lappeenranta, Finland Posts: 732 Rep Power: 23	You can try using more nodes. But I am not optimistic. I am no expert in the cluster and linux matters so you are pretty much on your own . As I mentioned that communication between the nodes could also be one issue. I still suggest that you test your case with a coarse mesh first. This will ensure that the case setup is fine, UDF is ok and it might pin-point the problem area much quicker.