|
[Sponsors] |
July 20, 2023, 13:16 |
Running simulations on clusters
|
#1 |
Member
Santhosh
Join Date: Nov 2021
Posts: 44
Rep Power: 4 |
Hello All,
I am encountering an issue with running interFoam solver in Compute Canada clusters. Using an adpatative mesh refinement (AMR) along with refined mesh cells near the walls. I have a situation that requires a lot of computational resources. So I am trying to parallelize as much as I can. However, as I tried to run the simulation on 64 CPUs for instance ( I always make sure, I am using one whole node). The simulation diverges as it seen in the attached picture (pressure residuals blows up). The decomposition on 4 processors on my computer works fine and it also works with a decomposition on 16 processors in Compute Canada! I don't understand why increasing the number of processors to 32, 64 make the simulation diverges. I tried to contact the support but they couldn't figure out the issue... I must mention that all simulations using uniform meshing works well on any type of decomposition (16,32,64,...). Meaning it must be linked to the refinement and AMR process but I can't figure out the issue. Has someone already encountered this type of problem ? Or have any suggestions ? Thanks for your time and help. Sincerely, Santhosh |
|
August 7, 2023, 12:28 |
|
#2 |
New Member
Join Date: Apr 2023
Posts: 3
Rep Power: 3 |
Interesting problem. I worked with OpenFOAM AMR for a little bit but am no expert.
Does your OpenFOAM AMR implementation have the ability for dynamic load balancing? What decomposition method did you use? |
|
August 7, 2023, 21:44 |
|
#3 |
Member
Santhosh
Join Date: Nov 2021
Posts: 44
Rep Power: 4 |
Hello jmt,
I didn't try the dynamic load balancing, it seems that there is possibility since some people manage to successfully implemented it on OpenFOAM. It might be the answer. I am using a simple decomposition. |
|
August 7, 2023, 21:49 |
|
#4 |
New Member
Join Date: Apr 2023
Posts: 3
Rep Power: 3 |
Cool. Can you send me your full log file (i imagine it is small since the crash happens iteration 1)? And your fvSolution file?
Thanks. |
|
August 7, 2023, 22:42 |
|
#5 |
Member
Santhosh
Join Date: Nov 2021
Posts: 44
Rep Power: 4 |
Sure! Thanks for the interest. You can find the files as attached files.
|
|
August 8, 2023, 02:26 |
|
#6 |
Senior Member
M
Join Date: Dec 2017
Posts: 694
Rep Power: 12 |
Trying with dynamic load balancing seems to be a good idea (have not tried this myself yet though). Another hint: I have seen threads in the CFX subforum here, where undesirable locations for the processor boundaries caused divergence in multiphase flows. If the decomposition interface coincides with a high volume fraction gradient, instabilities may result. Another decomposition might help to improve this? Just an idea.
|
|
August 8, 2023, 08:14 |
|
#7 | |
Member
Julian
Join Date: Sep 2019
Posts: 32
Rep Power: 7 |
Quote:
As AtoHM suggested, could you use some fraction of the node just to test? Decompose the domain for 32, 16, 8 ranks etc and run a few iterations on the cluster? Thanks for the input and log file. I'll take a look today. |
||
August 8, 2023, 12:12 |
|
#8 |
Member
Santhosh
Join Date: Nov 2021
Posts: 44
Rep Power: 4 |
Hello Ato, jmt
Thanks for the inputs! Those help me to get a clearer idea of what might be the issue. I will try to relaunch with another decomposition to see if it works. Otherwise,I'll give a shot to the dynamic load balancing. |
|
August 15, 2023, 13:50 |
|
#9 |
Member
Santhosh
Join Date: Nov 2021
Posts: 44
Rep Power: 4 |
Hello all,
Just a quick update. I switched the decomposition method from Simple to Scotch and it is working! Thanks again for the help |
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Help with Running Periodic Rotor Simulations | Kamranh01 | SU2 | 0 | March 9, 2023 14:56 |
Fluent exit frequently with error ‘Unable to parse’ running on remote clusters | Gang Shen | Fluent Multiphase | 5 | February 7, 2022 03:02 |
Something weird encountered when running OpenFOAM in parallel on multiple nodes | xpqiu | OpenFOAM Running, Solving & CFD | 2 | May 2, 2013 04:59 |
What do you CFD guys do during a long simulation running? | bearcat | Main CFD Forum | 5 | July 23, 2009 08:08 |
Kubuntu uses dash breaks All scripts in tutorials | platopus | OpenFOAM Bugs | 8 | April 15, 2008 07:52 |