[kkpal]

I also had the same error after the upgrade of our HPC. Before that the same case ran well. I think it might be a problem of system setting.

[fedvasu]

Originally Posted by smraniaki

I've had the same problem. This is one of those problem drive me crazy! I could unreasonably solve the problem by modifying fvSolution dictionary (changing a smoother). The other time I could get rid of it by changing my decomposition scheme. I still don't know how and why it happens but apparently it comes from ghost cells that are not identifiable by MPI.

Goodluck
Smran

Thank you that was very useful. I know this is a 4 year old post.

But any more insight would be extremely valuable. I have a custom PISO solver with GAMG as matrix solver and with CyclicBC.

Right now I use a GaussSeidel smoother, what smoother did you use to mitigate this problem?

Regards,

Originally Posted by kkpal

I also had the same error after the upgrade of our HPC. Before that the same case ran well. I think it might be a problem of system setting.

Did you mitigate it?

you have cyclic BCs?

[kkpal]

Yes. The HPC upgraded the openmpi version. I changed it back to the original version and the problem is solved. I do have cyclic BCs.

Originally Posted by fedvasu

Did you mitigate it?

you have cyclic BCs?

[fedvasu]

Originally Posted by kkpal

Yes. The HPC upgraded the openmpi version. I changed it back to the original version and the problem is solved. I do have cyclic BCs.

Thanks kkpal,

But may I know more specific details?

which version of OpenFOAM? did you/admin compile OF with system MPI or OPENMPI from ThirdParty?

which version of OPENMPI gave you this trouble and which(original) version solved it.

[uckmhnds]

That issue still exist with OF 6.0 but you may not have any problem with newer release OF 6.2xx. The problem is basically about the fact that OpenMPI divides CFD case into the processors to solve it simultaneously with different processors and fork() environment might do the same job as an sub-program (let's say) such that it blocks OpenMPI to communicate between processors. It is a bug about OpenMPI and OpenFOAM enviroments

[fedvasu]

Originally Posted by uckmhnds

That issue still exist with OF 6.0 but you may not have any problem with newer release OF 6.2xx. The problem is basically about the fact that OpenMPI divides CFD case into the processors to solve it simultaneously with different processors and fork() environment might do the same job as an sub-program (let's say) such that it blocks OpenMPI to communicate between processors. It is a bug about OpenMPI and OpenFOAM enviroments

thank youfor letting us know.

is there a specific bug report or reports of this to OF devs?

it is very difficult for me to reproduce, sometimes the simulation crashes/sometimes doesn't if i change my pressure solver from GAMG to PCG it helps and in some other cases it doesn't

[uckmhnds]

Originally Posted by fedvasu

thank youfor letting us know.

is there a specific bug report or reports of this to OF devs?

it is very difficult for me to reproduce, sometimes the simulation crashes/sometimes doesn't if i change my pressure solver from GAMG to PCG it helps and in some other cases it doesn't

Actually, i did not report a bug to OF but my institution to upgrade software in clusters. They installed the release OpenFOAM/OpenFoam.org-6.20180918.git, which is still a 6.x version so compatible with older 6.0 version. I would recommend you to do so and you would not need to change anything in your case folders as you had to do between different versions (i.e 2.x to 3.x).

[fedvasu]

Originally Posted by uckmhnds

Actually, i did not report a bug to OF but my institution to upgrade software in clusters. They installed the release OpenFOAM/OpenFoam.org-6.20180918.git, which is still a 6.x version so compatible with older 6.0 version. I would recommend you to do so and you would not need to change anything in your case folders as you had to do between different versions (i.e 2.x to 3.x).

Yeah Thanks, I now need to run my bigger cases, these segfault crashes are random and no particular remedy or workaround solves it.

What software did your institute upgrade? apart from you installing OF-dev?

[uckmhnds]

Originally Posted by fedvasu

Yeah Thanks, I now need to run my bigger cases, these segfault crashes are random and no particular remedy or workaround solves it.

What software did your institute upgrade? apart from you installing OF-dev?

They have just installed "OpenFOAM/OpenFoam.org-6.20180918.git". That solved the problem.

[uckmhnds]

If you still have the same problem with the new release. Please, post here the given error such that we might find another solution.

[fedvasu]

Originally Posted by uckmhnds

If you still have the same problem with the new release. Please, post here the given error such that we might find another solution.

Unfortunately, I have to install OF myself, I was unable to use system MPI and system Intel compiler. I had used GCC and ThirdParty OpenMPI.

I will try to install OF-dev with Intel compiler and MPI(system).

If I have same problem [random crashes of my cases], I will post them.

It might take a few days because sometimes the cases crash quickly and sometimes they don't.

I am going to make a detailed report of why this problem persists and ask my cluster admin to install it on my behalf.

[fedvasu]

Originally Posted by uckmhnds

If you still have the same problem with the new release. Please, post here the given error such that we might find another solution.

And could you please tell me what is your cluster environment,

specifically OS,
Compiler used to compile latest OF,
MPI used (whether it was system or ThirdParty)


This would be very helpful for me.

[olesen]

Originally Posted by fedvasu

And could you please tell me what is your cluster environment,

specifically OS,
Compiler used to compile latest OF,
MPI used (whether it was system or ThirdParty)


This would be very helpful for me.

I don't know if this is related but there was an issue with openmpi + fork on infiniband which was fixed around v1706 (or earlier/later).

If you cannot or do not wish to change OpenFOAM versions, often times you can avoid problems by making certain that all #calc and #code statements in the dictionaries are evaluated prior to sending things off in parallel. For example, by running 1 iteration in serial just to ensure that all dynamic code has been compiled.

[fedvasu]

The problem arose due to, compiling openfoam with ThirdParty OpenMPI, but compiling with IntelMPI and Intel compiler (System) seems to have alleviated the problem.

Previously I was unable to install using Intel C++ and MPi because I wasn't including the IMPI headers correctly.

@olesen I don't have any cases with calc or code in my dictionaries.

[abdullahbolek]

I encountered also with the same problem when I try to run my case in HPC, which has OpenFOAM 5.0 installed. I am running the same case in my desktop without any problem with OpenFOAM 6.0.

[fedvasu]

Originally Posted by abdullahbolek

I encountered also with the same problem when I try to run my case in HPC, which has OpenFOAM 5.0 installed. I am running the same case in my desktop without any problem with OpenFOAM 6.0.

try to reinstall OF-6 with intel-mpi with icc or opnmpi with gcc on your cluster, I think that is the only way you can solve this.

[Kraneberger]

Hello everyone,


recently I came across the discussed error. I am running pimpleFOAM with kOmegaSST RAS model in OFv1912. I was able to fix it by changing my initial conditions for k and omega.



Previous starting values at inlet and outlet that produced the error were:


k = 1e-10;
omega = 0;


The error disappeared with:


k= 0.07;
omega = 0.97;


Inlet is "inletOutlet" boundary condition for both and outlet "zeroGradient". The k and omega values were calculated for a turbulence intensity Tu = 0 and Tu = 0.1. The case also has some cyclicAMI boundary conditions.



Maybe this is helpful for someone.




Greetings,
Kraneberger