[t.oliveira]

Hi,

I need to decompose a case to run it in many nodes because the mesh is too large for the memory of a single node. It is so large that running decomposePar in a single node is not possible.
How can I decompose the case in parallel, using distributed memory? Running decomposePar in parallel, if possible, would be a solution.

Best wishes,
Thomas

P.S.: A question like this was asked nine years ago. I am asking again because things may have changed since then.

[akidess]

How did you generate the mesh? Can this be less memory consuming than the decomposition?

[t.oliveira]

Dear Anton,

Indeed, your question makes sense. I have also had problems to generate the mesh using blockMesh. Since my geometry is sufficiently simple, I wrote a program which directly writes down the files points, faces, owner, neighbour and boundary without using OpenFOAM classes.

An alternative would be to write another program that write those files in processor* directories, but this would require investigating how decomposePar does so. Anyway, this alternative would not solve the problem for the cases in which I have had access to a large-memory machine when creating the mesh but that machine is not available anymore for decomposing it.

Best wishes,
Thomas

[arjun]

Originally Posted by t.oliveira

Dear Anton,

Indeed, your question makes sense. I have also had problems to generate the mesh using blockMesh. Since my geometry is sufficiently simple, I wrote a program which directly writes down the files points, faces, owner, neighbour and boundary without using OpenFOAM classes.

An alternative would be to write another program that write those files in processor* directories, but this would require investigating how decomposePar does so. Anyway, this alternative would not solve the problem for the cases in which I have had access to a large-memory machine when creating the mesh but that machine is not available anymore for decomposing it.

Best wishes,
Thomas

I actually wrote an oct tree based mesh generator that uses hard disk memory and generated 150 million cell mesh using only 2GB ram laptop. So mesh generation without partitioning is possible.

I am surprised to learn that openFOAM does not do paritioning in parallel. In FVUS I do parallel partitioning using par metis but when i was porting the solver for windows par metis no longer reliably available and thus had to write partitioner myself. Since I am no expert the partitioner is serial only. I put it there very reluctantly because that was one step that handicapped the solver for large meshes and felt very bad about it.

openfoam is out there for years so its surprising.

[akidess]

Originally Posted by arjun

openfoam is out there for years so its surprising.

Not that surprising. I guess 90% of all users either have relatively small meshes or use parallel meshing (snappyHexMesh). Did any of your FVUS customers complain about your partitioner being serial only?

[arjun]

Originally Posted by akidess

Not that surprising. I guess 90% of all users either have relatively small meshes or use parallel meshing (snappyHexMesh). Did any of your FVUS customers complain about your partitioner being serial only?

No so far no-one complained. But all the people who are doing serious work using linux version that is parallel partitioning. Most people using it for VOF and multiphase so big sizes are avoided as much as possible. (There is someone going to try on more than 50 million cell cases but that would be in april).

Even on windows the only step which is serial is the partitioning part so only that graph structure that is used for partitioning is serial (that is collected on root process). Rest of the solver is parallel. That means FVUS loads the mesh into partitions user is running solver on, then partitioning is performed (serial on windows and parallel on linux) and redistribution takes place. Solver then continues.

Because only one graph structure is serial only the size of meshes user can run still be much higher than openFOAM.

[Cyp]

one solution is to create a coarse background grid (with blockMesh), then to decompose the domain on X CPUs and refine the background grid in parallel with refineMesh. Then you can use snappyHexMesh in parallel.

[crazzy.pirate43]

Originally Posted by arjun

No so far no-one complained. But all the people who are doing serious work using linux version that is parallel partitioning. Most people using it for VOF and multiphase so big sizes are avoided as much as possible. (There is someone going to try on more than 50 million cell cases but that would be in april).

Even on windows the only step which is serial is the partitioning part so only that graph structure that is used for partitioning is serial (that is collected on root process). Rest of the solver is parallel. That means FVUS loads the mesh into partitions user is running solver on, then partitioning is performed (serial on windows and parallel on linux) and redistribution takes place. Solver then continues.

Because only one graph structure is serial only the size of meshes user can run still be much higher than openFOAM.

Hi,

As you are talking about the decomposition process I have a problem here, I'm preparing a new boundary condition but it refuse to be decomposed during the decompsePar so I have to redistribute it manually for all processors , is there any way to use specific command to distribute it directly or I have to paste it in each processor manually:

Thanks in advance