[arslan.ali]

Hello FOAMers,

Recently I have been working on parallelization of OpenFOAM on BeoWulf clusters. Basically I’m having a very minor problem in my cluster setup which I’m unable to debug.



I have a cluster of 2 laptops.

Ubuntu 18.04LTS (dual-boot with windows)

OpenFOAM-v1806 installed on home directory

i7 processor, quad core, 64-bit

These specifications are same on both laptops.



- LAN connection has been setup between both laptops using a switch and
cables.
- Both laptops ping(0.3ms-0.6ms), and password-less ssh is also working
perfectly.
- Both laptops have the same case folder in the home directory.
- And in both laptops we run decomposePar separately after editing its Dict
file correctly.
- /etc/hosts file has been also edited on both laptops.
- Master node has the ‘machines’ file in the home folder.


Now we run the mpirun command but it doesnt run. After typing the command:

‘mpirun -np 8 --hostfile ~/machines pisoFoam -parallel’

We press enter and the terminal freezes on the next line. (Nothing happens)


I have uploaded screenshots of all necessary information you might need.

Thanks in Advance.


https://imgur.com/a/mWe0ZSF
https://imgur.com/a/TIQBEkc
https://imgur.com/a/hXe4DLq
https://imgur.com/a/k8FlHjC

[wyldckat]

Greetings arslan.ali,


I don't know if you have solved this issue, but from what I can see, the problem is that OpenFOAM is installed in different folders.


I don't know how you are sourcing the OpenFOAM environments on each machine, but the idea is that you should let them be sourced when your "/home/*/.bashrc" file is called.


The other possibility is to use multiple launch calls, namely as explained here: https://www.open-mpi.org/faq/?category=running#mpmd-run
My guess is that in your case, you should make the launch command something like this:

mpirun --hostfile ~/machines  -np 4 $HOME/OpenFOAM/OpenFOAM-v1806/bin/foamExec pisoFoam -parallel : -np 4 /home/otherUserAtSecondMachine/OpenFOAM/OpenFOAM-v1806/bin/foamExec pisoFoam -parallel

But the simplest would be to either have the same username on both machines or to simply install OpenFOAM at "/opt".


Best regards,
Bruno

[hokhay]

It can be the communication problem. Have you make sure the passwordless ssh is working on both sides. The node also needs to be able to ssh into the master

[arslan.ali]

Thanks wyldckat and hokhay for your replies.

Soon after this post I solved this problem by changing my openFoam installation directory to /opt. Just as wyldckat mentioned. However, later we received an error on the mpi command stating that "point files in polymesh could not be found".. We solved that issue aswell by using NFS shared directory method.

At this point we were able to run OpenFoam simulations using multiple processors from multiple PCs.


However, right now the problem we're facing is that our simulation clocktimes dont make sense. Basically, using 1 PC and 8 cores we get a simulation clocktime of 271s on a vortex shedding case while on a cluster of 2 PCs, 16 cores we get a clocktime of 530s which doesnt make sense. Also we dont think that there are critical communication barriers between PCs since the ping between the PCs is 0.1-0.3ms.

Cluster specifications for both PCs are:
i7-4770
3.3GHz x 8.
8Gb RAM.
UBUNTU 14.04 LTS

Can you please guide us on what can be the issue. We're really hhoping to hear from you soon. Thanks for your time.

[wyldckat]

Quick answer: The slow down you are seeing may be because either:

1. The number of cells in each subdomain should be at least 50000 cells. Below that, you will start seeing little benefit in performance; perhaps you only have 5000 cells in each subdomain (in each "processor" folder).
2. The case might be monstrous, for example, 50 million cells total, where it takes a ton of time to transfer information between machines.
3. Or you forgot to add the argument "-parallel" to the command that calls the solver, as I indicated in a previous post.
4. Or you are saving to disk too frequently.

There is a thread that tries to explain the following: How to give enough info to get help