[devayabir]

Hi All,
I have managed to get passwordless ssh started on the two computers through the following steps.
1. Setup NFS sever:

1.3 Add Directory: /home/user/OpenFOAM
1.4 Add Host: enter the host name (or IP). enter
rw,sync under options.
1.5 click Finish to start NFS server.
2. Setup NFS Client
2.1 start YaST. Select Network Service. Click "NFS Client".
2.2 Click "Add".
2.3 enter NFS Server name.
2.4 enter Remote File System: /home/user/OpenFOAM.
2.5 enter Mount Point: /home/user/OpenFOAM
3. Setup passwordless SSH
3.1 Client Side (This actually would be the NFS server, because the NFS server needs to log onto the clients)
3.1.1 Type "ssh-keygen -t dsa"
3.1.2 hit return twice when prompts for passphrase
3.1.3 type "cd ~/.ssh"
3.1.4 type "ls", you should get 3 files: id_dsa id_dsa.pub known_hosts
3.1.5 scp id_dsa.pub user@NFS_client<img src="images/smil...>ublic-key.tmp
3.2 Server side (this will be NFS client):
3.2.1 type "ls"
3.2.2 type "mkdir .ssh"
3.2.3 type "chmod 700 .ssh"
3.2.4 type "cd .ssh"
3.2.5 type "cat ~/public-key.tmp >> authorized_keys"
3.2.6 type "rm ~/public-key.tmp"
Note: from the NFS sever side, type "ssh -l userID NFS_client" to see if you can log in without a password.

After this I can connect to one computer to the other through ssh without the password.

I made the hostfile machine and tried running the mpirun command but it doesnt do anything.
It doesnt give me an error also


Please advice......

[derekm]

I have just got this going today myself. 16 processes , 8 on master, 8 on slave
I did it this way:
Get a parallel processing tutorial case working on the master PC
Install openfoam on the slave,verify it works.
Get /etc/hosts files in sync on both nodes
Set up password less ssh from master PC to slave PC for the same user on both machines .
Set up the master PC to Nfs serve the directory structure above the tutorial case eg /home
Set up slave to mount the served structure e.g. As /mnt/master-home
Change the openmpi default host file on master to have an entry for master and slave.

Master cpu=8
Slave cpu=8

Now you need to change the decomposepar files and Allrun in the case to use 16 processes.
In the topmost decomposepar you need to set the the distributed flag and describe the 7 roots for each master process and 8 roots for each slave process The root is the path to a directory level just above the case directory
The processes on master will have a root that begins with at /home the processes on the slave begins with /mnt/master-home


It took a lot of time to work it out, but easy once you have it Sussed. I will give a more detailed account tomorrow.

[devayabir]

Originally Posted by derekm

In the topmost decomposepar you need to set the the distributed flag and describe the 7 roots for each master process and 8 roots for each slave process The root is the path to a directory level just above the case directory
The processes on master will have a root that begins with at /home the processes on the slave begins with /mnt/master-home

I did the rest.Could you give me an example of how to set the root files.

[derekm]

Originally Posted by devayabir

I did the rest.Could you give me an example of how to set the root files.

This example is straight from a working system

The example has two nodes and 8 cores in each. The example ismultiRegionHeater

The master node is linux-kzcl
The slave is linux-uv4

Both boxes are opensuse 12.3 with OpenFOAM 2.3

The case is at
/home/derekm/OpenFOAM/derekm-2.3.0/run/multiRegionHeaterdmm16

/home on linux-kzcl is mounted on linux-uv4 as /mnt/linux-kzcl_home

on linux-kzcl : /usr/lib64/mpi/gcc/openmpi/et/openmpi-default-hostfile contains

linux-kzcl  cpu=8 
linux-uv34  cpu=8

the file on linux-kzcl at /home/derekm/OpenFOAM/derekm-2.3.0/run/multiRegionHeaterdmm16/system/decomposeParDict contains at the end :

...
/// Is the case distributed
distributed     yes;
//// Per slave (so nProcs-1 entries) the directory above the case.
roots
(
  "/home/derekm/OpenFOAM/derekm-2.3.0/run"//1
  "/home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/home/derekm/OpenFOAM/derekm-2.3.0/run"//7
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"  //8
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"
  "/mnt/linux-kzcl_home/derekm/OpenFOAM/derekm-2.3.0/run"  //15
}

/home/derekm/OpenFOAM/derekm-2.3.0/run/multiRegionHeaterdmm16/system/decomposeParDict

home/derekm/OpenFOAM/derekm-2.3.0/run/multiRegionHeaterdmm16/system/heater/decomposeParDict

home/derekm/OpenFOAM/derekm-2.3.0/run/multiRegionHeaterdmm16/system/rightsolid/decomposeParDict

home/derekm/OpenFOAM/derekm-2.3.0/run/multiRegionHeaterdmm16/system/topAir/decomposeParDict
...

all have

...
numberOfSubdomains  16;
...

home/derekm/OpenFOAM/derekm-2.3.0/run/multiRegionHeaterdmm16/Allrun contains

...
# Run
runParallel `getApplication` 16
...

[devayabir]

Hi,

The only difference with our case is that we are trying to run on two different computers.
They are connected via a lan cable, and I can ping into the other machine and viceversa.
Also I have made a password-less ssh access. Following is my set up,

comp(1) tetrahedrix@tetrahedrix-desktop
comp(2) kj@Stokes

I have divided the case as given in the post give under the following link


My hostfile is

 tetrahedrix@tetrahedrix-desktop cpu=4
kj@Stokes cpu=4

My decomposeParDict is

"/home/tetrahedrix/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node1"
"/home/tetrahedrix/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node2"
"/home/tetrahedrix/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node3"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node4"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node5"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node6"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node7"

Whenever I'm running the command through mpirun --hostfile it does nothing, please advice.

Thanks

[derekm]

have you got working in parallel but on a single node?
do that first

[devayabir]

yes on a single I have done parallel computing.That is pretty straight forward.
My issue being that this doesnt even give me an error.It does nothing.
If in the machine files I dont specify the username it asks me for password assuming the username of the first machine.(Usernames on both computers are different)
So I think there is an issue in the path for the processors in the second machine.

Please advice

[derekm]

Originally Posted by devayabir

Hi,

The only difference with our case is that we are trying to run on two different computers.
They are connected via a lan cable, and I can ping into the other machine and viceversa.
Also I have made a password-less ssh access. Following is my set up,

comp(1) tetrahedrix@tetrahedrix-desktop
comp(2) kj@Stokes

I have divided the case as given in the post give under the following link


My hostfile is

 tetrahedrix@tetrahedrix-desktop cpu=4
kj@Stokes cpu=4

My decomposeParDict is

"/home/tetrahedrix/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node1"
"/home/tetrahedrix/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node2"
"/home/tetrahedrix/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node3"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node4"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node5"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node6"
"kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node7"

Whenever I'm running the command through mpirun --hostfile it does nothing, please advice.

Thanks

reread my example... the root is the path to directory above the case, so why all this node1 node2 for each subdomain?
that implies a separate case for each subdomain.

are tetrahedrix@tetrahedrix-desktop and kj@stokes the host names in /etc/hosts? they look like username@host


kj@Stokes:/home/kj/OpenFOAM/OpenFOAM-2.3.x/tutorials/incompressible/simpleFoam/node4 does not look like a path for the slave processes to access the master node case.

work out the path to the master node case directory from both the master and the slave. Test it by logging into both node and see if the path really works.

the root is the path MINUS the case directory name. for the first 4 it will just be the path on the master node
on the second 4 it will be the path as seen by the slave node.

you appear to be using a non default directory structure for OpenFOAM

give me the hostnames
the path on the master node to access the case directory on the masternode
the path on the slavenode to access the case directory on the masternode

[derekm]

Originally Posted by devayabir

yes on a single I have done parallel computing.That is pretty straight forward.
My issue being that this doesnt even give me an error.It does nothing.
If in the machine files I dont specify the username it asks me for password assuming the username of the first machine.(Usernames on both computers are different)
So I think there is an issue in the path for the processors in the second machine.

Please advice

use the same username on both machines
Try to get it working on the simplest possible configuration using an standard example using the standard Allrun. Then make changes.

[devayabir]

Hi,

This is quite late but thanks, for your assistance were able to run OF on two machines in parallel. We are connecting the two machines via a LAN cable. We wanted to know if this can be a limiter to speed up of the simulation. How should the two machines be connected to have maximum speed.
In our case we have a 30 Million mesh and the configuration of our machines is i5 processor quad core with 32 GB RAM. How many processors should be kept for slave and master to have a maximum speed up.

Waiting for your reply,

Thanks,

[derekm]

Originally Posted by devayabir

Hi,

This is quite late but thanks, for your assistance were able to run OF on two machines in parallel. We are connecting the two machines via a LAN cable. We wanted to know if this can be a limiter to speed up of the simulation. How should the two machines be connected to have maximum speed.
In our case we have a 30 Million mesh and the configuration of our machines is i5 processor quad core with 32 GB RAM. How many processors should be kept for slave and master to have a maximum speed up.

Waiting for your reply,

Thanks,

The limiting number of cells per core often quoted on the forum as 100 000 so you would need about 300 cores to achieve the maximum scaling
my configuration is targeting 10 million cells with 88 cores. Above 20 cores is normally infiniband not ethernet according the sources on the forum.