Running OpenFOAM in parallel with different locations for each process
Posted December 15, 2011 at 17:56 by wyldckat
Tags multiple roots, openfoam, parallel
The other day a fellow forum user asked me for help about a problem dealing with the specific configuration of a case for running in a cluster. The specific configuration was simple: each slave machine of the cluster has it's own independent storage address on a similar path.
Below is an edited version of the reply I sent him.
OK, first detail - In the following file you have a (sort-of) well documented "decomposeParDict":
The part you are looking for says this:
Now, I had a hard time understanding how "/data" is really structured on each node, namely:
Below is an edited version of the reply I sent him.
OK, first detail - In the following file you have a (sort-of) well documented "decomposeParDict":
Code:
applications/utilities/parallelProcessing/decomposePar/decomposeParDict
Code:
//// Is the case distributed //distributed yes; //// Per slave (so nProcs-1 entries) the directory above the case. //roots //( // "/tmp" // "/tmp" //);
- Is it "/data/node1"?
- Or "/data/user" in all, but different disks?
- On the workstation or master node, setup the case and decompose - basing myself on the "tutorials/multiphase/interFoam/laminar/damBreak" case:
Code:blockMesh cp 0/alpha1.org 0/alpha1 setFields decomposePar
- Edit the "system/decomposeParDict" and set the roots like this (4 processes, 3 slaves):Note that the master node (node0) is not mentioned.Code:
distributed yes; roots ( "/data/node1" "/data/node2" "/data/node3" );
- Copy/clone the whole "damBreak" folder to each node, having this structure:Therefore, each node will have a copy of the whole case.Code:
/data/node0/damBreak /data/node1/damBreak /data/node2/damBreak /data/node3/damBreak
If you wish to save space, you'll need to do some trial and error in figuring out what's necessary on each node. So for now, lets stick to what works
- Run the solver in parallel, starting from the node0 (directly or indirectly). Since I was testing multi-roots in a single machine, I simply ran on my "/data/node0/damBreak":Code:
foam -s -p interFoam
- When it is done, you'll have 4 asynchronized cases. In the master node, you'll only have the "/data/node0/damBreak/processor0" filled up. On node1: "/data/node1/damBreak/processor1". And so on. To sync them back, you can do something like this, also running from "damBreak" at node0:That should sync them back into the master node0.Code:
for a in 1 2 3; do rsync -a node$a:/data/node$a/damBreak/ ./; done
- Then run:And you're doneCode:
reconstructPar
Total Comments 2
Comments
-
Hi Bruno,
For parallel OpenFoam run with distributed storage, does the roots need to be absolute path?
If the local temporary scratch disk for each node was set to an environmental variables "$TMPDIR" (same name but different location), can it be used in the decomposeParDict as follows,
distributed yes;
roots (
"$TMPDIR"
"$TMPDIR"
"$TMPDIR"
);
I suspect OpenFoam will replace $TMPDIR for each roots with path of the master node only (as the application starts on master node). Is there a way to check if the OpenFoam is using the correct distributed storage?Posted February 1, 2015 at 17:46 by katakgoreng
-
Hi katakgoreng,
The application decomposePar is only used in the master node, therefore it will only use the local temporary folder location.Quote:Originally Posted by katakgoreng
I suspect OpenFoam will replace $TMPDIR for each roots with path of the master node only (as the application starts on master node). Is there a way to check if the OpenFoam is using the correct distributed storage?
In order to use the temporary folder for a specific node (machine), then you'll have to first ask that node (machine) what is the path it's using for a temporary folder.
Best regards,
BrunoPosted February 1, 2015 at 18:06 by wyldckat
