[msrinath80]

Hello Foamers,

Attached are four plots. The first and second one compare different interconnects. Interestingly Gigabit fares just as well as infiniband for the case tested.







The other two shows what happens when you use dual-core based clusters. Basically highlighting the memory-bandwidth bottleneck. The good news of course is the OpenFOAM scales very nicely given a decent interconnect.

[msrinath80]

For the last two plots, I forgot to mention that IDEAL means that I use only one processor core from each dual core CPU and make sure that the other core is idle (all processes in this case are linked through infiniband). PRACTICAL means that I use both cores in every dual-core node.

[fhy]

Hi Srinath,

These are very interesting results.
What is the test case in these runs? If it is not a tutorial test case, can I get a copy of it. I would like use it to do some performance comparison of different MPIs and interconnects.

Thanks,
Huiyu

[msrinath80]

Huiyu, I will email you the test case when I get to my office tomorrow.

Alessandro, taskset is part of the Linux distribution. For example see [1].

Based on you /proc/cpuinfo output I see that you have two physical CPUs and each CPU is hyperthreaded. So when running parallel jobs you want to use one CPU with 'physical id' 0 and another with 'physical id' 3. This can be either of the combinations shown below:

1) 0,2
2) 1,2
3) 0,3
4) 1,3

All the four choices listed above are equivalent. So if you picked the first choice, the taskset command syntax would translate to:

taskset -c 0,2

Assuming that cat /proc/cpuinfo returns exactly the same output when run on all the other nodes, proceed as follows:

So in mpirun, you would supply:

mpirun -np 4 -hostfile your_hostfile taskset -c 0,2 icoFoam . case_name -parallel > case_name/log 2>&1 &

If that does not work, then run mpirun as usual without the taskset argument and after the processes have started get into each node and issue the taskset command for each of the OpenFOAM applications running. For example if icoFoam is running on node1 with PIDs 12345 and 67890, then issue:

taskset -cp 0 12345
taskset -cp 2 67890

MPI implementations are becoming smarter as we speak. So I am guessing it will not be long before they correctly recognize the nature of dual-core/hyperthreaded hardware and schedule processes accordingly. I know for a fact that the latest Fluent version (6.3) does this automatically. I am sure OpenFOAM will follow soon.

[1] http://www.cyberciti.biz/tips/setting-processor-affinity-certain-task-or-process .html

[gschaider]

I would not overestimate the cleverness of the Fluent-6.3-approach. It always assigns the first process to the first core of the first processor. 2nd process to the 1st core of the 2nd processor. (Assuming you start a two-processor run). If a 2nd run happens to be scheduled to that node it goes to the same cores (leaving 50% of the cores unused and having everything run at half the speed it could). It does not take into account that the cores are already used.

Their setup is tuned to benchmarking (where you have all the machines for yourself and are willing to manually assign processors to nodes) not for queueing systems.

I don't think that assigning processes to cores is something that OpenFOAM should do. It just introduces machine-dependencies. It should be done by the MPI-implementation and/or the scheduler.

[msrinath80]

"Their setup is tuned to benchmarking"

True.

"It should be done by the MPI-implementation and/or the scheduler."

Once again very true I think HP-MPI does this in Fluent. Not 100% sure though.

[gtg627e]

Hi Srinath,

Thank you very much for your suggestion. It works like a charm. Indeed I see all my work being done by cup's 0 and 2. The other 2 cpu's sit there idle.

At the moment, I am seeing about 2x speedup in the clocktime to go from dT(n) to dT(n+1), but GAMG seems to not converge when used in parallel.
Still sorting that out.

Thank again for your suggestions,

Alessandro

[fhy]

Hi Srinath,

I haven't got your test case. Can you send me?

Thanks,
Huiyu

[msrinath80]

Here you go. Please refine the mesh once or twice using refineMesh before running the tests.

test.tar.bz2

[fhy]

Hi Srinath,

Thanks for the test case.

I start to work on it. When I use refineMesh, I hit the following error:

huiyu@perf-hn:~/Desktop/openfoamdata> blockMesh . test

......
Writing polyMesh

huiyu@perf-hn:~/Desktop/openfoamdata> refineMesh . test
/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.4.1 |
| \ / A nd | Web: http://www.openfoam.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

Exec : refineMesh . test
Date : Nov 30 2007
Time : 08:54:26
Host : perf-hn
PID : 4325
Root : /home/huiyu/Desktop/openfoamdata
Case : test
Nprocs : 1
Create time

Create polyMesh for time = 0

Mesh edge statistics:
x aligned : number:1253340 minLen:0.0002272727272727267 maxLen:0.001254861204598251
y aligned : number:1222804 minLen:0.0002272727272727265 maxLen:0.0002678571428571461
z aligned : number:1197420 minLen:0.001465276562542167 maxLen:0.008000000000000007
other : number:0 minLen:1000000000000000 maxLen:-1000000000000000

Refining all cells

3D case; refining all directions

Writing refined cells (131583) to cellSet "0.0045/polyMesh/sets/refinedCells"

#0 Foam::error::printStack(Foam:stream&) in "/home/huiyu/work/openfoam/mpich/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/lib OpenFOAM.so"
#1 Foam::sigSegv::sigSegvHandler(int) in "/home/huiyu/work/openfoam/mpich/OpenFOAM/OpenFOAM-1.4.1/lib/linux64GccDPOpt/lib OpenFOAM.so"
#2 ?? in "/lib64/libc.so.6"
#3 main in "/home/huiyu/work/openfoam/mpich/OpenFOAM/OpenFOAM-1.4.1/applications/bin/linux6 4GccDPOpt/refineMesh"
#4 __libc_start_main in "/lib64/libc.so.6"
#5 __gxx_personality_v0 in "/home/huiyu/work/openfoam/mpich/OpenFOAM/OpenFOAM-1.4.1/applications/bin/linux6 4GccDPOpt/refineMesh"

refineMesh:4325 terminated with signal 11 at PC=2af87d1c8aa5 SP=7fff2ea9d1a8. Backtrace:
/lib64/libc.so.6(gsignal+0x35)[0x2af87d1c8aa5]
/lib64/libc.so.6[0x2af87d1c8b20]
refineMesh[0x40848a]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2af87d1b6154]
refineMesh(__gxx_personality_v0+0xb9)[0x405f39]

The version I am using is 1.4.1 built with gcc-4.1.0.

Any comments regarding the error?

Also, what does refineMesh do? Can I manually edit
test/constant/polyMesh/blockMeshDict, double mesh density in all dimensions to get the same effect as refineMesh?

Thanks,
Huiyu

[msrinath80]

refineMesh should not crash. However, for a quick solution, you can simply double the number of cells directly in blockMeshDict. It should work.