[tehache]

Hi all,

I am running OpenFoam on a cluster of 2Core/2Cpu AMD Opteron 275 (i.e. 4 cores per blade), which are connected by simple 1GB Ethernet.
Parallel efficiency is almost perfect if running all on 1 blade, but goes down drastically if I have processes on different blades.
Fluent, however, scales well even across blades. And yes, I have been experimenting with float transfer and scheduled transfer, made it only worse.
So I have 2 questions:

1.) Any idea what Fluent is making different?

2.) Does someboy have experience if Myrinet or Inifiband as inter-blade connection would change this?

Thanks for any comment!

regards,

Thomas

[msrinath80]

Have you tried MPI/GAMMA[1] instead of LAM/OpenMPI?

[1] http://www.disi.unige.it/project/gamma/mpigamma/

[mattijs]

We're getting very good behaviour for GAMMA or MPI/GAMMA (GAMMA with MPI compatibility layer). The latest version GAMMA is very stable.

- install GAMMA (not trivial)
- choose 'export WM_MPLIB=GAMMA' in your .OpenFOAM-xxx/bashrc and recompile Pstream:

http://www.cfd-online.com/cgi-bin/Op...1346#POST11346

[tehache]

Thank you !

Sounds good.
I am just a little scared because the gamma people specify the required kernel exactly as 2.6.18.1
We are running 2.6.16.13 (smp), and I doubt I can convince my admin to change the running kernel on the cluster.

Any experience if it works with other kernel versions too?

Thanks again!

Thomas

[mattijs]

It isn't too hard to patch a not-too-different kernel. From what I remember the only change is in the network driver (e1000.c) where you have to disable the network interface so GAMMA has exclusive rights to it. 'diff'ing the GAMMA supplied version with your kernel's e1000.c should tell you what to change.

[bastil]

Sorry for ading to this very old thread. However I have simular considerations. I use the same cpus but with a myrinet interconnect. I get nearly no speedup (less then 10%) from OpenFOAM when switching from 16 (4 blades x 2 cpus x 2 cores) to 32 (8 blades x 2 cpus x 2 cores) cores. This is totally different for FLUENT. This leads to the fact that FLUENT outperforms OpenFOAM on 32 cores by a factor of 2 whereas on 16 cpus speed is nearly the same. I have no idea what's going on there.