OpenFOAM, the open source CFD toolbox, is renowned for its robust parallel communication using domain decomposition. In this article we review the parallel communication in OpenFOAM and describe developments by OpenCFD that will be available in its next release.

- LAM and OpenMPI
OpenFOAM is currently shipped with one public domain MPI implementation, LAM, which has proven to give good performance and be extremely stable. The next release of OpenFOAM will additionally be shipped with another public domain MPI implementation, OpenMPI. OpenMPI is the amalgamation of three separate MPI projects including LAM. Compared to LAM, OpenMPI is more configurable, has more supported interconnects and automatic usage of all TCP networks.

- GAMMA
In addition, the next release of OpenFOAM will include a completely new Pstream implementation which uses the Genoa Active Message MAchine (GAMMA) communication library. GAMMA is a low-latency replacement for TCP/IP on gigabit and is supported for Intel platforms on modern Linux kernels (both 32 and 64 bit). It completely bypasses the Linux network stack to produce record breaking latency figures.

The current OpenFOAM release can be configured to run with the MPI compatibility layer on top of GAMMA (MPI-GAMMA), and has been extensively tested on OpenCFD’s 16 processor cluster. The lower latency affects especially those cases where the number of processors is large and the number of cells and processor faces is low. In real terms, using GAMMA instead of LAM or OpenMPI will give improved run times ranging from a few percent for large cases running on a small number of processors to a few hundred percent for small cases on a large number of processors (where the other MPI libraries can actually cause a decrease in run times compared to a non-parallel run).

For the next release of OpenFOAM, OpenCFD have been working closely with GAMMA developer Giuseppe Ciaccio to implement a direct GAMMA driver that bypasses the MPI layer. Because of the nature of the GAMMA protocol, the MPI layer causes a small overhead and bypassing it gives some speed-up especially in transmission of small messages. Taking the example of an unrealistically small testcase - 1000 cells per processor, 16 processors - we have seen a 30% improvement in run time. For more realistic test cases the benefits will be smaller. Apart from potential performance improvement the direct GAMMA driver also has dynamic-receive buffer sizing, removing the need to adapt the $MPI_BUFFER_SIZE environment variable for cases with extreme number of processor faces. We have found the direct GAMMA driver and MPI-GAMMA to be absolutely stable during running.

The improvement in communication speed of GAMMA over other MPI libraries is currently offset somewhat by: (1) problems relating to startup and shutdown of jobs; and, (2) the installation of the GAMMA library being difficult, requiring additional hardware in the form of a dedicated gigabit connection and switch and requiring patching of the linux kernel. However GAMMA is actively maintained and documentation is available for all steps of the installation.

Overall, working with GAMMA has been a very interesting experience for the OpenFOAM developers at OpenCFD. We feel there is a definite need for a public domain low-latency protocol on commodity hardware and GAMMA is by far the best candidate. It has been shown to be extremely applicable to the typical communications pattern of a domain decomposed CFD code.

We support the project and are reaping the benefit of faster communications on our own Linux cluster. We invite other OpenFOAM users running on a cluster to try GAMMA themselves and provide feedback of their experience to the GAMMA project.

OpenCFD Ltd.