[gocarts]

ofgpu is a free GPL library from Symscape that provides GPU linear solvers for OpenFOAM®. The experimental library targets NVIDIA CUDA devices on Windows, Linux, and (untested) Mac OS X. It uses the Cusp library's Krylov solvers to produce equivalent GPU (CUDA-based) versions of the standard OpenFOAM linear solvers:

• PCG - Preconditioned conjugate gradient solver for symmetric matrices (e.g., p)
• PBiCG - Preconditioned biconjugate gradient solver for asymmetric matrices (e.g., Ux, k)

ofgpu also has support for the OpenFOAM preconditioners:

• no
• diagonal

For more details see "GPU Linear Solver Library for OpenFOAM".

OpenFOAM is a registered trademark of OpenCFD and is unaffiliated with Symscape.

[Bernhard]

Hi Richard,

Thanks for the nice work. I was able to compile OpenFOAM-1.7.x and use your plugin on the GPU. Your testcase also worked nicely.

Some comments that may help other people that are installing this library.
- I set gcc 4.3 as a compiler in ofgpu for C, and g++ (this is standard) for C++.
- Other people in my department are already using CUDA, but not Cusp and Thrust, I downloaded Cusp and Thrust code-packages to /ofgpu/src
- To patch, I used patch -p0 < file.patch

To compile OpenFOAM 1.7.x with the gpu library, I did the following
- chmod 770 src/gpu/Allwmake
- touch src/gpu/gpuless/Make/options (this file was missing)
- I think there is a mistake in src/gpu/cuda/Make/options, I used LIB = $(FOAM_LIBBIN)/cuda/libgpu there
- In OF/lib I made a symbolic link named libofgpu.so to the library in the ofgpu directory

You can check the activity of your graphical card with nvidia-smi -a -l

For the standard GPU in my machine with nice CPUs, this test case (pitzDaily) there was only slow down of course.

Richard, thanks for the nice work, I read on the website you're working on implementing additional preconditioners. I'm very of course very interested in those. Keep up the good work!

Best regards,
Bernhard

[pablodecastillo]

Is It necessary to compile all openFoam, or just a library??

Pablo

[Bernhard]

In principle you only have to recompile those parts of the code that are affected by the patch I suppose. Since I have been working with 1.7.1, and this library was designed for 1.7.x, I decided to take the save route and download and compile 1.7.x separate from my other installation. It would be worth trying it on your existing compiled OpenFOAM version.

[gocarts]

Thanks for the feedback Bernard.

You raised some good points:

Originally Posted by Bernhard

- chmod 770 src/gpu/Allwmake
- touch src/gpu/gpuless/Make/options (this file was missing)

I'll update the source code patch shortly to fix these 2 problems.

Originally Posted by Bernhard

- I think there is a mistake in src/gpu/cuda/Make/options, I used LIB = $(FOAM_LIBBIN)/cuda/libgpu there
- In OF/lib I made a symbolic link named libofgpu.so to the library in the ofgpu directory

I think these 2 issues are caused because you are missing one or more of the following:

• CMAKE_INSTALL_PREFIX variable as <ofgpu-root>/install - while configuring ofgpu with ccmake
• ln -s <ofgpu-root>/install/release/lib <ofgpu-root>/install/lib - after 'make install'
• export CUDA_ARCH_PATH=<ofgpu-root> - prior to building and running OpenFOAM

I've edited the build description to make this clearer at: http://www.symscape.com/gpu-openfoam

[ahmadlashgar]

Hi,

I've successfully compiled ofgpu.During compilation of OpenFOAM there is casting problem between ofgpu::index and Foam::label.
I can solve the problem by manually cast-forcing. But, is there any quick solution?

Thanks.

The related error is shown here:
PBiCGgpu.C:103: error: no matching function for call to ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, Foam::label, Foam::label, ofgpu::index*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::index*, ofgpu::index*, ofgpu::real*, ofgpu::real*, ofgpu::real*, int, ofgpu::real, ofgpu::real, ofgpu::real, ofgpu::real, Foam::label, bool&)
/home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:46: note: candidates are: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const char*, ofgpu::index, ofgpu::index, const ofgpu::index*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, const ofgpu::real*, const ofgpu::index*, const ofgpu::index*, const ofgpu::real*, ofgpu::real*, const ofgpu::real*, ofgpu::index, ofgpu::real, ofgpu::real, ofgpu::real&, ofgpu::real&, ofgpu::index&, bool&)
/home/tesla/OpenFOAM/ofgpu/src/ofgpu/sparsematrixargs.h:24: note: ofgpu::SparseMatrixArgs::SparseMatrixArgs(const ofgpu::SparseMatrixArgs&)
make: *** [Make/linux64GccDPOpt/PBiCGgpu.o] Error 1

[gocarts]

So I guess on your system (Linux?) you have Foam::label defined as a long?

That would result from defining 'FOAM_LABEL_64' for label.H

Is this the case?

Do you really need access to integers larger than 2,000,000,000? Do you want to solve for more than 2 Billion cells/faces/nodes?

So my first suggestion is that if you don't need to define FOAM_LABEL_64 then don't (you save memory too) and recompile OpenFOAM.

Second option if you do want to keep FOAM_LABEL_64 then try changing the definition of ofgpu::index to long in index.h. This will cost you scarce GPU memory if it works.

[ahmadlashgar]

Dear gocarts,
Thanks for the quick reply.

You are right, FOAM_LABEL_64 is defined automatically because the operating system is "Ubuntu-Desktop-10.10 64-bit". I've solved the problem with casting.

The compilation proceeds and fails upon Allmake in "/home/tesla/OpenFOAM/OpenFOAM-1.7.1/applications/solvers/multiphase/compressibleInterFoam" directory.

Main errors are:
compressibleInterFoam.C:43:22: error: RASModel.H: No such file or directory
compressibleInterFoam.C:44:22: error: LESModel.H: No such file or directory
followed by:
compressibleInterFoam.C:51: error: "incompressible" has not been declared
compressibleInterFoam.C:52: error: "incompressible" has not been declared

This is caused by the patch file which makes some changes and adds "#include RASModel.H" and "#include LESModel.H" lines to compressibleInterFoam.C. However, these files are not exists in compressibleInterFoam directory.

These files are in the following path:
./src/turbulenceModels/incompressible/RAS/RASModel/RASModel.
./src/turbulenceModels/compressible/RAS/RASModel/RASModel.H

Should I make a soft link? or I have missed something in earlier stages

Thanks.

System: Ubuntu-10.10, OpenFOAM-1.7.1, CUDA 3.1 and GCC 4.4

[gocarts]

Wouldn't you prefer to build the latest version of ofgpu 0.2 with OpenFOAM 2.0.x? http://www.symscape.com/gpu-0-2-openfoam

By the looks of your errors it seems that your build may have failed to create the turbulence libraries. The missing includes should be in:

./src/turbulenceModels/incompressible/lnInclude
./src/turbulenceModels/compressible/lnInclude

[ahmadlashgar]

Happy to see OFGPU 0.2 is compatible with CUDA 4.0.
I have downgraded thrust, cusp and cuda to make the required compatibility.

I will test OFGPU 0.2 soon,
Thank you

[gocarts]

Just to follow up on:

Originally Posted by ahmadlashgar

FOAM_LABEL_64 is defined automatically

I can't see where this is defined by default. It's not about whether you are compiling 64-bit, it relates to the size of integers. If you can find where it's defined I suggest removing it. Most likely it's somewhere in:

./wmake/rules/<platform>/c++

Edit: Should be FOAM_LABEL64 not FOAM_LABEL_64

[ahmadlashgar]

Actually It is not defined! Following commands show that FOAM_LABEL_64 does not exist but FOAM_LABEL64 is used in label.H and uLabel.H

root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL_64
root@gpu30:~/OpenFOAM# grep -r ./ -e FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/label.H:#if FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/lnInclude/uLabel.H:#if FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/label/label.H:#if FOAM_LABEL64
./OpenFOAM-1.7.1/src/OpenFOAM/primitives/ints/uLabel/uLabel.H:#if FOAM_LABEL64

So the label type is int not long.

[gocarts]

I think then your problems are two fold:

• You need to patch the exact version of OpenFOAM 1.7.x (not 1.7.1) specified in the instructions at: http://www.symscape.com/gpu-openfoam
• I guess you are trying to compile in double precision, whereas ofgpu assumes single precision

I haven't tried double precision, so I can't help you with that, though someone in the comments for the instructions mentioned it.

[ahmadlashgar]

OK. I think the OpenFOAM is now compiled successfully.

I'm new in OpenFOAM. How can run "pitzDailyGpu" sample?
I've extracted the archive, now there exist 3 folders:
0 constant system

Thanks.

[romant]

Hej,

I still have a problem with the compiling

I use GCC 4.4, thrust 1.4, cusp 0.2, OF 2.0.x (Oct 21st) and applied the patch. Everything else before of-gpu is compiled without errors.

After applying the patch, I can't compile OF 2.0.x anymore. I get problems.

There must be a problem with the include or something because it is missing basically all the files for include like the following

Making dependency list for source file pimpleFoam.C
could not open file fvCFD.H for source file pimpleFoam.C
Making dependency list for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file fvCFD.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file dynamicFvMesh.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file volFieldsFwd.H for source file pimpleFoam.C
could not open file surfaceFieldsFwd.H for source file pimpleFoam.C
could not open file fvMatricesFwd.H for source file pimpleFoam.C
could not open file pimpleControl.H for source file pimpleFoam.C
could not open file createPhi.H for source file pimpleFoam.C
could not open file initContinuityErrs.H for source file pimpleFoam.C
could not open file readTimeControls.H for source file pimpleFoam.C
could not open file CourantNo.H for source file pimpleFoam.C
could not open file setDeltaT.H for source file pimpleFoam.C
could not open file continuityErrs.H for source file pimpleFoam.C
could not open file volFieldsFwd.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file surfaceFieldsFwd.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file fvMatricesFwd.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file basicKinematicCollidingCloud.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file createDynamicFvMesh.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file readGravitationalAcceleration.H for source file icoUncoupledKinematicParcelDyMFoam.C
could not open file createPhi.H for source file icoUncoupledKinematicParcelDyMFoam.C

Unfortunately, this happens at all stages during the compile process.

I tried compiling it without the patch and that works flawlessly.

[gocarts]

I think your compilation failed earlier. You need to find the first error to diagnose the problem.

The problem you are seeing is a symptom (not the cause) related to missing lnInclude directories.

My guess is that you patched the wrong version of OpenFOAM 2.0.x. Did you see errors during the patch process?

You have to exactly match the git version of OpenFOAM 2.0.x with the patch:

git commit: 2a7a68737dba87d82500df67029a15aeda662c42, Date: Sept. 14 11:28:07 2011

i.e.,

git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x
cd OpenFOAM-2.0.x
git checkout 2a7a68737dba87d82500df67029a15aeda662c42

[romant]

Hej Richard,

I did as you suggested

git clone http://github.com/OpenFOAM/OpenFOAM-2.0.x
cd OpenFOAM-2.0.x
git checkout 2a7a68737dba87d82500df67029a15aeda662c42

and then

installed the patch, which goes in without errors now

then i do

chmod ug+x <OpenFOAM-root>/src/gpu/Allwmake
WM_GPU='CUDA'
CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2

where I am not sure about the last one, ofgpu-0-2 contains

CMakeCache.txt  CMakeCache.txt~  CMakeFiles  cmake_install.cmake  CMakeLists.txt  COPYING.txt  install  install_manifest.txt  Makefile  ofgpu  README.txt

so i think i have it right, the library when I compiled it, never was under a install/release/lib folder but right away under install/lib so i didn't need to create the soft-link

I then start the compilation with

./Allwmake

and get the following first error

lex.yy.c:1391: warning: ‘yyunput’ defined but not used
lex.yy.c:1432: warning: ‘input’ defined but not used
+ set -e
+ wmakePrintBuild -check
version changed from previous build
+ /bin/rm -f OpenFOAM/Make/*/global.?
+ wmakeLnInclude OpenFOAM
wmakeLnInclude: linking include files to OpenFOAM/lnInclude
+ wmakeLnInclude OSspecific/POSIX
wmakeLnInclude: linking include files to OSspecific/POSIX/lnInclude
+ OSspecific/POSIX/Allwmake
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file signals/sigSegv.C
Making dependency list for source file signals/sigFpe.C
Making dependency list for source file signals/sigInt.C
Making dependency list for source file signals/sigQuit.C
Making dependency list for source file regExp.C
Making dependency list for source file fileStat.C
Making dependency list for source file timer.C
Making dependency list for source file POSIX.C
Making dependency list for source file clockTime/clockTime.C
Making dependency list for source file cpuTime/cpuTime.C
Making dependency list for source file memInfo/memInfo.C
Making dependency list for source file printStack.C
Making dependency list for source file fileMonitor.C
POSIX.C: In function ‘bool Foam::ping(const Foam::word&, Foam::label, Foam::label)’:
POSIX.C:1074:25: warning: use of old-style cast [-Wold-style-cast]
POSIX.C:1074:25: warning: use of old-style cast [-Wold-style-cast]
+ Pstream/Allwmake
+ wmake libso dummy
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file UIPread.C
Making dependency list for source file UOPwrite.C
Making dependency list for source file UPstream.C
+ set +x
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file UIPread.C
Making dependency list for source file UOPwrite.C
could not open file omp.h for source file UIPread.C
could not open file omp.h for source file UOPwrite.C
Making dependency list for source file UPstream.C
could not open file omp.h for source file UPstream.C
could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UIPread.C
Making dependency list for source file PstreamGlobals.C
could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UOPwrite.C
could not open file omp.h for source file PstreamGlobals.C
could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file UPstream.C
could not open file openmpi/ompi/mpi/cxx/mpicxx.h for source file PstreamGlobals.C

it then has exactly the same errors as before, that it can't open the header files for the C files, because it can't find them, like beginning with the gpu part

+ [ TRUE =  ]
+ gpu/Allwmake
+ wmake libso gpuless
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file PBiCGgpu.C
Making dependency list for source file PCGgpu.C
+ wmake libso cuda
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file cudaGpu.C
could not open file ofgpu/ofgpuconfig.h for source file cudaGpu.C
Making dependency list for source file PBiCGgpu.C
could not open file ofgpu/pbicg.h for source file PBiCGgpu.C
could not open file ofgpu/sparsematrixargs.h for source file PBiCGgpu.C
Making dependency list for source file PCGgpu.C
could not open file ofgpu/pcg.h for source file PCGgpu.C
could not open file ofgpu/sparsematrixargs.h for source file PCGgpu.C
PBiCGgpu.C:23:25: fatal error: ofgpu/pbicg.h: No such file or directorycudaGpu.C:23:31: fatal error: ofgpu/ofgpuconfig.h: No such file or directory
compilation terminated.

[gocarts]

Errors I see:

• "could not open file omp.h for source file UIPread.C"
• "could not open file ofgpu/ofgpuconfig.h for source file cudaGpu.C"

So there's something wrong with your paths.

Don't bother compiling MPI (doesn't work with GPU), so set "WM_MPLIB=NONE"
Check that "/home/roman/OpenFOAM/ofgpu-0-2/ofgpu/*.h" exists

The fact that you don't have "<ofgpu-root>/install/release/lib" sounds like you didn't set "Set the CMAKE_INSTALL_PREFIX variable to <ofgpu-root>/install" or you didn't set the ofgpu build type to "release"

Check that the ofgpu build and install was successful.

[gocarts]

Another thought.

You have to supply your settings "WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2"

as args to etc/bashrc e.g.,:

export FOAM_INST_DIR="$(HOME)/projects/of/of2.0"; source $(FOAM_INST_DIR)/OpenFOAM-2.0.x/etc/bashrc WM_GPU='CUDA' CUDA_ARCH_PATH=/home/roman/OpenFOAM/ofgpu-0-2 etc...

[ahmadlashgar]

Hi all,

First I should thank gocarts for supporting OFGPU

I've succeeded to compile and run OpenFOAM-1.7.1 and OFGPU-0.1 and evaluate it. Here I have reported the performance comparison between CPU and GPU. The benchmark is pitzDailyGPU. One can run the benchmark using following commands:
blockMesh
simpleFoam

System CPU: AMD Opteron 6134
System GPU: NVIDIA Tesla C2050

For 12225 Cells:
CPU Time: 16 s
GPU Time: 45 s

For 122250 Cells:
CPU Time: 303 s
GPU Time: 105 s

For 299100 Cells:
CPU Time: 808 s
GPU Time: 186 s