[newoscar]

Originally Posted by newoscar

After weeks of hard work simFlow team made impossible become possible - OpenFOAM solvers are now able to run on CUDA! And that’s just for starters.

All the computation is done entirely on GPU. No need to copy data during calculations between CPU and GPU. All of this for getting the job done faster.

Features:

• most incompressible and compressible solvers on static mesh are available
• all the calculations are done on the GPU
• no overhead for GPU-CPU memory copy
• can run in parallel on multiple GPUs

RapidCFD project page

______________

Moderator note: This thread has been created for discussing how to install and use RapidCFD. For announcements, subscribe to the announcement thread: http://www.cfd-online.com/Forums/ope...nning-gpu.html

[lordvon]

Are there any installation instructions?

I am currently attempting installation by:
-Logging in with 'sudo bash'
-Placing 'RapidCFD-dev' in the '/opt/' folder
-'source etc/bashrc' in the RapidCFD-dev folder
-'./Allwmake' in the RapidCFD-dev folder

I get a lot of the following errors (separated by line breaks):

UOPwrite.C:29:17: fatal error: mpi.h: No such file or directory (mpirun works on my computer)

warning: call through incomplete class "Foam::tmp<Foam::Field<Foam::scalar>>" will always produce an error when instantiated

/usr/local/cuda-7.0/include/math_functions.hpp:3191:22: error: call of overloaded ‘sqrt(double&)’ is ambiguous

/usr/bin/ld: cannot find -ltriSurface
/usr/bin/ld: cannot find -lsurfMesh
collect2: error: ld returned 1 exit status
make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libmeshTools.so] Error 1

/usr/bin/ld: cannot find -ltriSurface
/usr/bin/ld: cannot find -lmeshTools
/usr/bin/ld: cannot find -lsurfMesh
collect2: error: ld returned 1 exit status
make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libedgeMesh.so] Error 1

[newoscar]

There are no installation instructions yet, but with growing community we are planning to publish it.
The good news is that RapidCFD builds just like a regular OpenFOAM. If you are not familiar with this process follow this steps:
- set value of 'foamInstall' in 'etc/bashrc' file. This variable should point to the location of 'RapidCFD-dev' directory. By default this variable is set to $HOME/RapidCFD .
- source 'etc/bashrc' file
- run the 'Allwmake' file

To build in parallel set WM_NCOMPPROCS environment variable to the number of processes you want to use. This is recomended since compilation with nvcc is very slow.

Post a reply with your results

[aee]

I'm not completely clear: does this software use the same environment variables as OF? If so, I risk messing up my OF 3 installation, right?

[wyldckat]

Originally Posted by aee

I'm not completely clear: does this software use the same environment variables as OF? If so, I risk messing up my OF 3 installation, right?

Quick answer: You can have more than one OpenFOAM version installed. Check the instructions given here: http://openfoamwiki.net/index.php/In...with_the_Shell

[mesh-monkey]

Hi all,

I, like lordvon, am having troubles installing/compiling RapidCFD on my machine and am receiving similar errors.
I'm hoping someone here would be able to help me out as I've scoured as much of the web as I could to find answers but to no avail.

After following newoscar's instructions, as well as those on the github wiki and openfoam-wiki (thanks wyldckat!) as best possible, I seem to be receiving two types of issues, and many of each:

1. cuda math_functions, e.g.:
   

   /usr/local/cuda-6.5/targets/x86_64-linux/include/math_functions.h:14276:22: error: call of overloaded ‘sqrt(double&)’ is ambiguous
      return 1.0 / sqrt(a);
                         ^

2. cannot find -l* , e.g.:
   

   + wmake libso triSurface
   /usr/bin/ld: cannot find -lsurfMesh
   collect2: error: ld returned 1 exit status
   /home/tom/OpenFOAM/RapidCFD-dev/wmake/Makefile:172: recipe for target '/home/tom/OpenFOAM/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libtriSurface.so' failed
   make: *** [/home/tom/OpenFOAM/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libtriSurface.so] Error 1

The log file of ./Allwmake is attached.

Some additional information about my setup that might be helpful:

• Ubuntu 15.04 Desktop x64;
• gcc (Ubuntu 4.8.4-1ubuntu15) 4.8.4;
• CUDA version 6.5 & Quadro 6000 - confirmed working (CUDA samples run fine);
• OpenMPI 1.8.4;
• I've also tried on Ubuntu 14.04 Server and received identical errors.

Any help to figure out what's wrong would be greatly appreciated!

Cheers, Tom

[aee]

I have exactly the same problem compiling RapidCFD - both the Cuda math issue and the compiler looking for the -l* entries in usr/bin/ld (which does not exist). I haven't figured out where (or even if) that path is set in the bashrc or .sh files. Why would it look in usr/bin if the WM and FOAM env variables have been set?

I tried installing in opt/ and in $HOME. I tried setting the MPI to SYSTEMOPENMPI from OPENMPI (in line with my OpenFoam installations). I tried setting all the env variables directly. Now I am going to try giving up until someone else figures this out.

[mesh-monkey]

Hello again,

I think I rectified the issue, courtesy of aokomoriuta (via google translate) that led me to some old install instructions on the github wiki.

"However in CUDA headers there is a name clash between Foam and std math functions. Temporary remedy is to modify CUDA math_functions.h header to explicitly call std math functions. Modifications for CUDA 6.5 are available here.

After applying their patch from the link above (attached), the program compiled without a hitch. I will come back with solution results hopefully soon!

I definitely downloaded the latest code as .zip, so not sure if/why any code changes didn't come through or work, but I will log a bug on the github.

For those looking at this in the future (and previous me), the math_functions.h issues were what was causing the cannot find -l* errors. Essentially the error happened during the build of one of the first libso libraries (fileFormats) and once that failed, all of the others after that also failed - hence why at the end of the file there is a long list.

Cheers, Tom

[aee]

I have Cuda 7.5 installed. I suspect the patch might work with Cuda 7.5, but I am not going to be the first to try. Would it make sense to install Cuda 6.5 into the Thirdparty-dev folder, patch and compile it there, and then set the Cuda env variables to that?

[mesh-monkey]

Hi aee,

I don't see any issue doing as you described to install cuda 6.5. You could also use the default install directory (/usr/local) and change the sybomolic link (/usr/local/cuda) to point to cuda-6.5 rather than cuda-7.5.
As a note, I did find 6.5 to be a bit fussy with what gcc/g++ compiler version I used - had to be 4.7.

If you do feel like trying cuda 7.5, you could always just make a backup of the math_function file and try that way. FYI I think the patch would be applied to the math_functions.hpp file.

Another thing to note is what cuda architecture you build rapidcfd for - you will have to set this according to what is compatible with your gpu. By default it is set to sm_30 but I had to change this to sm_20 to make it work with the Q6000.

Tom

[aee]

I decided to try to use Cuda 7.5. The old math_functions.patch file for cuda 6.5 won't work, because the math functions targeted were moved to math_functions.hpp from Cuda 7 on, and the locations (line numbers) in the file are different. I edited the cuda 7.5 math_functions.hpp by hand based on a line-by-line read of the old patch file and then used #diff -c to build the attached patch file.
**** update Jan 14, 2016 ***
updated patch file with corrected version (was missing a "::erfc" that is in the old patch)
***** end update ***
The equivalent cuda 6.5 patch was applied to math_function.h starting at line 14273. In Cuda 7.5, the applicable section is found in math_functions.hpp starting at line 3377 (i.e., in the "all platforms" section).

Usage:
math_functions.hpp is found in /usr/local/cuda-7.5/targets/x86_64-linux/include/ (default install) or wherever you installed Cuda 7.5.

If in the default location, you will need root or sudo access for the following steps:
1) Place the patch file in the same directory.
2) Copy the original math_functions.hpp to math_functions.hpp.org and/or move it someplace safe.
2) execute
patch < Cuda7.5_math_functions.hpp.patch

Remaining issue:
Even though my Cuda 7.5 installation is native from the Ubuntu software center, and my environment variables are set according to the instructions, deviceQuery doesn't detect my Titan Black card.

The card appears properly set up...

lspci | grep -i nvidia

gives

02:00.0 VGA compatible controller: NVIDIA Corporation GK110B [GeForce GTX Titan Black] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)

Perhaps there was a conflict between existing drivers and Cuda that I didn't see.

[aee]

The issue with deviceQuery not seeing the card cleared up after a reboot. Of all things...

The compiler is still looking in usr/bin. I have to fix the openmpi-1.10.0 configuration issue (I am using Thirdparty-dev from the current openFoam). Maybe that will fix it.

[aee]

I was finally able to compile rapidCFD with the newly patched cuda7.5 math_function.hpp -- after cleaning out everyting in RapidCFD-dev and cloning the git, sourcing etc/bashrc, exporting WM_NCOMPPROCS=12 and running ./Allwmake. A detailed readme.txt is attached.

I can run small and large interFoam cases, after adjusting the case files for unsupported models, solvers and preconditoners.
e.g., for on the OF2.3.1 tutorial LES/nozzleFlow2D

Unknown LESdelta type smooth

Valid LESdelta types are :
1(vanDriest) and
4
(
IDDESDelta
Prandtl
cubeRootVol
maxDeltaxyz
)


From function LESdelta::New(const fvMesh&, const dictionary&)
in file LESdelta/LESdelta.C at line 121.

FOAM exiting

Even for this small case, one CPU core is maxed out, while nvidia-smi shows interfoam is using very little of the GPU. I may have to play with the case files to see what actually runs on the gpu. I imagine it doesn't help that I am using the GPU for display as well.

~/RapidCFD/andrew-dev$ nvidia-smi
Thu Jan 14 10:48:48 2016
+------------------------------------------------------+
| NVIDIA-SMI 352.68 Driver Version: 352.68 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+================= =====+======================|
| 0 GeForce GTX TIT... Off | 0000:02:00.0 On | N/A |
| 41% 67C P2 97W / 250W | 615MiB / 6143MiB | 47% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|================================================= ============================|
| 0 647 C interFoam 226MiB |
| 0 1400 G /usr/bin/X 236MiB |
| 0 2392 G compiz 131MiB |
| 0 23280 G /usr/lib/firefox/firefox 2MiB |
+-----------------------------------------------------------------------------+


On a larger mesh (1.4M cells), it uses more gpu RAM, but the cpu core remains maxed out.
~/RapidCFD/andrew-dev$ nvidia-smi
.
.
.

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|================================================= ============================|
| 0 969 C interFoam 1668MiB |
| 0 1400 G /usr/bin/X 225MiB |
| 0 2392 G compiz 131MiB |
| 0 23280 G /usr/lib/firefox/firefox 2MiB |
+-----------------------------------------------------------------------------+


Oddly, a laminar simulation seems to run at the same speed as a LES simulation. I assume that the limiting factor is something being done on the cpu.

[tripodtwostep]

Hello,
If I don't have root privilege of the cluster, is there any quick method to patch cuda math_function?
If not, should I compile cuda by myself?

Pin

Originally Posted by aee

Usage:
math_functions.hpp is found in /usr/local/cuda-7.5/targets/x86_64-linux/include/ (default install) or wherever you installed Cuda 7.5.

If in the default location, you will need root or sudo access for the following steps:
1) Place the patch file in the same directory.
2) Copy the original math_functions.hpp to math_functions.hpp.org and/or move it someplace safe.
2) execute
patch < Cuda7.5_math_functions.hpp.patch

[aee]

That might depend on your cluster setup. If the clients boot off the workstation on which you have a user account, and if you can set the environment for them, you should be able to do it. You'd have to install Cuda for your account, patch it and propogate it (set the env to point to your Cuda installation). I haven't tried RapidCFD on a cluster though, so you might ask around for more detailed help.

[tripodtwostep]

Firstly, I think patch cuda in its default directory /usr/local/cuda is not suitable for me because I have no root privilege in the remote cluster.
Secondly, after some investigations, I think it might be possible to install cuda toolkit in my own account. (see part 4.5 and installation option "--toolkitpath=<path>", http://docs.nvidia.com/cuda/cuda-get...#axzz46raIptgZ )
Thirdly, I read your RapidCFD installation note. It is very useful. I was also wondering which version of thirdparty package should I use before reading your note.
Thank you.

Originally Posted by aee

That might depend on your cluster setup. If the clients boot off the workstation on which you have a user account, and if you can set the environment for them, you should be able to do it. You'd have to install Cuda for your account, patch it and propogate it (set the env to point to your Cuda installation).

[aee]

I think I used the latest 3.0, but since it is only Paraview, you may not need to do anything other than point your env to it.

[aee]

I have a bunch of Aliases which set the bash env for of231, of30, of30x, mfix or rapidcfd. Once again, I don't know how that would work in your cluster, since it may be expecting each node to have the same software environment and to have the software installed local to the node (in which case, you are probably out of luck).

[tripodtwostep]

I installed the cuda toolkit somewhere in my home folder (use --silent --toolkit --toolkitpath=$MYCUDAHOME options), patched cuda 7.5 math_function.hpp, compiled RapidCFD-dev. Now I have solvers such as pisoFoam in ~/RapidCFD/RapidCFD-dev/platforms/linux64NvccDPOpt/bin.
However, pisoFoam crashed after "create mesh for time 0" with informations below:
---------start--------
terminate called after throwing an instance of 'thrust::system::system_error'
what(): function_attributes(): after cudaFuncGetAttributes: invalid device function
---------end---------
I looked up this error info in internet, somebody solved it after install cuda driver. For my case, I could compile cuda samples. After executing cuda deviceQuery, I get a positive answer:
CUDA Driver=CUDART, CUDA Driver Version=7.5, CUDA Runtime Version=7.5, Device0=Quadro 4000.

Could anybody give me a hint what is wrong? (I ran the cavity case in pisofoam-ras tutorials folder, used of3.0 blockMesh. I didn't do other modifications on the dictionaries. Should I change fvSolotion PCG to cudaCG like foam-extend?)

Originally Posted by aee

I have a bunch of Aliases which set the bash env for of231, of30, of30x, mfix or rapidcfd. Once again, I don't know how that would work in your cluster, since it may be expecting each node to have the same software environment and to have the software installed local to the node (in which case, you are probably out of luck).

[mesh-monkey]

I get a similar issue - I believe it is related to the cuda compute capability (CC, CUDA_ARCH or sm_xx). RapidCFD by default is designed to run using CC 3.0 - mostly cards with Kepler chips (see listing here). Your Quadro 4000, like my Quadro 6000, has a Fermi chip and is CC 2.0.

There was a previous pull request to adjust the code to make it work, but appears to have been abandoned.

The issue comes down to the difference in how CC 3.0 handles textures objects. The file in question is src/OpenFOAM/containers/Lists/gpuList/textures.H

I have tried unsuccessfully to modify the code to make it work, however I'm well over my head programming-wise.
Any cuda-programming geniuses around that could help?

Thanks, Tom

Originally Posted by tripodtwostep

I installed the cuda toolkit somewhere in my home folder (use --silent --toolkit --toolkitpath=$MYCUDAHOME options), patched cuda 7.5 math_function.hpp, compiled RapidCFD-dev. Now I have solvers such as pisoFoam in ~/RapidCFD/RapidCFD-dev/platforms/linux64NvccDPOpt/bin.
However, pisoFoam crashed after "create mesh for time 0" with informations below:
---------start--------
terminate called after throwing an instance of 'thrust::system::system_error'
what(): function_attributes(): after cudaFuncGetAttributes: invalid device function
---------end---------
I looked up this error info in internet, somebody solved it after install cuda driver. For my case, I could compile cuda samples. After executing cuda deviceQuery, I get a positive answer:
CUDA Driver=CUDART, CUDA Driver Version=7.5, CUDA Runtime Version=7.5, Device0=Quadro 4000.

Could anybody give me a hint what is wrong? (I ran the cavity case in pisofoam-ras tutorials folder, used of3.0 blockMesh. I didn't do other modifications on the dictionaries. Should I change fvSolotion PCG to cudaCG like foam-extend?)