|
[Sponsors] |
[RapidCFD] Discussion thread on how to install and use RapidCFD |
![]() |
|
LinkBack | Thread Tools | Search this Thread | Display Modes |
![]() |
![]() |
#1 | |
New Member
Oskar Grochowalski
Join Date: Jan 2015
Posts: 9
Rep Power: 12 ![]() |
Quote:
Moderator note: This thread has been created for discussing how to install and use RapidCFD. For announcements, subscribe to the announcement thread: http://www.cfd-online.com/Forums/ope...nning-gpu.html Last edited by wyldckat; May 1, 2016 at 15:49. Reason: see "Moderator note:" |
||
![]() |
![]() |
![]() |
![]() |
#2 |
Senior Member
Robert
Join Date: Sep 2010
Posts: 158
Rep Power: 16 ![]() |
Are there any installation instructions?
I am currently attempting installation by: -Logging in with 'sudo bash' -Placing 'RapidCFD-dev' in the '/opt/' folder -'source etc/bashrc' in the RapidCFD-dev folder -'./Allwmake' in the RapidCFD-dev folder I get a lot of the following errors (separated by line breaks): UOPwrite.C:29:17: fatal error: mpi.h: No such file or directory (mpirun works on my computer) warning: call through incomplete class "Foam::tmp<Foam::Field<Foam::scalar>>" will always produce an error when instantiated /usr/local/cuda-7.0/include/math_functions.hpp:3191:22: error: call of overloaded ‘sqrt(double&)’ is ambiguous /usr/bin/ld: cannot find -ltriSurface /usr/bin/ld: cannot find -lsurfMesh collect2: error: ld returned 1 exit status make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libmeshTools.so] Error 1 /usr/bin/ld: cannot find -ltriSurface /usr/bin/ld: cannot find -lmeshTools /usr/bin/ld: cannot find -lsurfMesh collect2: error: ld returned 1 exit status make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libedgeMesh.so] Error 1 Last edited by lordvon; August 4, 2015 at 16:26. |
|
![]() |
![]() |
![]() |
![]() |
#3 |
New Member
Oskar Grochowalski
Join Date: Jan 2015
Posts: 9
Rep Power: 12 ![]() |
There are no installation instructions yet, but with growing community we are planning to publish it.
The good news is that RapidCFD builds just like a regular OpenFOAM. If you are not familiar with this process follow this steps: - set value of 'foamInstall' in 'etc/bashrc' file. This variable should point to the location of 'RapidCFD-dev' directory. By default this variable is set to $HOME/RapidCFD . - source 'etc/bashrc' file - run the 'Allwmake' file To build in parallel set WM_NCOMPPROCS environment variable to the number of processes you want to use. This is recomended since compilation with nvcc is very slow. Post a reply with your results ![]() |
|
![]() |
![]() |
![]() |
![]() |
#4 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
I'm not completely clear: does this software use the same environment variables as OF? If so, I risk messing up my OF 3 installation, right?
|
|
![]() |
![]() |
![]() |
![]() |
#5 | |
Retired Super Moderator
Bruno Santos
Join Date: Mar 2009
Location: Lisbon, Portugal
Posts: 10,978
Blog Entries: 45
Rep Power: 128 ![]() ![]() ![]() ![]() ![]() ![]() |
Quote:
|
||
![]() |
![]() |
![]() |
![]() |
#6 |
New Member
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 11
Rep Power: 10 ![]() |
Hi all,
I, like lordvon, am having troubles installing/compiling RapidCFD on my machine and am receiving similar errors. I'm hoping someone here would be able to help me out as I've scoured as much of the web as I could to find answers but to no avail. After following newoscar's instructions, as well as those on the github wiki and openfoam-wiki (thanks wyldckat!) as best possible, I seem to be receiving two types of issues, and many of each:
Some additional information about my setup that might be helpful:
Cheers, Tom |
|
![]() |
![]() |
![]() |
![]() |
#7 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
I have exactly the same problem compiling RapidCFD - both the Cuda math issue and the compiler looking for the -l* entries in usr/bin/ld (which does not exist). I haven't figured out where (or even if) that path is set in the bashrc or .sh files. Why would it look in usr/bin if the WM and FOAM env variables have been set?
I tried installing in opt/ and in $HOME. I tried setting the MPI to SYSTEMOPENMPI from OPENMPI (in line with my OpenFoam installations). I tried setting all the env variables directly. Now I am going to try giving up until someone else figures this out. |
|
![]() |
![]() |
![]() |
![]() |
#8 | |
New Member
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 11
Rep Power: 10 ![]() |
Hello again,
I think I rectified the issue, courtesy of aokomoriuta (via google translate) that led me to some old install instructions on the github wiki. Quote:
I definitely downloaded the latest code as .zip, so not sure if/why any code changes didn't come through or work, but I will log a bug on the github. For those looking at this in the future (and previous me), the math_functions.h issues were what was causing the cannot find -l* errors. Essentially the error happened during the build of one of the first libso libraries (fileFormats) and once that failed, all of the others after that also failed - hence why at the end of the file there is a long list. Cheers, Tom |
||
![]() |
![]() |
![]() |
![]() |
#9 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
I have Cuda 7.5 installed. I suspect the patch might work with Cuda 7.5, but I am not going to be the first to try. Would it make sense to install Cuda 6.5 into the Thirdparty-dev folder, patch and compile it there, and then set the Cuda env variables to that?
|
|
![]() |
![]() |
![]() |
![]() |
#10 |
New Member
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 11
Rep Power: 10 ![]() |
Hi aee,
I don't see any issue doing as you described to install cuda 6.5. You could also use the default install directory (/usr/local) and change the sybomolic link (/usr/local/cuda) to point to cuda-6.5 rather than cuda-7.5. As a note, I did find 6.5 to be a bit fussy with what gcc/g++ compiler version I used - had to be 4.7. If you do feel like trying cuda 7.5, you could always just make a backup of the math_function file and try that way. FYI I think the patch would be applied to the math_functions.hpp file. Another thing to note is what cuda architecture you build rapidcfd for - you will have to set this according to what is compatible with your gpu. By default it is set to sm_30 but I had to change this to sm_20 to make it work with the Q6000. Tom |
|
![]() |
![]() |
![]() |
![]() |
#11 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
I decided to try to use Cuda 7.5. The old math_functions.patch file for cuda 6.5 won't work, because the math functions targeted were moved to math_functions.hpp from Cuda 7 on, and the locations (line numbers) in the file are different. I edited the cuda 7.5 math_functions.hpp by hand based on a line-by-line read of the old patch file and then used #diff -c to build the attached patch file.
**** update Jan 14, 2016 *** updated patch file with corrected version (was missing a "::erfc" that is in the old patch) ***** end update *** The equivalent cuda 6.5 patch was applied to math_function.h starting at line 14273. In Cuda 7.5, the applicable section is found in math_functions.hpp starting at line 3377 (i.e., in the "all platforms" section). Usage: math_functions.hpp is found in /usr/local/cuda-7.5/targets/x86_64-linux/include/ (default install) or wherever you installed Cuda 7.5. If in the default location, you will need root or sudo access for the following steps: 1) Place the patch file in the same directory. 2) Copy the original math_functions.hpp to math_functions.hpp.org and/or move it someplace safe. 2) execute patch < Cuda7.5_math_functions.hpp.patch Remaining issue: Even though my Cuda 7.5 installation is native from the Ubuntu software center, and my environment variables are set according to the instructions, deviceQuery doesn't detect my Titan Black card. The card appears properly set up... lspci | grep -i nvidia gives 02:00.0 VGA compatible controller: NVIDIA Corporation GK110B [GeForce GTX Titan Black] (rev a1) 02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1) Perhaps there was a conflict between existing drivers and Cuda that I didn't see. Last edited by aee; January 14, 2016 at 12:07. Reason: updated patch file |
|
![]() |
![]() |
![]() |
![]() |
#12 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
The issue with deviceQuery not seeing the card cleared up after a reboot. Of all things...
The compiler is still looking in usr/bin. I have to fix the openmpi-1.10.0 configuration issue (I am using Thirdparty-dev from the current openFoam). Maybe that will fix it. Last edited by aee; January 13, 2016 at 19:39. Reason: compiler finished with errors |
|
![]() |
![]() |
![]() |
![]() |
#13 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
I was finally able to compile rapidCFD with the newly patched cuda7.5 math_function.hpp -- after cleaning out everyting in RapidCFD-dev and cloning the git, sourcing etc/bashrc, exporting WM_NCOMPPROCS=12 and running ./Allwmake. A detailed readme.txt is attached.
I can run small and large interFoam cases, after adjusting the case files for unsupported models, solvers and preconditoners. e.g., for on the OF2.3.1 tutorial LES/nozzleFlow2D Unknown LESdelta type smooth Even for this small case, one CPU core is maxed out, while nvidia-smi shows interfoam is using very little of the GPU. I may have to play with the case files to see what actually runs on the gpu. I imagine it doesn't help that I am using the GPU for display as well. ~/RapidCFD/andrew-dev$ nvidia-smi Thu Jan 14 10:48:48 2016 +------------------------------------------------------+ | NVIDIA-SMI 352.68 Driver Version: 352.68 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+================= =====+======================| | 0 GeForce GTX TIT... Off | 0000:02:00.0 On | N/A | | 41% 67C P2 97W / 250W | 615MiB / 6143MiB | 47% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |================================================= ============================| | 0 647 C interFoam 226MiB | | 0 1400 G /usr/bin/X 236MiB | | 0 2392 G compiz 131MiB | | 0 23280 G /usr/lib/firefox/firefox 2MiB | +-----------------------------------------------------------------------------+ On a larger mesh (1.4M cells), it uses more gpu RAM, but the cpu core remains maxed out. ~/RapidCFD/andrew-dev$ nvidia-smi . . . +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |================================================= ============================| | 0 969 C interFoam 1668MiB | | 0 1400 G /usr/bin/X 225MiB | | 0 2392 G compiz 131MiB | | 0 23280 G /usr/lib/firefox/firefox 2MiB | +-----------------------------------------------------------------------------+ Oddly, a laminar simulation seems to run at the same speed as a LES simulation. I assume that the limiting factor is something being done on the cpu. Last edited by aee; January 16, 2016 at 12:40. Reason: clarify |
|
![]() |
![]() |
![]() |
![]() |
#14 | |
New Member
Pin Lyu
Join Date: Mar 2016
Location: Harbin
Posts: 6
Rep Power: 10 ![]() |
Hello,
If I don't have root privilege of the cluster, is there any quick method to patch cuda math_function? If not, should I compile cuda by myself? Pin Quote:
|
||
![]() |
![]() |
![]() |
![]() |
#15 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
That might depend on your cluster setup. If the clients boot off the workstation on which you have a user account, and if you can set the environment for them, you should be able to do it. You'd have to install Cuda for your account, patch it and propogate it (set the env to point to your Cuda installation). I haven't tried RapidCFD on a cluster though, so you might ask around for more detailed help.
|
|
![]() |
![]() |
![]() |
![]() |
#16 | |
New Member
Pin Lyu
Join Date: Mar 2016
Location: Harbin
Posts: 6
Rep Power: 10 ![]() |
Firstly, I think patch cuda in its default directory /usr/local/cuda is not suitable for me because I have no root privilege in the remote cluster.
Secondly, after some investigations, I think it might be possible to install cuda toolkit in my own account. (see part 4.5 and installation option "--toolkitpath=<path>", http://docs.nvidia.com/cuda/cuda-get...#axzz46raIptgZ ) Thirdly, I read your RapidCFD installation note. It is very useful. I was also wondering which version of thirdparty package should I use before reading your note. Thank you. Quote:
|
||
![]() |
![]() |
![]() |
![]() |
#17 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
I think I used the latest 3.0, but since it is only Paraview, you may not need to do anything other than point your env to it.
|
|
![]() |
![]() |
![]() |
![]() |
#18 |
Member
Andrew Eisenhawer
Join Date: Nov 2012
Location: Alberta, Canada
Posts: 35
Rep Power: 13 ![]() |
I have a bunch of Aliases which set the bash env for of231, of30, of30x, mfix or rapidcfd. Once again, I don't know how that would work in your cluster, since it may be expecting each node to have the same software environment and to have the software installed local to the node (in which case, you are probably out of luck).
|
|
![]() |
![]() |
![]() |
![]() |
#19 | |
New Member
Pin Lyu
Join Date: Mar 2016
Location: Harbin
Posts: 6
Rep Power: 10 ![]() |
I installed the cuda toolkit somewhere in my home folder (use --silent --toolkit --toolkitpath=$MYCUDAHOME options), patched cuda 7.5 math_function.hpp, compiled RapidCFD-dev. Now I have solvers such as pisoFoam in ~/RapidCFD/RapidCFD-dev/platforms/linux64NvccDPOpt/bin.
However, pisoFoam crashed after "create mesh for time 0" with informations below: ---------start-------- terminate called after throwing an instance of 'thrust::system::system_error' what(): function_attributes(): after cudaFuncGetAttributes: invalid device function ---------end--------- I looked up this error info in internet, somebody solved it after install cuda driver. For my case, I could compile cuda samples. After executing cuda deviceQuery, I get a positive answer: CUDA Driver=CUDART, CUDA Driver Version=7.5, CUDA Runtime Version=7.5, Device0=Quadro 4000. Could anybody give me a hint what is wrong? (I ran the cavity case in pisofoam-ras tutorials folder, used of3.0 blockMesh. I didn't do other modifications on the dictionaries. Should I change fvSolotion PCG to cudaCG like foam-extend?) Quote:
|
||
![]() |
![]() |
![]() |
![]() |
#20 | |
New Member
Tom
Join Date: Dec 2015
Location: Melbourne, Australia
Posts: 11
Rep Power: 10 ![]() |
I get a similar issue - I believe it is related to the cuda compute capability (CC, CUDA_ARCH or sm_xx). RapidCFD by default is designed to run using CC 3.0 - mostly cards with Kepler chips (see listing here). Your Quadro 4000, like my Quadro 6000, has a Fermi chip and is CC 2.0.
There was a previous pull request to adjust the code to make it work, but appears to have been abandoned. The issue comes down to the difference in how CC 3.0 handles textures objects. The file in question is src/OpenFOAM/containers/Lists/gpuList/textures.H I have tried unsuccessfully to modify the code to make it work, however I'm well over my head programming-wise. Any cuda-programming geniuses around that could help? Thanks, Tom Quote:
|
||
![]() |
![]() |
![]() |
|
|