[Kojote]

Hi

I tried to install cuda solvers for ext3.0. What i did ->

- installed 3.0 without problems
- installed nvidia driver and cuda 5.5
- then i changed in the prefs.sh ->
# System installed CUDA
export CUDA_SYSTEM=1
export CUDA_DIR=/usr/bin/cuda-5.5
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

- then i tried to compile the cudaSolvers

but i get a message like this

Found nvcc but \$CUDA_IGNORE set -- not enabling CUDA support.

Or where is this flag? $CUDA_IGNORE

OK found it in the /Allwmake.firstInstall

thx for help me out

Br

Christian

[Kojote]

Hi

Something is wrong with my setup.
I cannot compile Cuda.
Did someone the installation successfully?


Br

Christian

[dominik_christ]

Hi Christian,
the installation works fine on my machine.
Can you please post more information?

* Is the Cuda installation itself working, can you compile and run the Cuda tutorial cases?
* Where is your Cuda installation located? /usr/bin/cuda seems a little strange to me. If you installed from .deb package, Cuda is located in /usr/local/cuda/ . If you type "which nvcc" and "locate bin/nvcc" what output do you get?
* How do you try to compile the Cuda solvers in foam-extend? Which commands are you running?
* What is the exact error message that you get?

Best regards,
Dominik

[Kojote]

Hi

Thx for answer. I am at work in the moment. Will get the infos asap. But good to hear that cuda is possible.

Br

Christian

[Kojote]

* Is the Cuda installation itself working, can you compile and run the Cuda tutorial cases?
yes make is working in samples

* Where is your Cuda installation located? /usr/bin/cuda seems a little strange to me. If you installed from .deb package, Cuda is located in /usr/local/cuda/ . If you type "which nvcc" and "locate bin/nvcc" what output do you get? you are right, i posted wrong path

which nvcc ->
/usr/local/cuda/bin/nvcc

prefs.sh:
export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

settings.sh
# CUDA if available
# ~~~~~~~~~~~~~~~~~
[ -z "$CUDA_SYSTEM" ] && [ -e /usr/local/cuda/bin/nvcc ] && {
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
}



* How do you try to compile the Cuda solvers in foam-extend? Which commands are you running?

./Allwmake in the cudaSolvers

* What is the exact error message that you get?

Found nvcc but $CUDA_IGNORE set -- not enabling CUDA support

but yesterday the flag was ok and then there were errors, too.

what can i do?

[/QUOTE]

many thanks for help

christian

[dominik_christ]

Hi Christian,

When I search all subdirectories for occurrence of CUDA_IGNORE ("grep -r CUDA_IGNORE"), I find only two files: Allwmake.firstInstall and src/cudaSolvers/Allwmake.

Only Allwmake.firstInstall sets this environment variable, so it appears to me that you are running ./Allwmake.firstInstall before going into src/cudaSolvers and running ./Allwmake there.

When setting CUDA_IGNORE, the Allwmake.firstInstall script posts a message:

---
$CUDA_ARCH is required by nvcc compiler but not set.
Check section '-gpu-architecture' in 'man nvcc' for details.

Proceed without compiling cudaSolvers? [Y/n]
---

So the remedy would be to set CUDA_ARCH as appropriate for your hardware. I have K20 Tesla cards here and "export CUDA_ARCH=sm20" is the setting that I use.

To automate setting of CUDA_ARCH, you can use etc/presh.sh-EXAMPLE as a template to make a customized etc/prefs.sh with your local settings.

Hope this helps,
Dominik

[Kojote]

Originally Posted by dominik_christ

Hi Christian,

When I search all subdirectories for occurrence of CUDA_IGNORE ("grep -r CUDA_IGNORE"), I find only two files: Allwmake.firstInstall and src/cudaSolvers/Allwmake.

Only Allwmake.firstInstall sets this environment variable, so it appears to me that you are running ./Allwmake.firstInstall before going into src/cudaSolvers and running ./Allwmake there.

When setting CUDA_IGNORE, the Allwmake.firstInstall script posts a message:

---
$CUDA_ARCH is required by nvcc compiler but not set.
Check section '-gpu-architecture' in 'man nvcc' for details.

Proceed without compiling cudaSolvers? [Y/n]
---

So the remedy would be to set CUDA_ARCH as appropriate for your hardware. I have K20 Tesla cards here and "export CUDA_ARCH=sm20" is the setting that I use.

To automate setting of CUDA_ARCH, you can use etc/presh.sh-EXAMPLE as a template to make a customized etc/prefs.sh with your local settings.

Hope this helps,
Dominik

Hi Dominik,

Thanks for answer again.
i did all from scratch again. No my cuda settings are 100% OK. Cuda tutorils are compiled. When i run ./Allwmake.firstinstall the scipt sais something like "found nvcc, cuda is compiled by default" I adjusted the settings.sh and prefs.sh before that. While comping OpenFoam the massage with the flag is in the log file.
Cuda will be not compiled. When i do "wmake libso" in the cudaSolver folder then "make has no target" (i use the german ubuntu so i don't know the massage in english).

By the way what are your experience with cuda.

Br

Christian

[dominik_christ]

Hi Christian,

When you post, please provide exactly the steps you are taking with verbatim commands, post log files (as attachments if possible) and error messages as they are because it is very difficult to identify anything from a written summary (and I cannot tell if you e.g. forgot to source the etc/bashrc file at some point, or something like that)
I can live with German error messages.

Regarding my experience with cuda and foam-extend:
1) It works ...

2) ... except for coupled boundary conditions, like cyclic, processor and ggi. These need to be updated in every iteration that a solver (e.g. BiCG) is doing for a transport equation. This would require:
a) Uploading the intermediate data back to CPU, update the cyclic BCs and load the changed values back to the GPU (=very time-consuming)
or
b) Implementing all coupled boundary conditions with GPU code.

So this is tricky by principle. I believe that implementing general coupled boundary condition support for GPU is the major open task for the future.

3) I have not yet made a valid comparison, that is between a good CPU setup and a good GPU setup. When time allows, there will be something to show at the workshop.

Best regards,
Dominik

[Kojote]

Hi Dominik

OK. I have to reinstall a computer next week. Then i will write out all logs etc. I would be very nice if you could have a look at them if i failure again.

Some more questions:

Do you have to decompose a cuda case, or is this be done by the hardware?
And is it possible to run a case with lagrangian particle on cuda?

!!! Many thanks for your help !!!!

Br

Christian

[xxxx]

Hello!

I'm trying to install cudaSolvers too, but I found the same problems as Kojote.
I managed to compile foam-ext-3.0 with cuda ( "found nvcc, cuda is compiled by default") but when I try to run ./Allwmake in cudaSolvers dir I get:

linux@linux-Precision-T7600:~/foam/linux-3.0/src/cudaSolvers$ ./Allwmake
Found nvcc but $CUDA_IGNORE set -- not enabling CUDA support.

and if I try wmake libso instead of ./Allwmake I get:

linux@linux-Precision-T7600:~/foam/linux-3.0/src/cudaSolvers$ wmake libso
wmakeLnInclude: linking include files to ./lnInclude
make: *** No rule to make target `cudaCG/cgDiag.dep', needed by `Make/linux64Gcc46DPOpt/dependencies'. Stop.

How did you manage to make it work, dominik_christ?
Thanks in advance

[Kojote]

Hi

Now i did it all again on another computer. Ubuntu 12.04 64bit / CUDA 5.5.
Nvidia driver (319.37) from the cuda run file. My card is a GeForce GTX 650 Ti and from what i know i have to use sm_30.
And i have the same situation like Federico described.

I did -> export CUDA_IGNORE=0, with the Allwmake.firstInstall the flag is not written.

And with "./Allwmake" in the cudaSolver folder or "wmake libso" getting this:

Found nvcc -- enabling CUDA support.
make: *** Keine Regel vorhanden, um das Target »cudaCG/cgDiag.dep«,
benötigt von »Make/linux64Gcc46DPOpt/dependencies«, zu erstellen. Schluss.

Br

Christian

[xxxx]

Hello Kojote.

Since it seems that we are the only people interested in CUDA compiling on this forum, would you like to keep in contact? Do you have any news about CUDA compiling?

[Kojote]

Hi,

Yes to keep in touch, no to the news.

Br

Christian

[dominik_christ]

Dear Christian and Federico,

I could reproduce some of the issues and found some solutions:
1) The logic in Allwmake must be inverted: It tests for [ -z $CUDA_IGNORE], which is for the absence of CUDA_IGNORE. The logic has to be: If IGNORE is absent, do compile. The "then" and "else" part must therefore be swapped to:
if [ -z $CUDA_IGNORE ]
then
echo "Found nvcc -- enabling CUDA support."
wmake libso
else
echo "Found nvcc but \$CUDA_IGNORE set -- not enabling CUDA support."
fi

2) "wmake libso" should work independently from this. The compilation error that Christian is seeing comes from a broken dependency. To re-create dependencies, run "wclean" before "wmake libso" (but please read the points below before that).

3) The cufflink headers in "*.cu" and "include/*.H" in directory "cudaSolver/" have become messed up. For a quick fix, add a closing comment "*/" at the end of the descriptive header before the actual code for these files.

4) If you get any error with "gpu-architecture", then CUDA_ARCH is not set. As mentioned in the post above, you can permanently set it in etc/prefs.sh (use etc/prefs.sh-EXAMPLE as template). My configuration is:

export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib64
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

With these changes, I can compile cudaSolvers with the latest version of the git repository version. I will make a bugfix from above points and submit it to the foam-extend project.

Hope this solves the issues on your machines as well.

Best regards,
Dominik

[Kojote]

hi dominik

wow will try asap

manythx for your help and for your work

christian

[rkc.cfd]

Originally Posted by dominik_christ

Dear Christian and Federico,

The cufflink headers in "*.cu" and "include/*.H" in directory "cudaSolver/" have become messed up. For a quick fix, add a closing comment "*/" at the end of the descriptive header before the actual code for these files.

Dominik

That was funny when I found out that somebody really wanted us to read the code

Also when I compiled it was asking for the cusp definitions

lnInclude/cudaTypes.H:53:29: error: cusp/coo_matrix.h: No such file or directory

So I downloaded from the git repository and added the directory to make file options.

Thanks to all of you on this thread.

[xxxx]

Thank you dominik_christ,

Actually I already fixed this:

Originally Posted by dominik_christ

3) The cufflink headers in "*.cu" and "include/*.H" in directory "cudaSolver/" have become messed up. For a quick fix, add a closing comment "*/" at the end of the descriptive header before the actual code for these files.

And this:

Originally Posted by dominik_christ

4) If you get any error with "gpu-architecture", then CUDA_ARCH is not set. As mentioned in the post above, you can permanently set it in etc/prefs.sh (use etc/prefs.sh-EXAMPLE as template). My configuration is:

export CUDA_SYSTEM=1
export CUDA_DIR=/usr/local/cuda
export CUDA_BIN_DIR=$CUDA_DIR/bin
export CUDA_LIB_DIR=$CUDA_DIR/lib64
export CUDA_INCLUDE_DIR=$CUDA_DIR/include
export CUDA_ARCH=sm_20

with CUDA_ARCH=sm_30 (my machine!) before to try to compile with wmake libso but I still can't compile cudaSolvers.

Have you tested it? Do they work now? (the gpu solvers I mean)

[Kojote]

Hi

good to know that you were able to compile

Originally Posted by rkc.cfd

That was funny when I found out that somebody really wanted us to read the code

Also when I compiled it was asking for the cusp definitions

lnInclude/cudaTypes.H:53:29: error: cusp/coo_matrix.h: No such file or directory

So I downloaded from the git repository and added the directory to make file options.

Thanks to all of you on this thread.

could you upload your cuda solver folder here or somewhere else

thx

christian

[Kojote]

Hi

I'm still running in trouble. I did the fix for the headers and the script. After "wclean" and "wmake libso" i still get this.

Making dependency list for source file cudaSolver/cudaSolver.C
make: *** Keine Regel vorhanden, um das Target »cudaCG/cgDiag.dep«,
benötigt von »Make/linux64Gcc46DPOpt/dependencies«, zu erstellen. Schluss.

Br

Christian

[rkc.cfd]

Hi,

Sorry for the late reply, some how I was not subscribed to the thread and did not get the message.

The link to my github is

https://github.com/mechysolv/openfoa...extend-3.0.git

I have downloaded the cusp solvers to

$WM_THIRD_PARTY_DIR/LocalDev

from

https://github.com/cusplibrary/cusplibrary

Hope that helps,

RKC