[andrea.pasquali]

Hi,
I'm trying the SpeedIT Classic version with OpenFOAM plugin on OF 1.7.
I compiled the SpeedIT library and OF plugin well (I think) following all the README.txt files but when I try icoFoam with PCG_accel solver I obtain "ERROR : solver function returned -1".
My solver function is:
solver_result = sicl_gscsrcg( n_rows, pgpu_vals, pgpu_c_idx, pgpu_r_idx, pgpu_X, pgpu_B, precond, &n_iter, &eps) ;

Am I forgetting something?

Thanks in advance for any help

Andrea

[fsantos]

Hi andrea, have you recompiled OF in single precision?? It can be the reason of the solver error.

Best regard
Fábio Santos

[chegdan]

Instead of recompiling OF in single precision (and you want double precision), if you have a 1.3 compute capable card for double precision on your computer, change the floats to double in the speedIt cuda code (if you know some CUDA) and then try it again.

to compile CUDA code in doubel precision at the -arch sm-1.3 flag to your nvcc calls. this might be in a compile script.

Dan

[phan]

Hello! I'm also trying to set up the OpenFoam Plugin. But as I'm not that familiar with C I don't understand how to insert the solver functions correctly.
If I delete the comment signs of the given functions and compile I get the following error message:

"si_... was not declared in this scope"

And if I don't insert any function it can be compiled without any problem but as soon I start the simulation I get the message:

"Call your own solver from PBiCG_accel.C file"

Do I need to edit the PBiCG_accel.C file at all?
I switched OpenFoam already to single precision.

I would really appreciate your help. Thanks in advance,
Stephan

[Lukasz]

Please post the question to the forum at speedit.vratis.com as it does not concern OF users directly (you need to register first). We will be happy to answer your question.

[xizhijun]

dear lukasz ,please respect that openfoam is free software and that you should respect everybodys opinion if you take the freedom to advertise your products in this forum .i and my colleges don't like this style .can somebody please post the openfoam cuda plugin to the git repository so that we can evaluate the code without registering to this company?
thanks zhijun !

[Lukasz]

Dear All,

We are happy to announce a new release of the OpenFOAM plugin 1.1 (GPL License).
Here is the list of features:

-Multi-GPU support.
-Tested on Fermi architecture (GTX460 and Tesla C2050).
-Automated submission of the domain to the GPU cards (using decomposePar from OpenFOAM).
-Optimized submission of computational tasks to the best GPU card in the system for any number of computational threads.
-Plugin picks the most powerful GPU card for a single thread cases.

You can freely download it at speedit.vratis.com. Enjoy!

[andrea.pasquali]

Hi,
I'm coming back to use Speed IT + OpenFOAM plug in.
I installed the new CUDA toolkit 3.2, the Speed IT 1.0, and the OpenFOAM plug in 1.0.
When I run the "icoFoam" test with PCG_accel I have:

--> FOAM FATAL ERROR:
Can not allocate GPU memory

FOAM exiting

I don't know where is the probem,
could anyone help me ?

Thanks

Andrea

[andrea.pasquali]

I found Cuda Error number 38 = cudaErrorNoDevice = No available CUDA device.
I meet this error when PCG_accel.C calls the first cuda command (cudaMalloc).
What means "No available CUDA device" ?
I tested the examples in SDK and they work well.
Any help?

Thanks

[andrea.pasquali]

Hi,
I solve my (first) problem . Now I have problem in single / double precision. I'm trying to compile speed it in double (I changed the float in double inside .cu and .h files)
I run make ad I obtained:

si_classic.cu(226): error: argument of type "const double *" is incompatible with parameter of type "const float *"

si_classic.cu(242): error: argument of type "const double *" is incompatible with parameter of type "const float *"

si_classic.cu(242): error: argument of type "const double *" is incompatible with parameter of type "const float *"

226 is cublasSnrm2 that is float (in cublas.h)
226 is cublasSdot that is float (in cublas.h)

What I have to do? I'm trying to rechange they from double to float but I found other errors... have I to change the cublas.h file?

Thanks for any help

Andrea

[andrea.pasquali]

I recompiled the si_classic.cu and si_classic.h in double precision but my GPU is not still working. This is what I did:

- In .cu file I changed the "cublasSnrm2" and "cublasSdot" functions in "cublasDnrm2" and "cublasDdot", the "sicl_gs..." solver functions in "si_gd..." and all floats in double;
- In .h file I changed the "sicl_gs..." solver functions in "sicl_gd..." and and all floats in double;
- In Makefile I added at nvcc command the option "--gpu-architecture sm_13" and the compilation with "make" command is ok without any errors;
- Then I recompiled the PCG_accel.C lib adding the "sicl_gdcsrcg" solver function in "#elif defined(WM_DP) // scalar is double" statment without success. I have error "sicl_gdcsrcg was not declared in this scope";
- But if I add the "sicl_gdcsrcg" solver function in "#if defined(WM_SP) // scalar is float" statment the compilation is up to date! My OpenFOAM is in double precision! However the "icoFoam" test gives me this result at pressure iteartion:

ERROR : solver function returned -1
diagonalPCG_accel: Solving for p, Initial residual = 0, Final residual = 1e-06, No Iterations 1000

Am I forgetting anything else needs to be changed from float to double? Or am I wronging anything else?
Thanks for any help

Andrea

[kyle]

Did anyone ever get this working? I got everything compiled and installed, but I am getting a seg fault immediately after "Time = 1". If I remove the "_accel" from the solvers then it runs fine, so everything is good FOAM wise.

I am using CUDA 3.2, OpenFOAM 1.6-ext, GCC 4.4.4 and RHEL 6.

To get the plugin compiled I had to go in and make several private members of "lduMatrix::solver" public. It seems like this should be documented if it is actually required, so I am wondering if I am using an incompatible set of software versions, or if I am just doing something else wrong.

[kyle]

Judging from this Japanese blog post, we are looking not looking at any benefit anyway. This guy saw a 3x slowdown when compared to unaccelerated GAMG. This is the only data I have seen that isn't from the developers themselves. He used extremely small cases, so that might be limiting the GPU solvers somehow. Also if Google translate is to be trusted, the GPU accelerated solutions are not even accurate.

http://www.geocities.co.jp/SiliconVa...T-Classic.html

[wyldckat]

Greetings Kyle,

I've never used SpeedIT so I don't know why the code isn't working for you. But I have accompanied some of the posts about SpeedIT and I can at least advise about one detail: the SpeedIT Classic is for single precision only, so it can only be fairly comparable with the single precision build of OpenFOAM.
And by my experience, GAMG doesn't always work for every case.

So in either case, it really depends on the case you're trying to solve and if single precision is enough! It's like running the case decomposed in parallel with CPUs only: the usual measure for estimated performance is to have at least 50k cells per core. But this is only for some usual case scenarios that some people have. It all depends on the equations that need to be solved and how well it can be done in parallel.

As for putting GPU into the fighting ring, it also depends on the balance between the number of calculations to be done per iteration vs the amount of data to be transferred between the CPU's (system) RAM and GPU's (graphics card) RAM.

Either way, that same blogger has an interesting comparison with Inria's CNC: http://www.geocities.co.jp/SiliconVa...M/CNC/CNC.html

Best regards,
Bruno

[kyle]

wyldckat,

I am running single precision, which is fine for my use case. Thanks for the link, I did not know there was another OpenFOAM solver that ran on the GPU. It looks like it might be a little better than speedit.

And as far as copying between system RAM and GPU RAM, this should only happen at the beginning and end of the run. If you are exchanging the entire solution over the PCI-e bus every iteration then any and all advantage the GPU might have had is gone. Unfortunately this means you need to be able to fit the entire mesh and solution into GPU RAM, which if you have a decent sized case requires a lot of video cards!

[stainboy]

Hi,

I'm from the Speedit team and I hope that I might help here

I just finished the tests of Speedit Classic with OpenFoam 1.7.1. Everything works fine.
I have Ubuntu 10.10 32 bit with Cuda 3.2 on GTX 460 and gcc 4.4.4.

1. I have compiled OpenFoam from sources with single precision flag ($WM_PRECISION_OPTION variable set to SP in OpenFOAM/etc/basrc file). Speedit Classic offer solvers only single precision calculations.
2. Compiled Speedit Classic available from the webpage. http://speedit.vratis.com/
3. Compiled SpeeitOpenFoam plugin 1.1. Here some additional changes.
4. Because classic version of Speedit allows you to run only single precision calculations you can run only PCG_accel solver and functions with prefix sicl_ from si_classic.h file.
5. In PCG_accel.C and PBiCG_accel.C uncomment speedit classic header
   

   #include "si_classic.h"
   //#include "speedit_ex.h"

6. To be sure in file PBiCG_accel.c comment all available solvers and uncoment FoamFatalError lines. Those solvers are from Extreme version
   

   lines (213-239)
   /*
   	INSERT SOLVER FUNCTION HERE
   */
   #if defined(WM_SP)     // scalar is float
   
   Foam::FatalError << "Call your own solver from PBiCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
   
   //solver_result = si_cscsrbicgstab( n_rows, 
   //						   p_vals, p_c_idx, p_r_idx, 
   // 			 		 	   X, B,
   // 						   precond,
   // 					 	  &n_iter, &eps) ;
   
   //solver_result = si_gscsrbicgstab( n_rows, 
   //						pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
   // 			 			pgpu_X, pgpu_B,
   //						precond,
   // 						&n_iter, &eps) ;
   
   #elif defined(WM_DP)   // scalar is double
   		
   Foam::FatalError << "Call your own solver from PBiCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
   
   	//	solver_result = si_cdcsrbicgstab( n_rows, 
   	//						p_vals, p_c_idx, p_r_idx, 
   	// 			 			X, B,
   	//						precond,
   	// 						&n_iter, &eps) ;
   
   //solver_result = si_gdcsrbicgstab( n_rows, 
   //					pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
   //			 		pgpu_X, pgpu_B,
   //					precond,
   //					&n_iter, &eps) ;

7. Go to PCG_accel.C and make solver from classic version available
   

   line 208
   /*
   	INSERT SOLVER FUNCTION HERE
   */
   #if defined(WM_SP)     // scalar is float
   
   //Foam::FatalError << "Call your own solver from PCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
   
   //solver_result = si_cscsrcg( n_rows, 
   //				p_vals, p_c_idx, p_r_idx, 
   // 			        X, B,
   //				precond,
   //	 			&n_iter, &eps) ;
   
   //solver_result = si_gscsrcg( n_rows, 
   //					pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
   //		 			pgpu_X, pgpu_B,
   //					precond,
   //		 			&n_iter, &eps) ;
   
   solver_result = sicl_gscsrcg( n_rows, 
   					pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
   					pgpu_X, pgpu_B,
   					precond,
   					&n_iter, &eps) ;
   
   #elif defined(WM_DP)   // scalar is double
   		
   Foam::FatalError << "Call your own solver from PCG_accel.C file" << Foam::exit(Foam::FatalError, -1) ;
   
   	//	solver_result = si_cdcsrcg( n_rows, 
   	//			p_vals, p_c_idx, p_r_idx, 
   	// 			X, B,
   	//		        precond,
   	//			&n_iter, &eps) ;
   
   //	solver_result = si_gdcsrcg( n_rows, 
   //				pgpu_vals, pgpu_c_idx, pgpu_r_idx, 
   //				pgpu_X, pgpu_B,
   //				precond,
   //				&n_iter, &eps) ;
   
   #endif

8. compile plugin by typing
   

   ~/OpenFOAM/OpenFOAM-1.7.1/OpenFOAM_SpeedIT_plugin$ wmake libso

9. Copy the libraries libSpeedIT_Classic.so libcublas.so libcudart.so
   to $FOAM_LIBBIN directory. In my case in $FOAM_LIBBIN among many files you should have also
   

   /home/kuba/OpenFOAM/OpenFOAM-1.7.1/lib/linuxGccSPOpt/
   libexternalsolv.so
   libSpeedIT_Classic.so
   libcudart.so
   libcublas.so

10. Now modify the fvSolution file from icoFoam/cavity/system folder. Change solvers part to:
    

    solvers
    {
        p
        {
            solver              PCG_accel;
            preconditioner  diagonal;
            tolerance         1e-06;
            relTol              0;
        }
    
        U
        {
            solver             PBiCG;
            preconditioner  DILU;
            tolerance         1e-05;
            relTol               0;
        }
    }

11. Add following entry at the end of the controlDict to tell OpenFoam which library are necessary to use with PCG_accel solver
    

    runTimeModifiable yes;
    
    libs
        (
            "libcudart.so"
            "libcublas.so"
            "libSpeedIT_Classic.so"
            "libexternalsolv.so"
        );

12. Run the blockMesh and then icoFoam.
    
    I hope that this post helped.
    Kuba.

[Di.Lee]

The plugin works. Thank Jacub very much.

[onyir]

Originally Posted by andrea.pasquali

Hi,
I'm coming back to use Speed IT + OpenFOAM plug in.
I installed the new CUDA toolkit 3.2, the Speed IT 1.0, and the OpenFOAM plug in 1.0.
When I run the "icoFoam" test with PCG_accel I have:

--> FOAM FATAL ERROR:
Can not allocate GPU memory

FOAM exiting

I don't know where is the probem,
could anyone help me ?

Thanks

Andrea

Hi, Andrea, how did you manage to solve that issue? I'm having the same problem.

Thanks.

[Lukasz]

We are about to release a new version of the OpenFOAM plugin where you could run SpeedIT Classic as well as SpeedIT 2.1/Multi-GPU in case of larger cases that do not fit into one GPU card. The new release should solve the problem.

[mborgraeve]

Hello !
I have at least one question :
I am trying to use SpeedIT classic with the plugin2.1 for some benchmarks, and i don't manage to use the accel_solvers : the compilation of the plug-in crashes, and tell me he is missing some vector_type.h file.
I'm running on Kubuntu 12.04 64b fully up-to-date, installed cuda and OpenFoam managing an error concerning binutils and binutils-gold which give two version of l, each compile and run without problem)
Has anybody try or manage to use a recent version of it ?
Best regards,

EDIT Actually i found : i have to make a Allwmake FREE instead of COMMERCIAL