[mnajafi]

Hi,
I'm trying to develop a CFD-DEM code. for the DEM, I've developed an in-house CUDA-based code. For CFD, I want to use OPENFOAM.
Before adding the fluid effect (OPENFOAM), my code was saved in a .cu file and I used nvcc to compile the code. The compiler can handle both native C and CUDA.
But then I add OPENFOAM to my code. When I try to compile the new code with nvcc, it doesn't recognize OPENFOAM's library (while I've installed OPENFOAM on the system). When I use wmake, it doesn't recognize CUDA libraries.
The code is like below:

C++ libraries
CUDA libraries
OPENFOAM libraries
some variable definition
CUDA kernels
int main()
{
C++ codes
CUDA kernel launch
OPENFOAM codes (calling mesh, solver etc)
}
So, what can I do? Just say that I want openfoam to be executed on CPU and not GPU.
Thanks

[Hillie]

Hi Mohammad,

I can only give you some possible hints here from my somewhat limited experience with cuda and OpenFoam. From what I have seen all the current openfoam methods work the same. The solvers that you define in the fvSolution file are the solvers that actually send the data to the CPU to solve for it. When you want to use GPU acceleration, you change the solvers in your fvSolution folder to solvers that send the data to the GPU. (this is were my main experience with GPU acceleration ends.)

The GPU solvers are explained here: http://openfoamwiki.net/index.php/GPGPU

These GPU solvers are the interface between the openFoam world and the cuda world. You need an interface object to handle the communication between the Openfoam and Cuda since i don't think you can compile them directly as you are trying to do.

It is a similar problem with FSI: Fluid-Structure Interaction (FSI) with OpenFOAM you need some way to handle the data transfer.

An other way to handle the issue is to have all the computation in cuda space. Have a look at the nvidia page for GPU accelerated CFD codes http://www.nvidia.com/object/computa..._dynamics.html.

Hope it helps,
Cheers

[mnajafi]

Hi Hilbert
Thanks very much for your quick response. I really appreciate your kindness. But actually I'm not intending to accelerate openfoam through CUDA. What I'm looking for is to couple CPU-based openfoam with an independent code written in CUDA. The CUDA code do something completely independent from openfoam but these two codes (openfoam and CUDA) are executed in the same main function. The problem is to compile the code. CUDA compiler (nvcc) has problem with oenfoam part of the code and openfoam (wmake) has problem with CUDA part. If I omit the openfoam part, nvcc can compile the code fine. Also wmake compile the code with no problem if I omit the CUDA part.
I've read something about separate compile and making object and linking but I've no practical idea yet.
Thanks

[Hillie]

Hi Mohammad,

I understand that you are trying to couple of openfoam with a cuda code.I don't have any practical experience, with this so I can only give you advice from what I have read.

Since you have CPU code and GPU code, you probably will want to look into an interface object. If you look at this thread https://www.researchgate.net/post/Ho...ram_be_written they discuss openACC, which might help.

Also have a think about the performance of the code though. These interface objects will slow the code down quite a bit. I am not sure how fancy your CFD needs to be , but you could think about writing your own CFD based on cuda so the entire code runs on the GPU. This will help a lot on the performance side.

Cheers

[mnajafi]

Thanks very much.
DEM time step is 1e-6 and CFD time step is 1e-4. So the impact of data transfer between CPU and GPU and execution of CFD is not so important compare to DEM execution time. However what you say is the best to transfer CFD to GPU as well but unfortunately I'm out of time and this is something I can't test it now.
By the way I'm really thankful for your helps.

[chegdan]

One easier way to couple CUDA with openFOAM is to treat your CUDA as a separate function in a separate library and then include a call to this function in OpenFOAM as an extern C call. Data can be transferred via custom container classes that will compile in CUDA and in C++ easily.

This is how I was able to create the linear system solvers (CUFFLINK) that come with foam extend now.

Good luck.

[mnajafi]

Originally Posted by chegdan

One easier way to couple CUDA with openFOAM is to treat your CUDA as a separate function in a separate library and then include a call to this function in OpenFOAM as an extern C call. Data can be transferred via custom container classes that will compile in CUDA and in C++ easily.

This is how I was able to create the linear system solvers (CUFFLINK) that come with foam extend now.

Good luck.

Thanks very much for your help. it's a big help.

[LHeisler]

Hello,

good you maybe post a minimal working example for such a case?