[fluidizedbed]

Dear all,


I have been working with a Discrete Ordinate radiation model (DOM) on top of using species and volumetric reaction model using a UDF. Due to the large mesh size (cell counts) as well as high number of directional discretization used in the DOM, my case files can have a size of 100ish MB while the data files range from 10 GB to 60 GB.


In the HPC cluster, normally I use 128 cores to do calculations in Fluent. The cluster is AMD EPYC 7H12 2x64 cores with 512 GB RAM running in Linux. Typically I often do calculations and save the cas and dat file and then reopen to continue calculations later. Also this is used to do step-by-step calculation iterations (flow->radiation and energy->species & flow), some sensitivity testings, or testing the solver settings.


The problem now is I can save the cas and dat file after doing some calculations but I cannot open the dat file when I want to open them in any future sessions.

There is always an error that causes Fluent process to stop. The error message in the TUI says:

Reading from nodeXXXX.XXXX.os:".../datafile.dat.h5" in NODE0 mode ...


  Reading results.
(cx-use-window-id 51)

==============================================================================

Node 0: Process 3281202: Received signal SIGSEGV.

==============================================================================
 The fluent process could not be started.

I have tried several things with my findings below:

• I did some trials to open the data file using various no of CPU. Nothing works except when opening the cas and dat file with 1 CPU. The problem with opening the dat file using 1 CPU is that any calculation that is made after that will be extremely slow (1 CPU vs 128 CPUs)
• I tried changing all combination of Interconnects and MPI types setting in the Fluent launcher for the Parallel setting. The default settings seem to be the best one (default for Interconnects and MPI types), also when tried to open with 1 CPU.
• I tried to add a compiler directive to parallelize the UDF, compile and load the parallelized UDF, redo calculations, and save as a new cas and dat file. I still cannot open the data file using 128 CPUs.
• I tested single precision setting (normally double precision) and also not use UDMI (normally I need UDMI) but none of them were the problems.

I am a bit running out of ideas and could find out how to solve this.
Anyone here might be familiar with such issues?

Is there any other step(s) that I can do also to help me understand what happens and report this error properly to the System Administrators?

[CFDKareem]

Hello, I have received a similar SIGSEGV working with very large meshes. I solved my problem by unselecting my GPU in the Fluent launcher. For my problem I believe it was exceeding the GPU VRAM.

I know you didn't mention you were using a GPU for computing, but hope it may help!

-Kareem

[fluidizedbed]

Originally Posted by CFDKareem

Hello, I have received a similar SIGSEGV working with very large meshes. I solved my problem by unselecting my GPU in the Fluent launcher. For my problem I believe it was exceeding the GPU VRAM.

I know you didn't mention you were using a GPU for computing, but hope it may help!

-Kareem

Hello Kareem, thanks for the reply!

My setting has been always with 0 GPU. So, this mean it is already done without GPU I suppose?

[CFDKareem]

Originally Posted by fluidizedbed

Hello Kareem, thanks for the reply!

My setting has been always with 0 GPU. So, this mean it is already done without GPU I suppose?

Yes, what solved it for me was switching GPU from 1 to 0. So unfortunately my fix isn't going to work for you!

This is now all speculation, but maybe try turning GPGPU computing on if you have a capable card. The DO models is one of the few models that can be accelerated with a GPU. See this link for turning on the acceleration and using NVIDIA MPS: https://cfdresearch.com/ansys-fluent-cfd-post-scripts/

Other than that I am not sure where to go. I'll try to do some more research on your setup and report back if I find anything. I've always struggled to "root cause" SIGSEGV errors.

Good Luck!

-Kareem