[tlandfried]

Morning all,

Sorry, this is a repost, I posted this in the Mesh Utilities forum sometime early last week and haven't received any responses. I thought perhaps this is a better forum to post it, where it might be seen. Anyhow, on to my problem.

I've run into a requiring problem that I can't seem to fix. So, I thought I'd bring the problem here and see if anyone has some familiarity with it or any insight. Additionally, I've posted the problem on collab.pitt.edu.

The problem I'm having involves using the mapFields utility on a supercomputing cluster, SAM at Pitt. When I run locally, the utility has no problems. However, when I run on the cluster, the utility likes to create "nan" entries in the pressure fields. The problem then being, when I run my solver, simpleFoam in my case, errors occur when the first "nan" entry is encountered. I get the error:

wrong token type - expected Scalar, found on line 3076 the word 'nan'

file: /home/mkimber/dtl5/confinementStudies/Re_32/R08_A2000_180/1500/p at line 3076.

From function operator>>(Istream&, Scalar&)
in file lnInclude/Scalar.C at line 91.

Its worth noting that I've only seen this error when running large meshes, approximately 1 million cells. Additionally, this is a low speed, laminar simulation, so the velocity in some locations is very small, approximately 10^(-6).

From what I've been able to gather, the reason I only have this problem on the cluster could be due to a discrepancy between the $FOAM_SETNAN entry I have locally and on the cluster, however, when I checked the setting on both, they were the same. As of this point, I am unsure what exactly I should do. I have previously just run mapFields locally and then uploaded the newly initialized time to SAM. However, this isn't a convenient way to do it, and will be extremely time consuming in the future when I have larger, more intensive simulations.


Any help or suggestions?

-Tyler

[wyldckat]

Hi Tyler,

OK, a few questions:

1. Which OpenFOAM version are you using?
2. How are you running the commands on the cluster?
3. You mention "FOAM_SETNAN"... do you mean that the variable is defined? The default value is to have it unset!
   Assuming you're using bash - what does this command output on each system:

   export | grep FOAM_SETNAN

   If it outputs nothing, then there shouldn't be any "nan" values when using mapFields, unless there is in fact an operation that leads to a "nan" value, which would be detected if "FOAM_SIGFPE" was set and should have triggered a crash...
4. Any chance you have a small example we can use for testing this as well? Perhaps something derived from an existing tutorial case?

Best regards,
Bruno

[tlandfried]

Morning Bruno,

Thanks for the reply. Sorry for the poor first post, you are right about a few things. First to answer your questions.

1. I run 2.1.0 locally, and 2.1.x on the cluster.

2. Sorry, not totally sure if I'm answering this fully but:

#(these two just load the openFoam module for me)  
module load openfoam/2.1.x-double-gcc45
. $OPENFOAM_RC

blockMesh
mapFields ../R08_A2000_080/ -consistent

I think this is pretty standard use. Except for loading the openFoam module, this is also the same way I run it on the local machine.

3. You are right, I meant to say that FOAM_SETNAN is unset. You are correct, and as you'd expect

export | grep FOAM_SETNAN

didn't return anything.

4. I do have a case, but its pretty large. As in, when compressed, the initialization and the actual test case are over a gig. I can make a simpler version, but am not sure if it will reproduce the error correctly, as I'm only seeing it when extrapolating to larger domains, or large number of elements. Let me think about this some more, as to how I could make a small example.

I do have a possible idea, the more that I'm thinking about the possible problems. So, since like you made clear, the issue is an operation actually producing the "nan" entry. What I'm doing, is taking a standard geometry, a round jet issuing axially into a round pipe, and changing the size of the confinement pipe.

So what I was trying to do, was initialize my simulation using results from a different confinement pipe case. And this seemed to work well, until I tried to do it for large meshes. Which, I suppose the issue is either the increased number of elements or perhaps the increased domain size (due to large diameter)?

Final thought, as I examine in much more detail when the "nan" entries occur, the values around it are very small, 10^(-300), or basically zero. This seems extremely low when considering machine precision.

[wyldckat]

Mmm... there is a rather big difference between 2.1.0 and 2.1.x. You might have gotten a commit from 2.1.x that was midway of a bug fix or an accidental bug might have been introduced. In the log of the applications you run on the cluster, you can see the version and commit hash number for the 2.1.x version you're using.

Originally Posted by tlandfried

Final thought, as I examine in much more detail when the "nan" entries occur, the values around it are very small, 10^(-300), or basically zero. This seems extremely low when considering machine precision.

I think the limit is around 10^(-308). This looks like a bad field initialization or something similar, like file/memory problems...

The simplest diagnosis might be the following steps:

1. Get the git commit hash number from 2.1.x on the cluster. Example of the top of a log file:
   

   /*---------------------------------------------------------------------------*\
   | =========                 |                                                 |
   | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
   |  \\    /   O peration     | Version:  2.1.x                                 |
   |   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
   |    \\/     M anipulation  |                                                 |
   \*---------------------------------------------------------------------------*/
   Build  : 2.1.x-334ec14f742f

   334ec14f742f is the hash number.
2. Build 2.1.x on your machine, but after you do "git clone" and before running Allwmake, run this command (example):
   

   git checkout 334ec14f742f

   Where the hexadecimal number is the hash number you got from your cluster's 2.1.x.

If you're using modules, then you probably know how to have more than one OpenFOAM installation in your machine

[wyldckat]

In case this doesn't do the trick, then creating a variant of this tutorial might be the best way to go: compressible/rhoCentralFoam/LadenburgJet60psi

[tlandfried]

Hm.. oddly enough when I check for the git commit hash it only says '2.1.x'

So.. I'm looking into this some more and will post when I know more.

EDIT:

So on the server and my local machine, I've tried 2.1.1, and an old git repository of 2.1.x. However, when reviewing the results, the "nan" entry only shows up on the supercomputer. Suggesting this is something involved with the settings that are different between the two.

What possibly could be causing this? I know were using two different compilers, gcc44 locally, gcc45 on the server. Possible? Not sure about this.

[tlandfried]

Hey all,

So just looking for more ideas as to what operation may be causing this problem on the cluster, but not locally.

I tried GCC-4.4.4 and GCC 4.5.1, but still had the same issue where mapFields is creating "nan" entries in my pressure field. Any ideas?

-Tyler

[wyldckat]

Hi Tyler,

I think I've forgotten to ask this before, but what operating system and architecture is your cluster using? Because that might influence how OpenFOAM is built, leading to these differences!

Additionally, are you able to create a simple test case, for example one based on the tutorial "compressible/rhoCentralFoam/LadenburgJet60psi"? This way it would be easier to diagnose the issue.

Best regards,
Bruno