[bastil]

Hi all,

I want to try to switch on floatTransfer for paralle calculations with 1.5-dev. I tried creating ~/.OpenFOAM-1.5-dev/controlDict and set it to 1 there - without effect.

Where does the local controlDict have to be placed?
~/.OpenFOAM-1.5-dev/etc/controlDict)
~/OpenFOAM/.OpenFOAM-1.5-dev/controlDict?
Is it dependent on Installation location?

Thanks.

[l_r_mcglashan]

The controlDict is in:

OpenFOAM-1.5-dev\etc

[lin]

~/OpenFOAM/OpenFOAM-dev/etc/controlDict

[bastil]

Thanks, I will try this out. Documentation of this is not very accurate.

Regards BastiL

[olesen]

Originally Posted by bastil

I want to try to switch on floatTransfer for paralle calculations with 1.5-dev. I tried creating ~/.OpenFOAM-1.5-dev/controlDict and set it to 1 there - without effect.

From my (bad) experience, I would advise against using floatTransfer=1.
In some cases it is fine, but for a few particular problems (and perhaps particular decompositions thereof) I think rounding error issues become a problem. In my case, the solution ran well for 800-1000 iter or so and then the hit a very strange spike in the convergence behaviour.

Where does the local controlDict have to be placed?
~/.OpenFOAM-1.5-dev/etc/controlDict
~/OpenFOAM/.OpenFOAM-1.5-dev/controlDict?

None of the above. The main controlDict is searched using the "~OpenFOAM/" text expansion semantics mentioned in the docs:

//- Search for @em name in the following hierarchy:
//  -# personal settings:
//    - ~/.OpenFOAM/\<VERSION\>/
//      <em>for version-specific files</em>
//    - ~/.OpenFOAM/
//      <em>for version-independent files</em>
//  -# site-wide settings:
//    - $WM_PROJECT_INST_DIR/site/\<VERSION\>
//      <em>for version-specific files</em>
//    - $WM_PROJECT_INST_DIR/site/
//      <em>for version-independent files</em>
//  -# shipped settings:
//    - $WM_PROJECT_DIR/etc/

You probably remember using something like ~/.OpenFOAM-<VERSION>, but this was for older versions.

Note: unlike other dictionaries, do not attempt to use #include or other directives within the controlDict. It is read before these mechanisms are available within OpenFOAM.

/mark

[bastil]

Thanks Mark,

I found this code in src/OpenFOAM/include/OSspecific.H. Do I have to uncomment some of it? And where do I finally have to place my file to?
The reason why I would like to try floatTransfer is very poor scaling of OF on our Myrinet-Cluster. It seems there is nearly no speedup between 16 and 64 cores - ths behaviour is completely different to commerical codes.

[olesen]

Originally Posted by bastil

I found this code in src/OpenFOAM/include/OSspecific.H. Do I have to uncomment some of it?

No.

And where do I finally have to place my file to?

~/.OpenFOAM/1.5.x/etc/controlDict would be a good place.
Then set floatTransfer 1; in that file

[bastil]

Thanks. However, that does not seem to be much quicker at all.... I have no idea:

16 Cores:
Time = 250

PBiCG: Solving for Ux, Initial residual = 0.000399256, Final residual = 3.3002e-05, No Iterations 2
PBiCG: Solving for Uy, Initial residual = 0.00462622, Final residual = 0.000422648, No Iterations 2
PBiCG: Solving for Uz, Initial residual = 0.00475788, Final residual = 0.000443842, No Iterations 1
GAMG: Solving for p, Initial residual = 0.0483258, Final residual = 0.000298896, No Iterations 3
time step continuity errors : sum local = 0.000588824, global = 1.06764e-05, cumulative = -0.000579853
PBiCG: Solving for epsilon, Initial residual = 4.90512e-05, Final residual = 1.28975e-07, No Iterations 1
bounding epsilon, min: -0.403354 max: 6.75487e+08 average: 41716.5
PBiCG: Solving for k, Initial residual = 5.70964e-05, Final residual = 2.91034e-07, No Iterations 1
ExecutionTime = 26228.3 s ClockTime = 26328 s

64 Cores:
Time = 250

PBiCG: Solving for Ux, Initial residual = 0.000395327, Final residual = 1.16597e-05, No Iterations 3
PBiCG: Solving for Uy, Initial residual = 0.00493349, Final residual = 0.000152124, No Iterations 3
PBiCG: Solving for Uz, Initial residual = 0.00514608, Final residual = 0.000143217, No Iterations 3
GAMG: Solving for p, Initial residual = 0.0501587, Final residual = 0.000316504, No Iterations 4
time step continuity errors : sum local = 0.000624036, global = -1.80558e-05, cumulative = -0.00034109
PBiCG: Solving for epsilon, Initial residual = 7.91971e-05, Final residual = 6.22651e-07, No Iterations 1
bounding epsilon, min: -0.42092 max: 1.81212e+09 average: 49460.6
PBiCG: Solving for k, Initial residual = 9.47916e-05, Final residual = 1.49895e-06, No Iterations 1
ExecutionTime = 24337.1 s ClockTime = 24453 s

This is a Myrinet CLuster, I use OpenMPI 1.26 as with OF1.5 compiled with mx turned on and I use the --mca mtl Option. Ideas?