[Daniel_Khazaei]

Hi foamers,

This piece of code does compile and run in serial mode normally, however when I try to run it in parallel it compiles successfully (although it takes much longer to compile) it only create the output file but it doesn't write anything into the file!

Perhaps the code is not ready for parallel? where do I need to modify to make it work?

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | foam-extend: Open Source CFD                    |
|  \\    /   O peration     | Version:     4.0                                |
|   \\  /    A nd           | Web:         http://www.foam-extend.org         |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application     interfsiFoam;

startFrom       startTime;

startTime       0;

stopAt          endTime;

endTime         60;

deltaT          1e-5;

writeControl    adjustableRunTime;

writeInterval   0.02;

purgeWrite      0;

writeFormat     ascii;

writePrecision  6;

writeCompression off;

timeFormat      general;

timePrecision   7;

runTimeModifiable yes;

adjustTimeStep  yes;

maxCo           0.2;

maxDeltaT       1e-3;

maxAlphaCo      0.2;

maxFourier    1.0;

InfoSwitches
{
    allowSystemOperations 1;
}

//- real-time temperature monitoring utility
functions
(
    temperatureRange
    {
    functionObjectLibs ("libutilityFunctionObjects.so");
    type         coded;
    redirectType       temperatureRange;
    outputControl      timeStep;
    outputInterval     10;
    code           
    #{
        //- Get temperture field
        const volScalarField& T = mesh().lookupObject<volScalarField>("T");

        //- Get simulation time
        scalar t = mesh().time().value();

        //- Now printing out the results
        if(Pstream::master())
        {
            std::ofstream fs;
            fs.open ("temperatureRange.dat", std::fstream::app);
            fs.precision(12);
            fs << t << "\t" << min(T).value() << "\t" << max(T).value() <<"\n";
            fs.close();
        }

    #};

    codeInclude
    #{
        #include <fstream>
    #};
    }
);

// ************************************************************************* //

Regards,
D.Khazaei

[usv001]

Hello Daniel,

I am facing the same problem. My coded function object compiles and runs properly in serial but it is completely ignored (not even compiled!) in parallel. Did you manage to find a solution? I am using OpenFOAM v2.4 by the way.

USV

[Daniel_Khazaei]

Originally Posted by usv001

Hello Daniel,

I am facing the same problem. My coded function object compiles and runs properly in serial but it is completely ignored (not even compiled!) in parallel. Did you manage to find a solution? I am using OpenFOAM v2.4 by the way.

USV

unfortunately I haven't found any solution to this problem yet...I'm using foam-extend-4.0!
mine does compile but as I said it takes much longer and only creates the output file without writing anything into it!

probably @wyldckat may be able to help us if he has time

[Daniel_Khazaei]

anybody can help us regarding this problem?

[olesen]

I don't have any experience with extend, but are the min()/max() functions there meant for parallel or serial?


If they are parallel operations, I'd think they'd block indefinitely since you have them only being invoked on the master process.



If that doesn't work for you. I'd suggest grabbing some other version (1806, 1812,..) and trying exactly the same bit of coding on one of the tutorials.


Another option. Try just writing out some the time (without min/max) in your file. If this produces content for you, you've found the problem.


/mark

[Daniel_Khazaei]

Originally Posted by olesen

I don't have any experience with extend, but are the min()/max() functions there meant for parallel or serial?

If they are parallel operations, I'd think they'd block indefinitely since you have them only being invoked on the master process.

If that doesn't work for you. I'd suggest grabbing some other version (1806, 1812,..) and trying exactly the same bit of coding on one of the tutorials.

Another option. Try just writing out some the time (without min/max) in your file. If this produces content for you, you've found the problem.

/mark

thank you very much mark using your suggestions, I have tried to run the case without min/max and writing only t and dt values in parallel and it worked! then I changed the code as follow and everything is OK now:

//- real-time temperature monitoring utility
functions
(
    temperatureRange
    {
    functionObjectLibs ("libutilityFunctionObjects.so");
    type         coded;
    redirectType       temperatureRange;
    outputControl      timeStep;
    outputInterval     50;
    code           
    #{
        //- Get temperture field
        const volScalarField& T = mesh().lookupObject<volScalarField>("T");

        //- Get velocity vector field
        const volVectorField& U = mesh().lookupObject<volVectorField>("U");
        const volScalarField magU = mag(U);

        //- Get simulation time
        scalar t = mesh().time().value();
        scalar dt = mesh().time().deltaTValue();

        //- Get the bounds
        scalar maxMagU = max(magU).value();
        scalar maxT = max(T).value();
        scalar minT = min(T).value();

        //- Now printing out the results
        if(Pstream::master())
        {
        std::ofstream fs;
        fs.open ("temperatureRange.dat", std::fstream::app);
        fs.precision(12);
        fs << t << "\t" << dt << "\t" << maxMagU << "\t"
           << minT << "\t" << maxT << "\n";
        fs.close();
        }

    #};

    codeInclude
    #{
        #include <fstream>
    #};
    }
);

From past I thought that gMax and gMin are meant for parallel and min/max should work in serial! The only problem now is that the compilation process takes much longer when I run the case in parallel!

Regards,
Daniel

[wyldckat]

Originally Posted by Daniel_Khazaei

The only problem now is that the compilation process takes much longer when I run the case in parallel!

Quick questions: Can you provide the Allrun complete script you are using? Or at least indicate which exact commands you are running in the command line?

[Daniel_Khazaei]

Originally Posted by wyldckat

Quick questions: Can you provide the Allrun complete script you are using? Or at least indicate which exact commands you are running in the command line?

Dear Bruno,
Here is the AllrunPar script I'm using for this case:

#!/bin/sh

# Source tutorial run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions

# Get application name
application=`getApplication`

mkdir -p ../solid/0
cp -r ../solid/orig/* ../solid/0/

runApplication -l log.fluent3DMeshToFoam.solid fluent3DMeshToFoam -case ../solid/ ../solid/solid.msh
runApplication -l log.changeDictionary.solid changeDictionary -case ../solid
runApplication -l log.renumberMesh.solid renumberMesh -overwrite -case ../solid
runApplication -l log.setSet.solid setSet -case ../solid -batch ../solid/setBatch
runApplication -l log.setsToZones.solid setsToZones -case ../solid -noFlipMap
runApplication -l log.setFields.solid setFields -case ../solid
runApplication -l log.decomposePar.solid decomposePar -case ../solid

mkdir -p 0
cp -r orig/* 0/

runApplication fluent3DMeshToFoam fluid.msh
runApplication changeDictionary
runApplication renumberMesh -overwrite
runApplication setSet -batch setBatch
runApplication setsToZones -noFlipMap
runApplication setFields
runApplication decomposePar

cd ..

./makeLinks fluid solid

cd fluid

runParallel $application 2
runApplication reconstructPar
runApplication -l log.reconstructPar.solid reconstructPar -region solid

# ----------------------------------------------------------------- end-of-file

[wyldckat]

Hi Daniel,

Something doesn't sound right here... a few more questions:

1. Which version of foam-extend are you using?
2. Is it really 4.0 or from the "nextRelease" branch on the 4.0 repository?

Because I believe that decomposePar should have compiled the function object, well before the solver was launched in parallel...


The only reason for it to be slower in compiling when running in parallel, is if:

• the function object either triggers the build on all cores, hence taking X times longer...
• or because the other slave processes are waiting for the master process to be done with the build, but they might hog the CPU during that wait, for whatever reason...

Either way, this might be something you should report here: https://sourceforge.net/p/foam-extend/tickets/

Best regards,
Bruno

[Daniel_Khazaei]

Originally Posted by wyldckat

Hi Daniel,

Something doesn't sound right here... a few more questions:

1. Which version of foam-extend are you using?
2. Is it really 4.0 or from the "nextRelease" branch on the 4.0 repository?

Because I believe that decomposePar should have compiled the function object, well before the solver was launched in parallel...

This actually is a foam-extend-4.0 but I have back-ported a few functionalities from OpenFOAM-v1806 and the nextRelease branch of foam-extend...I have also upgraded a few libraries to the recent changes in nextRelease branch of foam-extend including bugfixes...

Originally Posted by wyldckat

The only reason for it to be slower in compiling when running in parallel, is if:

• the function object either triggers the build on all cores, hence taking X times longer...
• or because the other slave processes are waiting for the master process to be done with the build, but they might hog the CPU during that wait, for whatever reason...

Either way, this might be something you should report here: https://sourceforge.net/p/foam-extend/tickets/

Best regards,
Bruno

Well, I will try to check what actually causing the delay! if I wasn't able to figure it out, I will report the problem there.

thank you very much Bruno

Regards,
Daniel

[wyldckat]

Originally Posted by Daniel_Khazaei

This actually is a foam-extend-4.0 but I have back-ported a few functionalities from OpenFOAM-v1806 and the nextRelease branch of foam-extend...I have also upgraded a few libraries to the recent changes in nextRelease branch of foam-extend including bugfixes...

Oooohhh... OK... so you've opened up yourself to introducing bugs of your own!
In that case, you must first test if this also occurs with foam-extend 4.1 (4.0-nextRelease) or not, just to ensure that this wasn't something you introduced yourself

Additionally, you might want to hack into the code that handles the calls to wmake for the coded function object and use "Pout" instead of "Info", to make it output the messages not only for the master process, but also the slave ones.

[Daniel_Khazaei]

Originally Posted by wyldckat

Oooohhh... OK... so you've opened up yourself to introducing bugs of your own!
In that case, you must first test if this also occurs with foam-extend 4.1 (4.0-nextRelease) or not, just to ensure that this wasn't something you introduced yourself

Additionally, you might want to hack into the code that handles the calls to wmake for the coded function object and use "Pout" instead of "Info", to make it output the messages not only for the master process, but also the slave ones.

Dear Brouno,

I have just tried to follow your suggestion and installed fresh copy of foam-extend-4.1 (nextRelease branch)! The behavior is the same...so the bug is not mine what I have forgot to say before is that the compilation process itself runs just fine, however the solver waits for a long time after the

'....so' is up to date

message.

Here is a sample log:

Selecting turbulence model type RASModel
Selecting RAS turbulence model kEpsilon
kEpsilonCoeffs
{
    Cmu             0.09;
    C1              1.44;
    C2              1.92;
    C3              -0.33;
    sigmak          1;
    sigmaEps        1.3;
    Prt             1;
}

Creating field DpDt


Starting time loop

Using dynamicCode for functionObject temperatureRange at line 70 in "::temperatureRange"
Creating new library in "dynamicCode/temperatureRange/platforms/linux64GccDPOpt/lib/libtemperatureRange_960cb3ad9b821b535c5532223dd972a7d13833d0.so"
Invoking "wmake -s libso /home/shadowfax/foam/shadowfax-4.1/run/tutorials/compressible/rhoPimpleFoam/angledDuct/dynamicCode/temperatureRange"
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file functionObjectTemplate.C
Making dependency list for source file FilterFunctionObjectTemplate.C
SOURCE=functionObjectTemplate.C ;  g++ -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-200 -g -I/home/shadowfax/foam/foam-extend-4.1/src/finiteVolume/lnInclude -I/home/shadowfax/foam/foam-extend-4.1/src/meshTools/lnInclude  -IlnInclude -I. -I/home/shadowfax/foam/foam-extend-4.1/src/foam/lnInclude -I/home/shadowfax/foam/foam-extend-4.1/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64GccDPOpt/functionObjectTemplate.o
SOURCE=FilterFunctionObjectTemplate.C ;  g++ -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-200 -g -I/home/shadowfax/foam/foam-extend-4.1/src/finiteVolume/lnInclude -I/home/shadowfax/foam/foam-extend-4.1/src/meshTools/lnInclude  -IlnInclude -I. -I/home/shadowfax/foam/foam-extend-4.1/src/foam/lnInclude -I/home/shadowfax/foam/foam-extend-4.1/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64GccDPOpt/FilterFunctionObjectTemplate.o
'/home/shadowfax/foam/shadowfax-4.1/run/tutorials/compressible/rhoPimpleFoam/angledDuct/dynamicCode/temperatureRange/../platforms/linux64GccDPOpt/lib/libtemperatureRange_960cb3ad9b821b535c5532223dd972a7d13833d0.so' is up to date.

then after abnormally longer period the actual simulation starts:

Courant Number mean: 0 max: 0 velocity magnitude: 0
Time = 1

diagonal:  Solving for rho, Initial residual = 0, Final residual = 0, No Iterations 0
PIMPLE: iteration 1
BiCGStab:  Solving for Ux, Initial residual = 1, Final residual = 0.00374194, No Iterations 1
BiCGStab:  Solving for Uy, Initial residual = 1, Final residual = 0.00575923, No Iterations 1
BiCGStab:  Solving for Uz, Initial residual = 1, Final residual = 0.00920929, No Iterations 1
BiCGStab:  Solving for h, Initial residual = 1, Final residual = 0.00613828, No Iterations 1
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.00996037, No Iterations 82
diagonal:  Solving for rho, Initial residual = 0, Final residual = 0, No Iterations 0
time step continuity errors : sum local = 1.12402, global = -0.0112633, cumulative = -0.0112633
rho max/min : 1.18758 1.1863
BiCGStab:  Solving for epsilon, Initial residual = 0.898069, Final residual = 2.79157e-06, No Iterations 4
BiCGStab:  Solving for k, Initial residual = 1, Final residual = 5.4789e-06, No Iterations 2
PIMPLE: iteration 2
BiCGStab:  Solving for Ux, Initial residual = 0.680134, Final residual = 0.0040336, No Iterations 1
BiCGStab:  Solving for Uy, Initial residual = 0.288343, Final residual = 0.00156818, No Iterations 1
BiCGStab:  Solving for Uz, Initial residual = 0.721388, Final residual = 0.00352804, No Iterations 1
BiCGStab:  Solving for h, Initial residual = 1, Final residual = 0.0063911, No Iterations 1
DICPCG:  Solving for p, Initial residual = 0.501305, Final residual = 0.00459142, No Iterations 83
diagonal:  Solving for rho, Initial residual = 0, Final residual = 0, No Iterations 0
time step continuity errors : sum local = 1.21116, global = 0.00796014, cumulative = -0.00330312
rho max/min : 1.20425 1.1863
BiCGStab:  Solving for epsilon, Initial residual = 0.363985, Final residual = 8.11722e-07, No Iterations 6
BiCGStab:  Solving for k, Initial residual = 0.221285, Final residual = 3.71386e-06, No Iterations 2
PIMPLE: iteration 3
BiCGStab:  Solving for Ux, Initial residual = 0.781972, Final residual = 0.00337158, No Iterations 1
BiCGStab:  Solving for Uy, Initial residual = 0.723679, Final residual = 0.00315982, No Iterations 1
BiCGStab:  Solving for Uz, Initial residual = 0.613821, Final residual = 0.00339603, No Iterations 1
BiCGStab:  Solving for h, Initial residual = 0.889189, Final residual = 0.00303323, No Iterations 1
DICPCG:  Solving for p, Initial residual = 0.106732, Final residual = 0.000885835, No Iterations 81
diagonal:  Solving for rho, Initial residual = 0, Final residual = 0, No Iterations 0
time step continuity errors : sum local = 2.89355, global = -0.0584832, cumulative = -0.0617864
rho max/min : 1.21736 1.1863
BiCGStab:  Solving for epsilon, Initial residual = 0.306246, Final residual = 7.63149e-06, No Iterations 5
BiCGStab:  Solving for k, Initial residual = 0.178564, Final residual = 6.25217e-06, No Iterations 2
PIMPLE: iteration 4
BiCGStab:  Solving for Ux, Initial residual = 0.609119, Final residual = 0.00447596, No Iterations 1
BiCGStab:  Solving for Uy, Initial residual = 0.303115, Final residual = 0.00142949, No Iterations 1
BiCGStab:  Solving for Uz, Initial residual = 0.221475, Final residual = 0.00141429, No Iterations 1
BiCGStab:  Solving for h, Initial residual = 0.659225, Final residual = 0.00407236, No Iterations 1
DICPCG:  Solving for p, Initial residual = 0.136302, Final residual = 0.00117668, No Iterations 81
diagonal:  Solving for rho, Initial residual = 0, Final residual = 0, No Iterations 0
time step continuity errors : sum local = 4.01168, global = -0.0285645, cumulative = -0.0903508
rho max/min : 1.21538 1.18647

I don't know whether this is expected behavior or not when running in parallel, perhaps I should also check with OpenFOAM solvers for this. The solver is rhoPimpleFoam form foam-extend-4.1! The modified test case tutorial is also attached.

Best Regards and Happy new year
Daniel

[wyldckat]

Quick answer: I haven't built foam-extend 4.1 (nextRelease) yet and don't have plans to do so.
So either someone else here on the forum picks this up, or it's better for you to simply report this on the foam-extend bug tracker.

Happy New Year, may 2019 be even more foamy...