[makaveli_lcf]

Hi all!

Today I faced a problem with the reconstruction of the parallel case with dynamic mesh refinement used in the solver.
First guess was that it is my solver's problem which could write not all the mesh data required for the reconstruction.
But first trial with the interDyMFoam in parallel (4 CPUs) indicated the same problem. Let's go into details.

So, simple run of the reconstructPar utility reports following symptoms:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.1.1                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.1.1-221db2718bbb
Exec   : reconstructPar
Date   : Jul 12 2012
Time   : 15:08:53
Host   : "pc169"
PID    : 28123
Case   : /media/Sim/alexanderv/New_CD_Labor/Term_03/VOF/DyM_Reconstruct/OFv211/damBreakWithObstacle
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Disallowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Time = 0.02



--> FOAM FATAL IO ERROR:
cannot find file

file:   /media/Sim/alexanderv/New_CD_Labor/Term_03/VOF/DyM_Reconstruct/OFv211/damBreakWithObstacle/processor0/0.02/polyMesh/pointProcAddressing  at line 0.

    From function regIOobject::readStream()
    in file db/regIOobject/regIOobjectRead.C at line 73.

FOAM exiting

Searching on the CFD-online gave tips and a way to solution:

1. Use reconstructParMesh for mesh reconstruction
2. Use reconstructPar for the data reconstruction

Step 1.
a) First problem is that reconstructParMesh uses reconstruction tolerance 1e-07 and suppose we used default values writing simulation data as well as mesh modifications in ASCII format with 1e-06 precision.
To override the defaults we use reconstructParMesh with -mergeTol 1e-06
b) Secondly reconstructParMesh during the run processes only one time step (correct?) either first or the lasr (wirh -latestTime flag). In any case I didn’t find automatic procession of all time dirs and -time <time> option works with the singe parameter only on the contrary to the reconstructPar -time <ranges>. So we will put it in a loop to reconstruct all time dirs.

Step 2.
Fails without clear reasons:

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.1.1                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.1.1-221db2718bbb
Exec   : reconstructPar
Date   : Jul 13 2012
Time   : 16:04:04
Host   : "pc169"
PID    : 19765
Case   : /media/Sim/alexanderv/New_CD_Labor/Term_03/VOF/DyM_Reconstruct/OFv211/damBreakWithObstacle_clean.run1
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Disallowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Time = 0.02

#0  Foam::error::printStack(Foam::Ostream&) in "/opt/openfoam211/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#1  Foam::sigSegv::sigHandler(int) in "/opt/openfoam211/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#2   in "/lib/libc.so.6"
#3  Foam::objectRegistry::checkOut(Foam::regIOobject&) const in "/opt/openfoam211/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#4  Foam::regIOobject::~regIOobject() in "/opt/openfoam211/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#5  Foam::IOList<int>::~IOList() in "/opt/openfoam211/platforms/linux64GccDPOpt/lib/libfiniteVolume.so"
#6  Foam::processorMeshes::read() in "/opt/openfoam211/platforms/linux64GccDPOpt/lib/libreconstruct.so"
#7  Foam::processorMeshes::readUpdate() in "/opt/openfoam211/platforms/linux64GccDPOpt/lib/libreconstruct.so"
#8  
 in "/opt/openfoam211/platforms/linux64GccDPOpt/bin/reconstructPar"
#9  __libc_start_main in "/lib/libc.so.6"
#10  
 in "/opt/openfoam211/platforms/linux64GccDPOpt/bin/reconstructPar"
Segmentation fault

Solution is to set “startFrom startTime;” in the controlDict and removing processor*/0 folders. In the attached script I backup them and then move back after data reconstruction.

So hereby I’ve attached a bash script for automatic reconstruction of e.g. interDyMFoam parallel case to avoid modified mesh isues.

#!/bin/bash


##  Merge tollerance (to override default value 1e-07 for ASCII output)
TOL=1e-06


##  Log file name
LOG='log.reconstructParMesh'


##  Start time folder name
START='0'


##  Check for empty time folders
TIME_DIRS=''
for proc_ in processor0/*; do
    dir_=`echo $proc_ | sed 's/processor0\///'`
    if [ ! -d $dir_ ]; then
        echo "Time $dir_ ist not reconstructed"
        ##  For empty list
        if [ ! -n "$TIME_DIRS" ]; then
            TIME_DIRS=$dir_        
        else
            TIME_DIRS="$TIME_DIRS $dir_"
        fi  
    fi
done


##  Exit if everything is reconstructed
if [ ! -n "$TIME_DIRS" ]; then
    echo -e "\nNothing to reconstruct! Exiting..."
    exit 0
else
    ##  Clean old log files
    rm -fv $LOG.*


    ##  Reconstruct mesh and data
    echo -e "Reconstructing mesh for:\n"

    ##  Mesh
    for dir_ in $TIME_DIRS; do
        echo -e "\tTime = $dir_ sec\n"
        reconstructParMesh -time $dir_ -mergeTol $TOL 2>&1 | tee $LOG.mesh
    done


    ##  Set startFrom to startTime
    sed -i "s/startFrom.*\;/startFrom    startTime\;/g" system/controlDict

    ##  Rename zero time folders in the processor directories
    for proc_ in processor*; do
        mv -fv $proc_/$START $proc_/$START.backup
    done

    ##  Data
    echo -e "Reconstructing data for times: $TIME_DIRS"
    reconstructPar -time `echo $TIME_DIRS | sed s/\ /,/g` 2>&1 | tee $LOG.data

    ##  Rename zero time folders backups in the processor directories
    for proc_ in processor*; do
        mv -fv $proc_/$START.backup $proc_/$START
    done
fi

It is possible to adjust the settings in the script for tolerance, start time and logfile name.

Here are some simulation and post-processing results:

https://docs.google.com/open?id=0BzC...1k5YmVTZDdOT1k

[linch]

Hi Alexander,

thanks for the script. Unfortunately, the data reconstruction doesn't work in my case (OF-2.1.x). I can't decrypt the error message:

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Time = 0.0230874

#0 Foam::error:rintStack(Foam::Ostream&) in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#1 Foam::sigSegv::sigHandler(int) in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#2 in "/lib/x86_64-linux-gnu/libc.so.6"
#3 Foam:bjectRegistry::checkOut(Foam::regIOobject&) const in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#4 Foam::regIOobject::~regIOobject() in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/libOpenFOAM.so"
#5 Foam::IOList<int>::~IOList() in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/libfiniteVolume.so"
#6 Foam:rocessorMeshes::read() in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/libreconstruct.so"
#7 Foam:rocessorMeshes::readUpdate() in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/lib/libreconstruct.so"
#8
in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/bin/reconstructPar"
#9 __libc_start_main in "/lib/x86_64-linux-gnu/libc.so.6"
#10
in "/home/shevchuk/OpenFOAM/OpenFOAM-2.1.x/platforms/linux64GccDPOpt/bin/reconstructPar"

Do you have any ideas?

Best regards,
Ilya

[linch]

Solved. the error was because of insufficient RAM.

[makaveli_lcf]

Effective)

[meng9019]

Hi Ilya,

How did you solve this problem?
I am running into the same problem now.
Thank you in advance.

Originally Posted by linch

Solved. the error was because of insufficient RAM.

[quarkz]

Hi makaveli_lcf, thanks for the script. It really worked! Btw, I'm using OF v2012.

[makaveli_lcf]

Hi! great that after 9 years it still works)

[Luke99]

also works for v2212

[Hypersonicc]

I used a simplified script command:
for dir in processor*; do [ -d "$dir" ] && find "$dir" -mindepth 1 -maxdepth 1 -type d -name '0.0012' -exec rm -rv {} \;; done

0.0012 is just an example time and can be replaced with other times. Actually, you don't need to make a backup because your initial file existed before partitioning, so I chose to delete it directly.

Anyway, thank you for your script. It still helps me 12 years later today!