CFD Online Logo CFD Online URL
www.cfd-online.com
[Sponsors]
Home > Forums > Software User Forums > OpenFOAM > OpenFOAM Programming & Development

Sharing large scalarList among multiple processors

Register Blogs Community New Posts Updated Threads Search

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
Old   August 25, 2017, 06:06
Default Sharing large scalarList among multiple processors
  #1
Member
 
hannes
Join Date: Mar 2013
Posts: 47
Rep Power: 13
hanness is on a distinguished road
Hi All,

I'm currently looking for a solution to efficiently share data in a parallel simulation.
The problem is as follows:
In a custom (temperature) boundary condition I want to interpolate the temperature based on several parameters. The data for the interpolation (list of scalars) is read from a file during initialisation of the bc. This list is rather large (36Mio values). My current implementation works fine, but it reads and stores this list for every thread which makes it very slow in the beginning and causes trouble with memory when running on 20 or more processors.

Instead, we thought of using a pointer to the list and reading the list only on the master processor but somehow I can't make the code run properly. And here comes the problem. The relevant constructor of the BC currently looks like this:
Code:
Foam::myTemperatureFvPatchScalarField::
myTemperatureFvPatchScalarField
(
    const fvPatch& p,
    const DimensionedField<scalar, volMesh>& iF,
    const dictionary& dict
)
:
    fixedValueFvPatchScalarField(p, iF),
    phiName_(dict.lookupOrDefault<word>("phi", "phi")),
    airTempIntTblName_(dict.lookup("airTempIntTblName")),
    airTempIntTbl_
    (
            airTempIntTblName_,
            this->patch().boundaryMesh().mesh(),
            IOdictionary
            (
                IOobject
                (
                    airTempIntTblName_,
                    this->patch().boundaryMesh().mesh().time().constant(),
                    this->patch().boundaryMesh().mesh(),
                    IOobject::MUST_READ,    // must exist, otherwise failure
                    IOobject::NO_WRITE,      // dict is only read by the solver
                    false // register
                )
            ),
            true //initialise only
    ),
    dataPointer_(),
    ...

{
    Pout << " in constructor " << endl;

    ...

    if (Pstream::master())
    {

        Pout << "in master loop ... " << endl;

        uniformMatrixInterpolationTable<scalar> foo
        (
        uniformMatrixInterpolationTable<scalar>
        (
            IOobject
            (
                airTempIntTblName_,
                this->patch().boundaryMesh().mesh().time().constant(),
                this->patch().boundaryMesh().mesh(),
                IOobject::MUST_READ,    // must exist, otherwise failure
                IOobject::NO_WRITE,      // dict is only read by the solver
                true
            )
        )
        );

        scalarList tblData=scalarList(foo.size(),Zero);
        scalarList *dataPtr;
        forAll(foo,i)
        {
            tblData[i]=foo[i];
        }
        dataPtr = &tblData;
        dataPointer_.set(dataPtr);
        Pout << "dataPtr " << *dataPtr << endl;

    }
//    Pstream::scatter(dataPointer_,Pstream::blocking);
}
dataPointer_ is defined in the respective .H file as
Code:
 
        autoPtr<scalarList> dataPointer_;
Now, although it compiles fine, there are several problems with this code. First of all, when running on a single thread it ends with the following error (after finishing the iterations):
Code:
End

*** Error in `buoyantSimpleFoam': double free or corruption (out): 0x00007ffd10596a30 ***
Aborted (core dumped)
When running on multiple threads (here two) it crashes with the following error message during initialisation ( - for debugging purposes the table was shortened to 32 entries):
Code:
[1]  in constructor
[0]  in constructor
[0] in master loop ...
[0] dataPtr 32{800}
[1]
[1]
[1] --> FOAM FATAL IO ERROR:
[1] incorrect first token, expected <int> or '(', found on line 0 the word 'dictionary'
[1]
[1] file: IOstream at line 0.
[1]
[1]     From function Foam::Istream& Foam::operator>>(Foam::Istream&, Foam::List<T>&) [with T = int]
[1]     in file /home/ubuntu/OpenFOAM/OpenFOAM-4.1/src/OpenFOAM/lnInclude/ListIO.C at line 148.
[1]
FOAM parallel run exiting
[1]
I thought this happens because the pointer is only available on the master and that I'd have to scatter the pointer to the other processors as well. But when uncommenting the Pstream line in the above code it won't compile:
Code:
g++ -std=c++0x -m64 -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -O3  -DNoRepository -ftemplate-depth-100 -I/opt/openfoam4/src/finiteVolume/lnInclude -I/opt/openfoam4/src/TurbulenceModels/incompressible/lnInclude -I/opt/openfoam4/src/TurbulenceModels/compressible/lnInclude -I/opt/openfoam4/src/turbulenceModels/compressible/lnInclude -I/opt/openfoam4/src/TurbulenceModels/turbulenceModels/lnInclude -I/opt/openfoam4/src/transportModels/compressible/lnInclude -I/opt/openfoam4/src/thermophysicalModels/basic/lnInclude  -IlnInclude -I. -I/opt/openfoam4/src/OpenFOAM/lnInclude -I/opt/openfoam4/src/OSspecific/POSIX/lnInclude   -fPIC -c myTemperatureFvPatchScalarField.C -o Make/linux64GccDPInt32Opt/myTemperatureFvPatchScalarField.o
In file included from /opt/openfoam4/src/OpenFOAM/lnInclude/Pstream.H:352:0,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/mapDistributeBase.H:75,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/Field.C:30,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/Field.H:405,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/labelField.H:39,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/primitiveFields.H:37,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/pointField.H:36,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/edge.H:40,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/edgeList.H:32,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/PrimitivePatch.H:56,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/primitivePatch.H:35,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/polyPatch.H:43,
                 from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatch.H:39,
                 from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchField.H:47,
                 from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchFields.H:29,
                 from myTemperatureFvPatchScalarField.H:55,
                 from myTemperatureFvPatchScalarField.C:26:
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C: In instantiation of ‘static void Foam::Pstream::scatter(const Foam::List<Foam::UPstream::commsStruct>&, T&, int, Foam::label) [with T = Foam::autoPtr<Foam::List<double> >; Foam::label = int]’:
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C:227:70:   required from ‘static void Foam::Pstream::scatter(T&, int, Foam::label) [with T = Foam::autoPtr<Foam::List<double> >; Foam::label = int]’
myTemperatureFvPatchScalarField.C:303:52:   required from here
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C:184:27: error: no match for ‘operator>>’ (operand types are ‘Foam::IPstream’ and ‘Foam::autoPtr<Foam::List<double> >’)
                 fromAbove >> Value;
                           ^
/opt/openfoam4/src/OpenFOAM/lnInclude/gatherScatter.C:184:27: note: candidates are:
In file included from /opt/openfoam4/src/OpenFOAM/lnInclude/string.H:48:0,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/word.H:42,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/wordList.H:41,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/patchIdentifier.H:38,
                 from /opt/openfoam4/src/OpenFOAM/lnInclude/polyPatch.H:42,
                 from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatch.H:39,
                 from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchField.H:47,
                 from /opt/openfoam4/src/finiteVolume/lnInclude/fvPatchFields.H:29,
                 from myTemperatureFvPatchScalarField.H:55,
                 from myTemperatureFvPatchScalarField.C:26:
/opt/openfoam4/src/OpenFOAM/lnInclude/char.H:49:10: note: Foam::Istream& Foam::operator>>(Foam::Istream&, char&)
 Istream& operator>>(Istream&, char&);
...
So, I've got a few questions.
1. Is the approach using pointers in this case reasonable or is there a better alternative?
2. Is there anything wrong in the way the pointer is used. (see the error message for a single thread run above)
3. Why doesn't it run in parallel?

I really appreciate any comments and thoughts regarding the above and thanks a lot!
Hannes
hanness is offline   Reply With Quote

Old   August 28, 2017, 04:51
Default
  #2
New Member
 
Join Date: Dec 2012
Posts: 13
Rep Power: 14
Vesposo is on a distinguished road
Hi,

I'm struggeling with a similar question and would appreciate any hints on scattering a smart pointer over processors while only the master initializes it.
Vesposo is offline   Reply With Quote

Reply

Tags
autoptr, parallel, pstreams


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Calculate field and patch averages in multiple zones, large mesh buffi OpenFOAM Post-Processing 2 July 21, 2022 06:45
Socket 2011-3 processors - an overwiew flotus1 Hardware 10 June 17, 2017 10:47
timeVaryingMappedFixedValue: large data mapping on complex inlet boundaries vkrastev OpenFOAM Pre-Processing 7 June 2, 2016 16:50
Large Epsilon 3D lepsilon Main CFD Forum 0 July 10, 2012 11:43
64-bit processors for home computing Ananda Himansu Main CFD Forum 2 March 16, 2004 13:48


All times are GMT -4. The time now is 21:14.