Using PETSc with OpenFoam

palarcon · July 19, 2018, 10:11

Hello everybody

I'm trying to include PETSc into my OpenFOAM code in order to be able to perform some topology optimization developments (mainly becase there is a version of the MMA optimizer already coded with PETSc).

So far, I have been unable to include petsc.h, even if I added the directories and the libraries in the option file (but is pretty sure that I make a mistake).

Does anybody has experience doing this kind of linking?

I know that there is an ancient post related to the subject, but the answer was about an ancient version of PETSc and the person who posed the question, was not able to solve the problem.

This is my options file

Code:

EXE_INC = \
    -I$(LIB_SRC)/TurbulenceModels/turbulenceModels/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/incompressible/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/compressible/lnInclude \
    -I$(LIB_SRC)/transportModels \
    -I$(LIB_SRC)/transportModels/incompressible/singlePhaseTransportModel \
    -I$(LIB_SRC)/finiteVolume/lnInclude \
    -I$(LIB_SRC)/meshTools/lnInclude \
	-I$(PETSC_DIR}/include/lnInclude \
	-I$(PETSC_DIR}/$(PETSC_ARCH)/include/lnInclude \

EXE_LIBS = \
    -lturbulenceModels \
    -lincompressibleTurbulenceModels \
    -lincompressibleTransportModels \
    -lfiniteVolume \
    -lmeshTools \
    -lfvOptions \
	-L$(PETSC_DIR)/lib/petsc/conf \
	-L$(PETSC_LIBDIR)

having that and adding to my main file #include "petsc.h" I get the error

fatal error: petsc.h: No such file or directory
compilation terminated.

The suggestion that was given a long time ago is

Code:

include $(RULES)/mplib$(WM_MPLIB)

EXE_INC = \
     $(PFLAGS) $(PINC) \
    -I$(PETSC_DIR)/include -I$(PETSC_DIR)/$(PETSC_ARCH)/include \
    -I$(LIB_SRC)/finiteVolume/lnInclude

EXE_LIBS = \
    -lfiniteVolume \
    -L$(PETSC_LIBDIR) -lpetsc -lpetscmat -lpetscvec -lpetscksp -lpetscdm

I tried adding the libraries -lpetsc with no luck, so I'm completely lost.
If anybody has a way to include PETSc into OpenFOAM (the simples way will do the job actually) I will be really thankful.

akidess · July 19, 2018, 10:48

The library entries are completely irrelevant at this point, don't bother. You need to make sure that you get your includes right. You have obvious errors in your bracketing, and I simply doubt you're picking up the right directories for petsc. I'd recommend hard coding the include directory for petsc.h for starters.

palarcon · July 19, 2018, 11:18

I did the hard coding has suggested and now, even if recognize the file petsc.h, once inside that file, doesn't recognize another .h file that is called inside that one, even if I added the directories in this way.

Code:

EXE_INC = \
    -I$(LIB_SRC)/TurbulenceModels/turbulenceModels/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/incompressible/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/compressible/lnInclude \
    -I$(LIB_SRC)/transportModels \
    -I$(LIB_SRC)/transportModels/incompressible/singlePhaseTransportModel \
    -I$(LIB_SRC)/finiteVolume/lnInclude \
    -I$(LIB_SRC)/meshTools/lnInclude \
	-I/home/pablo/PETSC/3.7.4/include/lnInclude \
	-I/home/pablo/PETSC/3.7.4/include/petsc/finclude/lnInclude \
	-I/home/pablo/PETSC/3.7.4/linux-dbg/include/lnInclude \

EXE_LIBS = \
    -lturbulenceModels \
    -lincompressibleTurbulenceModels \
    -lincompressibleTransportModels \
    -lfiniteVolume \
    -lmeshTools \
    -lfvOptions \
	-L/home/pablo/PETSC/3.7.4/lib/petsc/conf \
	-L/home/pablo/PETSC/3.7.4/linux-dbg/lib

olesen · July 19, 2018, 19:28

Quote:

Originally Posted by palarcon

I did the hard coding has suggested and now, even if recognize the file petsc.h, once inside that file, doesn't recognize another .h file that is called inside that one, even if I added the directories in this way.

Code:

EXE_INC = \
    -I$(LIB_SRC)/TurbulenceModels/turbulenceModels/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/incompressible/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/compressible/lnInclude \
    -I$(LIB_SRC)/transportModels \
    -I$(LIB_SRC)/transportModels/incompressible/singlePhaseTransportModel \
    -I$(LIB_SRC)/finiteVolume/lnInclude \
    -I$(LIB_SRC)/meshTools/lnInclude \
    -I/home/pablo/PETSC/3.7.4/include/lnInclude \
    -I/home/pablo/PETSC/3.7.4/include/petsc/finclude/lnInclude \
    -I/home/pablo/PETSC/3.7.4/linux-dbg/include/lnInclude \

EXE_LIBS = \
    -lturbulenceModels \
    -lincompressibleTurbulenceModels \
    -lincompressibleTransportModels \
    -lfiniteVolume \
    -lmeshTools \
    -lfvOptions \
    -L/home/pablo/PETSC/3.7.4/lib/petsc/conf \
    -L/home/pablo/PETSC/3.7.4/linux-dbg/lib

I guess that your petsc installation was using the standard PETSc configure/install. In which case you won't have the wmake-style 'lnInclude' at all, but just the plain include directory. Eg,

Code:

 

    -I/home/pablo/PETSC/3.7.4/include \

If you have any code to share (quality doesn't matter much), I'd appreciate taking a look at it as well.

/mark

palarcon · July 20, 2018, 04:35

Hello and thank you all for the detailed answers.
I'm using the standard adjointshapeoptimizationfoam, where in addition to separate in different files the computation of the primal and dual velocities/pressures without any problem (to make it easier to read), I added an #include to call petsc.h (I will put my modifications in colors).
Also, the PETSc installation was done using the standard PETSc configure/install parameters (in a PETSc test case I compiled using the standard make).

At first my main code looked like this

Code:

#include "fvCFD.H"
#include "singlePhaseTransportModel.H"
#include "turbulentTransportModel.H"
#include "simpleControl.H"
#include "fvOptions.H"
#include "petsc.h"

template<class Type>
void zeroCells
(
    GeometricField<Type, fvPatchField, volMesh>& vf,
    const labelList& cells
)
{
    forAll(cells, i)
    {
        vf[cells[i]] = Zero;
    }
}


// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

int main(int argc, char *argv[])
{
    #include "postProcess.H"

    #include "setRootCase.H"
    #include "createTime.H"
    #include "createMesh.H"
    #include "createControl.H"
    #include "createFields.H"
    #include "createFvOptions.H"
    #include "initContinuityErrs.H"
    #include "initAdjointContinuityErrs.H"

    turbulence->validate();

    // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

    Info<< "\nStarting time loop\n" << endl;

    while (simple.loop())
    {
        Info<< "Time = " << runTime.timeName() << nl << endl;

		#include "updatePorosityField.H"
		
        // Pressure-velocity SIMPLE corrector
        {
			#include "UEqn.H"
			#include "pEqn.H"			
        }

        // Adjoint Pressure-velocity SIMPLE corrector
        {
			#include "UaEqn.H"
			#include "paEqn.H"
        }

        laminarTransport.correct();
        turbulence->correct();
		
		sens=Ua&U;
		
		#include "costFunction.H"

        runTime.write();

        Info<< "ExecutionTime = "
            << runTime.elapsedCpuTime()
            << " s\n\n" << endl;
    }

    Info<< "End\n" << endl;

    return 0;
}

and the options file like this

Code:

EXE_INC = \
    -I$(LIB_SRC)/TurbulenceModels/turbulenceModels/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/incompressible/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/compressible/lnInclude \
    -I$(LIB_SRC)/transportModels \
    -I$(LIB_SRC)/transportModels/incompressible/singlePhaseTransportModel \
    -I$(LIB_SRC)/finiteVolume/lnInclude \
    -I$(LIB_SRC)/meshTools/lnInclude \
	-I$(PETSC_DIR}/include/lnInclude \
	-I$(PETSC_DIR}/$(PETSC_ARCH)/include/lnInclude \

EXE_LIBS = \
    -lturbulenceModels \
    -lincompressibleTurbulenceModels \
    -lincompressibleTransportModels \
    -lfiniteVolume \
    -lmeshTools \
    -lfvOptions \
	-L$(PETSC_DIR)/lib/petsc/conf \
	-L$(PETSC_LIBDIR)

Doing that I got the error fatal error: petsc.h: No such file or directory compilation terminated.

That was because I was sloppy with the parenthesis on the options file, but following a subsequent advice I did the hard coding for the petsc.h call and to include the folders in the options file.

After that, my petsc.h call was done like this on the main code

Code:

#include "/home/pablo/PETSC/3.7.4/include/petsc.h"

and the options file looks like this now (I was able to correctly write that using environmental variables)

Code:

EXE_INC = \
    -I$(LIB_SRC)/TurbulenceModels/turbulenceModels/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/incompressible/lnInclude \
    -I$(LIB_SRC)/TurbulenceModels/compressible/lnInclude \
    -I$(LIB_SRC)/transportModels \
    -I$(LIB_SRC)/transportModels/incompressible/singlePhaseTransportModel \
    -I$(LIB_SRC)/finiteVolume/lnInclude \
    -I$(LIB_SRC)/meshTools/lnInclude \
    -I$(PETSC_DIR)/include \
    -I$(PETSC_DIR)/$(PETSC_ARCH)/include \
    -I$(PETSC_DIR)/include/petsc/finclude \


EXE_LIBS = \
    -lturbulenceModels \
    -lincompressibleTurbulenceModels \
    -lincompressibleTransportModels \
    -lfiniteVolume \
    -lmeshTools \
    -lfvOptions \
    -L/home/pablo/PETSC/3.7.4/lib/petsc/conf \
    -L/home/pablo/PETSC/3.7.4/linux-dbg/lib

Now, the main code is able to find the petsc.h file and all the subsequent .h files, but stop again now with an error related to the language itself related to the "pow" operation (I guess that linking OpenFOAM with PETSc language is not as straightforward).

/home/pablo/PETSC/3.7.4/include/petscmath.h:326:38: error: call of overloaded ‘pow(PetscScalar&, PetscScalar&)’ is ambiguous
#define PetscPowScalar(a,b) pow(a,b)

Any idea on how solve this last problem related to the "pow" operator?

palarcon · July 23, 2018, 09:48

Problem solved

Given the fact that OpenFOAM and PETSc have a definition for the pow operator, this one had an ambiguous definition.
To solve it, I had to add "::" in the pow definition inside the PETSc file, in order to avoir the confusion on the definition.

nuaawqf · December 28, 2018, 04:45

Hi Pablo:

I am interesting in how you using the MMA in PETSc to manipulate the OpenFOAM fields such as alpha. Can you show some hints in your codes?
Thanks much.

palarcon · January 6, 2019, 09:35

Hello and sorry for the late answer.
Finally, and for a number of reasons (mainly the fact that I couldn't find a reliably C++ version of MMA, even if there are) I declined using, at least for the moment, MMA with PETSc.
At the time I performed a lot of test case and I wasn't happy with the results.
Sorry!

July 19, 2018, 10:11	Using PETSc with OpenFoam	#1
palarcon Member Pablo Alarcón Join Date: Mar 2018 Location: Liège Posts: 59 Rep Power: 8	Hello everybody I'm trying to include PETSc into my OpenFOAM code in order to be able to perform some topology optimization developments (mainly becase there is a version of the MMA optimizer already coded with PETSc). So far, I have been unable to include petsc.h, even if I added the directories and the libraries in the option file (but is pretty sure that I make a mistake). Does anybody has experience doing this kind of linking? I know that there is an ancient post related to the subject, but the answer was about an ancient version of PETSc and the person who posed the question, was not able to solve the problem. This is my options file Code: EXE_INC = \ -I$(LIB_SRC)/TurbulenceModels/turbulenceModels/lnInclude \ -I$(LIB_SRC)/TurbulenceModels/incompressible/lnInclude \ -I$(LIB_SRC)/TurbulenceModels/compressible/lnInclude \ -I$(LIB_SRC)/transportModels \ -I$(LIB_SRC)/transportModels/incompressible/singlePhaseTransportModel \ -I$(LIB_SRC)/finiteVolume/lnInclude \ -I$(LIB_SRC)/meshTools/lnInclude \ -I$(PETSC_DIR}/include/lnInclude \ -I$(PETSC_DIR}/$(PETSC_ARCH)/include/lnInclude \ EXE_LIBS = \ -lturbulenceModels \ -lincompressibleTurbulenceModels \ -lincompressibleTransportModels \ -lfiniteVolume \ -lmeshTools \ -lfvOptions \ -L$(PETSC_DIR)/lib/petsc/conf \ -L$(PETSC_LIBDIR) having that and adding to my main file #include "petsc.h" I get the error fatal error: petsc.h: No such file or directory compilation terminated. The suggestion that was given a long time ago is Code: include $(RULES)/mplib$(WM_MPLIB) EXE_INC = \ $(PFLAGS) $(PINC) \ -I$(PETSC_DIR)/include -I$(PETSC_DIR)/$(PETSC_ARCH)/include \ -I$(LIB_SRC)/finiteVolume/lnInclude EXE_LIBS = \ -lfiniteVolume \ -L$(PETSC_LIBDIR) -lpetsc -lpetscmat -lpetscvec -lpetscksp -lpetscdm I tried adding the libraries -lpetsc with no luck, so I'm completely lost. If anybody has a way to include PETSc into OpenFOAM (the simples way will do the job actually) I will be really thankful.

July 19, 2018, 10:48		#2
akidess Senior Member Anton Kidess Join Date: May 2009 Location: Germany Posts: 1,377 Rep Power: 30	The library entries are completely irrelevant at this point, don't bother. You need to make sure that you get your includes right. You have obvious errors in your bracketing, and I simply doubt you're picking up the right directories for petsc. I'd recommend hard coding the include directory for petsc.h for starters. __________________ On twitter @akidTwit Spend as much time formulating your questions as you expect people to spend on their answer.

July 23, 2018, 09:48		#6
palarcon Member Pablo Alarcón Join Date: Mar 2018 Location: Liège Posts: 59 Rep Power: 8	Problem solved Given the fact that OpenFOAM and PETSc have a definition for the pow operator, this one had an ambiguous definition. To solve it, I had to add "::" in the pow definition inside the PETSc file, in order to avoir the confusion on the definition. yuminghao likes this.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Frequently Asked Questions about Installing OpenFOAM	wyldckat	OpenFOAM Installation	3	November 14, 2023 12:58
OpenFOAM Training, London, Chicago, Munich, Sep-Oct 2015	cfd.direct	OpenFOAM Announcements from Other Sources	2	August 31, 2015 14:36
OpenFOAM Foundation releases OpenFOAM 2.2.2	opencfd	OpenFOAM Announcements from ESI-OpenCFD	0	October 14, 2013 08:18
Adventure of fisrst openfoam installation on Ubuntu 710	jussi	OpenFOAM Installation	0	April 24, 2008 15:25
The OpenFOAM extensions project	mbeaudoin	OpenFOAM	16	October 9, 2007 10:33

December 28, 2018, 04:45		#7
nuaawqf New Member Wu Qifeng Join Date: Mar 2016 Posts: 1 Rep Power: 0	Hi Pablo: I am interesting in how you using the MMA in PETSc to manipulate the OpenFOAM fields such as alpha. Can you show some hints in your codes? Thanks much.

January 6, 2019, 09:35		#8
palarcon Member Pablo Alarcón Join Date: Mar 2018 Location: Liège Posts: 59 Rep Power: 8	Hello and sorry for the late answer. Finally, and for a number of reasons (mainly the fact that I couldn't find a reliably C++ version of MMA, even if there are) I declined using, at least for the moment, MMA with PETSc. At the time I performed a lot of test case and I wasn't happy with the results. Sorry!