[Mibte]

Hi,
I am running an internal combustion engine with turbDyMEngineFoam using a dynamicTopoFvMesh approach. The case works good in serial but in parallel it cannot load the decomposition libraries. Every 200 time-steps there is a dynamic parallel load-balancing process using parMetis and this is what the terminal says:

--> FOAM Warning : 
    From function decompositionMethod::loadExternalLibraries()
    in file decompositionMethod/decompositionMethod.C at line 453
    Loading of decomposition library libscotchDecomp.so unsuccesful. Some decomposition methods may not be  available
--> FOAM Warning : 
    From function decompositionMethod::loadExternalLibraries()
    in file decompositionMethod/decompositionMethod.C at line 453
    Loading of decomposition library libmetisDecomp.so unsuccesful. Some decomposition methods may not be  available
--> FOAM Warning : 
    From function decompositionMethod::loadExternalLibraries()
    in file decompositionMethod/decompositionMethod.C at line 453
    Loading of decomposition library libparMetisDecomp.so unsuccesful. Some decomposition methods may not be  available
Selecting decompositionMethod parMetis

I have checked for the libraries and they do exist, maybe they are not in the right place, where should they be located? Does anyone knows how could I solve this problem? I am running OpenFOAM 3.0

Finally the case crashes almost at the end with 4 subdomains (with 2 subdomains it gets to the end without the following error) showing this error:

[0] --> FOAM FATAL ERROR:
[0] Mapping for inserted boundary face is incorrect. Found an empty masterObjects list. 
 Face: 46884
 Patch: liner
[0]
[0]        From function void topoPatchMapper::calcInsertedFaceAddressing( ) const in file fieldMapping/topoPatchMapper.C at line 137
[0]
FOAM parallel run aborting
[0]
--------------------------------------------------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1.

Could anyone help me to solve this problem?

Thank you very much.

[Mibte]

Hello, I am trying to run some case in parallel with scotch, I compiled and installed it but I did not do it as I should because I always have the following error:

--> FOAM Warning : 
    From function decompositionMethod::loadExternalLibraries()
    in file decompositionMethod/decompositionMethod.C at line 453
    Loading of decomposition library libscotchDecomp.so unsuccesful. Some decomposition methods may not be  available

Is there any guide where I can follow step by step what to do and what commands should I use? I am running it on Ubuntu with OpenFOAM-3.0

Thank you for your help.

[wyldckat]

Greetings Mibte and welcome to the forum!

I believe you're referring to foam-extend 3.0 and not to OpenFOAM 3.0, since OpenFOAM is currently only in version 2.3.

Please indicate which Linux distribution you're using, what architecture (32 or 64 bit) and which installation instructions did you follow.

Best regards,
Bruno

PS: I've moved your two posts to the same thread, as one complements information for the other.

[Mibte]

Hi Bruno.

Yes, I am referring to foam-extend 3.0. It is 64 bits and in order to install it I followed the instructions from this webpage for Ubuntu 13.10:

http://openfoamwiki.net/index.php/In...u#Ubuntu_13.10

I could use either parMetis or scotch but I have the same problem with both. When compiling and installing parMetis from the ThirdPARTY Packages from the BUILD directory it says that it is made by default in usr/local. The problem is that I should point the parMetisDecomp sources to the correct compiled location, but I do not know how to do it.
Has it to be either to the ThirdParty directories or to usr/local?
Should it be pointed in the files or options files within the make directory from the parMetisDecomp folder within the decompositionMethods directory ? or both?

This is what I currently have in the files and options files respectively:

parMetisDecomp.C

LIB = $(FOAM_MPI_LIBBIN)/libparMetisDecomp

include $(RULES)/mplib$(WM_MPLIB)

EXE_INC = \
$(PFLAGS) $(PINC) \
-I$(PARMETIS_INCLUDE_DIR) \
-I../decompositionMethods/lnInclude \
-I../metisDecomp/lnInclude \
-I../scotchDecomp/lnInclude

LIB_LIBS = \
-L$(FOAM_MPI_LIBBIN) \
-L$(PARMETIS_LIB_DIR) -lparmetis

Thanks for your help.

[wyldckat]

Hi Mibte,

Mmm... something is missing.
OK, the first detail is the instructions at http://openfoamwiki.net/index.php/In...u#Ubuntu_13.10 might be a bit outdated, because foam-extend 3.0 is gradually evolving. Therefore, as suggested at the top of that page, the up-to-date installation instructions are these: http://openfoamwiki.net/index.php/In...oam-extend-3.0

The other detail is that even though you've indicated you're following the instructions meant for Ubuntu 13.10, you didn't specify if you are indeed using Ubuntu 13.10?!

As for the problems with building parMetis, I get the feeling that there were some steps you've done differently for installing foam-extend 3.0!? Because the "/usr/local" installation location is only meant for system-wide installation and the installation instructions usually focus on keeping things installed in the user's home folder.

The other feeling is that you might be having problems with a conflict between having more than one OpenFOAM version/variant. If you do have more than one version installed, please describe what procedure are you using to change between each version.

So, beyond the requests/ideas above, my question is this: if you follow the step #11 from this section http://openfoamwiki.net/index.php/In...u#Ubuntu_13.10 - what is the content of the file "log.AllMake"?

Best regards,
Bruno

[Mibte]

Hello Bruno,

Should I delete it and reinstall it again following the new instructions you send to me?

Actually I am running Ubuntu 13.10.

With respect to building parMetis I just install foam and the Thirparty as usual, but like it did not work I tried to install it from the following BUILD.txt file within /ThirdParty/rpmBuild/BUILD/parMetis-4.0.3:

------------------------------------------------------------------------------
Building ParMETIS requires CMake 2.8, found at http://www.cmake.org/, as well as
GNU make. Assumming CMake and GNU make are installed, two commands should
suffice to build ParMETIS:

     $ make config
     $ make


Configuration
-------------
ParMETIS is primarily configured by passing options to make config. For
example:

     $ make config shared=1 

would configure ParMETIS to be built as a shared library.

Common configuration options are:
  cc=[compiler]   - The C compiler to use [default is mpicc]
  cxx=[compiler]  - The C++ compiler to use [default is mpicxx]
  shared=1        - Build a shared library instead of a static one 
                    [off by default]
  prefix=[PATH]   - Set the installation prefix [/usr/local/ by default]

Advanced debugging related options:
  gdb=1       - Build with support for GDB [off by default]
  debug=1     - Enable debugging support [off by default]
  assert=1    - Enable asserts [off by default]
  assert2=1   - Enable very expensive asserts [off by default]

Installation
------------
To install ParMETIS, run

    $ make install

The default installation prefix is /usr/local. To pick an installation 
prefix for ParMETIS pass prefix=[path] to make config. For example,

    $ make config prefix=~/myroot/

will cause ParMETIS to be installed in ~/myroot/ when make install is run.


Other make commands
-------------------
   $ make uninstall 
          Removes all files installed by 'make install'.
   
   $ make clean 
          Removes all object files but retains the configuration options.
   
   $ make distclean 
          Performs clean and completely removes the build directory.

------------------------------------------------------------------------------

So I used the make config, make and make install commands.

With respect to other versions, I just have the foam extended-3.0 version, I never installed any other different version.

I attach in this message the log.AllMake file from step number 11.

I would like also to show you what happens when running the foamTestInstallation. Before I did not get any error (although parMetis was not working anyway), but yesterday I reinstalled everything again and I get the following:

Executing /home/alvaro/foam/foam-extend-3.0/bin/foamInstallationTest:


Checking basic setup...
-------------------------------------------------------------------------------
Shell:              bash
Host:               alvaro-Lenovo-IdeaPad-Z500
OS:                 Linux version 3.11.0-17-generic
-------------------------------------------------------------------------------


Checking main foam-extend env variables...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory                Valid      Crit
-------------------------------------------------------------------------------
$WM_PROJECT_INST_DIR /home/alvaro/foam                        yes       yes
$WM_PROJECT_USER_DIR /home/alvaro/foam/alvaro-3.0             yes       no
$WM_THIRD_PARTY_DIR  ...varo/foam/foam-extend-3.0/ThirdParty  yes       yes
-------------------------------------------------------------------------------


Checking the foam-extend env variables set on the PATH...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory                Valid Path Crit
-------------------------------------------------------------------------------
$WM_PROJECT_DIR      /home/alvaro/foam/foam-extend-3.0        yes  yes  yes

$FOAM_APPBIN         ...0/applications/bin/linux64Gcc48DPOpt  yes  yes  yes
$FOAM_SITE_APPBIN    .../foam/site/3.0/bin/linux64Gcc48DPOpt  no        no
$FOAM_USER_APPBIN    ...0/applications/bin/linux64Gcc48DPOpt  yes  yes  no
$WM_DIR              /home/alvaro/foam/foam-extend-3.0/wmake  yes  yes  yes
-------------------------------------------------------------------------------


Checking the foam-extend env variables set on the LD_LIBRARY_PATH...
-------------------------------------------------------------------------------
Environment_variable Set_to_file_or_directory                Valid Path Crit
-------------------------------------------------------------------------------
$FOAM_LIBBIN         ...oam-extend-3.0/lib/linux64Gcc48DPOpt  yes  yes  yes
$FOAM_SITE_LIBBIN    .../foam/site/3.0/lib/linux64Gcc48DPOpt  no        no
$FOAM_USER_LIBBIN    ...oam/alvaro-3.0/lib/linux64Gcc48DPOpt  yes  yes  no
$MPI_ARCH_PATH       /usr                                     yes   no  yes
-------------------------------------------------------------------------------


Third party software
-------------------------------------------------------------------------------
Software Version   Location 
-------------------------------------------------------------------------------
WARNING: gcc version does not match gcc supplied with this release of foam-extend
         Supplied version: 4.3.3
         User version    : 4.8.1
         Minimum required: 4.3.1

gcc      4.8.1    
WARNING:  Conflicting installations:
          foam-extend settings        : /bin/gcc
          current path             : /usr/bin/gcc
          CRITICAL ERROR

gzip     1.6       /bin/gzip                                                
tar      1.26      /bin/tar                                                 
icoFoam  3.0       ...-extend-3.0/applications/bin/linux64Gcc48DPOpt/icoFoam
-------------------------------------------------------------------------------


Summary
-------------------------------------------------------------------------------
Base configuration ok.

The foam installation contains 1 critical error(s).

Review the output for warning messages and consult 
the installation guide for trouble shooting.

done.

Thank you very much.

Best regards,
Alvaro.

[wyldckat]

Hi Alvaro,

The log file indicates that there is nothing wrong with the build for the ThirdParty software.
The output from foamInstallationTest can be ignored, as the code is a bit outdated.

The other important log file is the "make.log" file from step #13.

Can you provide a simple test case where I can test turbDyMEngineFoam to check this error myself? I ask this because I haven't been able to find any example online.

Best regards,
Bruno

[Mibte]

Hi Bruno,

I attach with this message the make.log from step number 13.

With respect to the turbDyMEngineFoam case I tried to attach it but the size is too big, it does not allow me to attach it. Could you provide me with an email address where I could send it to you please?

Best regards,
Alvaro

[wyldckat]

Hi Alvaro,

The log file doesn't indicate any build errors. The problem must be something more specific, such as turbDyMEngineFoam not being prepared to properly load up the decomposition libraries...

As for sending me the case, please use Dropbox or a similar service. If the data is sensitive, you can send me the link over private message. The reason why I ask this, is because I have to keep OpenFOAM related questions on the forum, otherwise I'll loose track of the questions that have been asked to me

Best regards,
Bruno

[Mibte]

Hello Bruno,

I understand, I just sent it to you with a dropbox link in a private message.

Thank you very much, Alvaro

[wyldckat]

Hi Alvaro,

OK, I've run the case with "parMetis" and the first issue is now clear to me: it's just a warning and doesn't affect operations. The reason for this happening is as follows:

1. In the file "dynamicTopoFvMesh/coupledMesh/dynamicTopoFvMeshCoupled.C" (part of the updated "dynamicTopoFvMesh" library kit), the method "executeLoadBalancing" creates a new instance of the object "decompositionMethod".
2. When creating a new instance of that object (in the main source code, "src/decompositionMethods/decompositionMethods/decompositionMethod/decompositionMethod.C"), the first thing it does is to call the method "loadExternalLibraries", which will load the libraries in question, namely "scotchDecomp", "metisDecomp" and "parMetisDecomp".
3. Now, once these libraries are loaded into memory, they only need to be loaded once. Problem is that since a new instance of the object "decompositionMethod" is created on each balance, it will try to load the libraries again and fail, hence the warning that it failed to load them.

This should not affect operations, as since it failed, it should not have made any damage to the data/libraries already loaded in memory.


Now, for it to be crashing, the problem seems to be that you have disregarded the importance of the mesh and of the Courant Number. Here is the output of the iteration where for me it started to go wrong and lead to crashing a few steps later:

Volume: new = 0.00021076 old = 0.00021076 change = 0
Motion continuity errors : sum local = 7.0896e-05, maximum = 0.0728194
Courant Number mean: 0.0626986 max: 1.14966 velocity magnitude: 153.974
Time = 20.2

Solving for point motion:  Initial residual: 1 Final residual: 0.00991242 No Iterations: 10
Solving for point motion:  Initial residual: 1 Final residual: 0.00844706 No Iterations: 11

 ~~~ Mesh Quality Statistics ~~~ 
 Min: 0.31196
 Max: 0.99892
 Mean: 0.833472
 Cells: 89817
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

 Topo modifier time: 0.15968 s
 Bisections :: Total: 12, Surface: 3
 Collapses  :: Total: 0, Surface: 0
 Swaps      :: Total: 84, Surface: 3
 Mapping time: 0.047496 s
 Reordering time: 0.047907 s
 ~~~ No flux correction ~~~ 

 ~~~ Mesh Quality Statistics ~~~ 
 Min: 0.31196
 Max: 0.99892
 Mean: 0.833526
 Cells: 89875
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Volume: new = 0.00021096 old = 0.00021076 change = 1.9942e-07
Motion continuity errors : sum local = 6.61234e-05, maximum = 0.0427241
DICPCG:  Solving for pcorr, Initial residual = 1, Final residual = 0.0433402, No Iterations 2
DICPCG:  Solving for pcorr, Initial residual = 0.0533498, Final residual = 0.00524394, No Iterations 132
time step continuity errors : sum local = 2.28419e-05, global = -4.93908e-07, cumulative = -0.00056392
Courant Number mean: 0.0621874 max: 1.15332 velocity magnitude: 141.013
DILUPBiCG:  Solving for Ux, Initial residual = 0.00200019, Final residual = 0.000181056, No Iterations 1
DILUPBiCG:  Solving for Uy, Initial residual = 0.000978087, Final residual = 464.5, No Iterations 1000
DILUPBiCG:  Solving for Uz, Initial residual = 0.00267227, Final residual = 0.000113367, No Iterations 1
DICPCG:  Solving for p, Initial residual = 0.999753, Final residual = 0.0985986, No Iterations 37
DICPCG:  Solving for p, Initial residual = 0.320856, Final residual = 0.0314022, No Iterations 3
time step continuity errors : sum local = 1.84645, global = -0.000641904, cumulative = -0.00120582
DICPCG:  Solving for p, Initial residual = 0.421007, Final residual = 0.0397563, No Iterations 25
DICPCG:  Solving for p, Initial residual = 0.53964, Final residual = 9.98505e-07, No Iterations 303
time step continuity errors : sum local = 3.40427e-05, global = -2.76501e-07, cumulative = -0.0012061
ExecutionTime = 466.46 s  ClockTime = 481 s

Volume: new = 0.00021096 old = 0.00021096 change = 0
Motion continuity errors : sum local = 6.61234e-05, maximum = 0.0427241
Courant Number mean: 13.5836 max: 4851.13 velocity magnitude: 784106

The last line clearly indicates that things have gone seriously wrong.

If you analyze the "time step continuity errors" lines right after each load balance, you'll see that the continuity has some issues, even though it seems to solve them later on. My suspicion is that since a fixed step of 0.1s leads to a Courant Number greater than 0.5 and 1.0, this causes the simulation to be rather unstable and leads to the eventual crash.


Regarding the mesh: you only have a 1 or 2 cells between certain walls. This leads to a considerable high amount of mass to be transfered between cells, which can lead to simulation problems. In this blog post of mine you'll see some examples of what this can lead to: OpenFOAM: Interesting cases of bad meshes and bad initial conditions


As for the issue with Scotch... it's not parallel-aware, therefore it cannot be used during the run. Which means that either you have to use "ptscotch" or stay with "parMetis".... er, "ptscotch" doesn't exist in foam-extend, so it can't be used.


But as far as I can see, although I have not confirmed this, the best approach seems to be for you to use "engineScotch" for the initial decomposition and then to not use load balancing during the run.

Best regards,
Bruno

[Mibte]

Hi Bruno, thank you for all the information.

When you say that it does not affect the process that means that it distributes equitably the number of cells among the processors for you? Could I see your log file when the load balancing process takes part?

With respect to why it is crashing, after all the changes I made to the case I sent to you, it has been working differently than before. I have sent you another case along with the log file and a picture of the error, it is almost the same but it works perfectly for me until 358.4º out of 360º. The error is not apparently related with the Courant Number, it says the following:

--> FOAM FATAL ERROR: 
[0]  Mapping for inserted boundary face is incorrect. Found an empty masterObjects list.
 Face: 41770
 Patch: liner
[0] 
[0]     From function void topoPatchMapper::calcInsertedFaceAddressing() const
[0]     in file fieldMapping/topoPatchMapper.C at line 137.
[0] 
FOAM parallel run aborting

Regarding "engineScotch", do you have a decomposePar file where I can see the different coefficients in order to make it work? Because I have tried to search for it but I could not find anything. Thank you.

Best regards,
Alvaro.

[wyldckat]

Hi Alvaro,

Attached is the log file from the run I made yesterday. By the way, in order to preserve the whole output in a single log file, for example use this:

mpirun -np 4 moveDynamicMesh -parallel > log.move 2>&1

The "2>&1" at the end will send the error output to the standard output, so that all output is saved in the file.
As for the moveDynamicMesh, it's because I'm testing only the mesh motion, instead of running the whole simulation.
-----------
edit: The result from moveDynamicMesh was not what I expected. It simply didn't move
-----------


And I did some digging and testing but have not been successful in using the "engineScotch" decomposition method. All I know is this:

1. The dictionary will have to contain something like this:
   

   numberOfSubdomains 4;
   
   method          engineScotch;
   
   engineScotchCoeffs
   {
     slidingPatchPairs ((a b));
     expandSliding false;
   }

   The "(a b)" is just an example of listing a pair, namely that the patch "a" and patch "b" make up a sliding pair.
   If nothing is given, i.e.:
   

     slidingPatchPairs ();

   it decomposes without any error messages.
2. The description on the header file "src/decompositionMethods/scotchDecomp/engineScotchDecomp/engineScotchDecomp.H" says this:
   

   Description
       Domain decomposition for internal combustion engine simulations with
       topological changes.
   
       The algorithm attempts to be as general as possible and operates in the
       following steps:
       - identify piston region and peform a decomposition biased towards the
       piston axis.  This aims to preserve load balancing when cell layers
       are added
       - identify static regions and decompose them separately

3. It needs a patch named "piston".

And that's all I could figure out.
But in your case, you might instead want to rely on adding the entry "preservePatches", e.g.:

numberOfSubdomains 4;

preservePatches (piston);

method          parMetis;

Regarding the error you got: it's still running the mesh motion in my machine. And I changed the "writeInterval" to 0.1, so that I can keep track of all mesh changes.
My first guess is that this is a bug in either in foam-extend or in dynamicTopoFvMesh. But only after the mesh motion run is complete, will I be able to try and figure out what happened.


In the mean time, I'll be moving this thread to the "solving" forum, as this isn't an issue with installation.

Best regards,
Bruno

[Mibte]

Hi Bruno,

I have looked at your output file and it seems to be happening the same issue as I had, not making correctly the load balancing process. I tried to run it without the load balancing method, but it crashes even earlier.

I am currectly running again the case as you proposed by using "preservePatches" in order to see if it gets to the end without any error, apart from the warnings of the decomposition libraries:

numberOfSubdomains 4;

preservePatches (piston);

method          parMetis;

For making the load balancing process work I was told to compile parMetis correctly in the ThirdParty directory and then point the parMetisDecomp sources to the compiled location and recompile parMetisDecomp. I tried to do so. I modified my options file from src/decompositionMethods/parMetisDecomp/Make like this:

include $(RULES)/mplib$(WM_MPLIB)

EXE_INC = \
    $(PFLAGS) $(PINC) \
    -I$(PARMETIS_INCLUDE_DIR) \
    -I../decompositionMethods/lnInclude \
    -I../metisDecomp/lnInclude \
    -I/home/alvaro/foam/foam-extend-3.0/src/decompositionMethods/decompositionMethods/lnInclude \
    -I/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64GccDPOpt/include/ \
    -I../scotchDecomp/lnInclude

LIB_LIBS = \
    -L$(FOAM_MPI_LIBBIN) \
    -L$(PARMETIS_LIB_DIR) -lparmetis \
    -L/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64GccDPOpt/lib -lparmetis

Then I tried to recompile parMetis using wmake, being located at the directory src/decompositionMethods/parMetisDecomp/ but it does not work correctly, saying the following:

g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -Wnon-virtual-dtor -O3  -DNoRepository -ftemplate-depth-200 -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread     -I/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64Gcc48DPOpt/include     -I../decompositionMethods/lnInclude     -I../metisDecomp/lnInclude     -I/home/alvaro/foam/foam-extend-3.0/src/decompositionMethods/decompositionMethods/lnInclude     -I/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64GccDPOpt/include/     -I../scotchDecomp/lnInclude -IlnInclude -I. -I/home/alvaro/foam/foam-extend-3.0/src/foam/lnInclude -I/home/alvaro/foam/foam-extend-3.0/src/OSspecific/POSIX/lnInclude   -fPIC -fpermissive -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64Gcc48DPOpt/parMetisDecomp.o -L/home/alvaro/foam/foam-extend-3.0/lib/linux64Gcc48DPOpt \
          -lfoam -liberty -ldl   -lm -o foam.out
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 0 has invalid symbol index 11
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 1 has invalid symbol index 12
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 2 has invalid symbol index 2
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 3 has invalid symbol index 2
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 4 has invalid symbol index 11
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 5 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 6 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 7 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 8 has invalid symbol index 12
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 9 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 10 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 11 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 12 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 13 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 14 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 15 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 16 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 17 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 18 has invalid symbol index 13
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 19 has invalid symbol index 21
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_line): relocation 0 has invalid symbol index 2
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `Foam::parMetisDecomp::parMetisDecomp(Foam::dictionary const&, Foam::polyMesh const&)':
parMetisDecomp.C:(.text+0x1a): undefined reference to `vtable for Foam::decompositionMethod'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `Foam::parMetisDecomp::decompose(Foam::Field<int>&, Foam::Field<int>&, Foam::Field<Foam::Vector<double> > const&, Foam::Field<int>&, Foam::Field<int>&, Foam::List<int> const&, Foam::List<int>&)':
parMetisDecomp.C:(.text+0x2411): undefined reference to `ParMETIS_V3_PartGeomKway'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `Foam::parMetisDecomp::decompose(Foam::List<Foam::List<int> > const&, Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&)':
parMetisDecomp.C:(.text+0x2f98): undefined reference to `Foam::decompositionMethod::calcCSR(Foam::List<Foam::List<int> > const&, Foam::List<int>&, Foam::List<int>&)'
parMetisDecomp.C:(.text+0x374c): undefined reference to `Foam::metisDecomp::metisDecomp(Foam::dictionary const&, Foam::polyMesh const&)'
parMetisDecomp.C:(.text+0x3762): undefined reference to `Foam::metisDecomp::decompose(Foam::List<Foam::List<int> > const&, Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&)'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `Foam::parMetisDecomp::decompose(Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&)':
parMetisDecomp.C:(.text+0x4055): undefined reference to `Foam::decompositionMethod::calcDistributedCSR(Foam::polyMesh const&, Foam::List<int>&, Foam::List<int>&)'
parMetisDecomp.C:(.text+0x5089): undefined reference to `Foam::metisDecomp::metisDecomp(Foam::dictionary const&, Foam::polyMesh const&)'
parMetisDecomp.C:(.text+0x509e): undefined reference to `Foam::metisDecomp::decompose(Foam::Field<Foam::Vector<double> > const&, Foam::Field<double> const&)'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `Foam::parMetisDecomp::~parMetisDecomp()':
parMetisDecomp.C:(.text._ZN4Foam14parMetisDecompD2Ev[_ZN4Foam14parMetisDecompD5Ev]+0x3): undefined reference to `vtable for Foam::decompositionMethod'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `Foam::parMetisDecomp::~parMetisDecomp()':
parMetisDecomp.C:(.text._ZN4Foam14parMetisDecompD0Ev[_ZN4Foam14parMetisDecompD0Ev]+0x3): undefined reference to `vtable for Foam::decompositionMethod'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `Foam::decompositionMethod::adddictionaryMeshConstructorToTable<Foam::parMetisDecomp>::~adddictionaryMeshConstructorToTable()':
parMetisDecomp.C:(.text._ZN4Foam19decompositionMethod35adddictionaryMeshConstructorToTableINS_14parMetisDecompEED2Ev[_ZN4Foam19decompositionMethod35adddictionaryMeshConstructorToTableINS_14parMetisDecompEED5Ev]+0x1): undefined reference to `Foam::decompositionMethod::destroydictionaryMeshConstructorTables()'
Make/linux64Gcc48DPOpt/parMetisDecomp.o: In function `_GLOBAL__sub_I_parMetisDecomp.C':
parMetisDecomp.C:(.text.startup+0xc1): undefined reference to `Foam::decompositionMethod::constructdictionaryMeshConstructorTables()'
parMetisDecomp.C:(.text.startup+0xda): undefined reference to `Foam::decompositionMethod::dictionaryMeshConstructorTablePtr_'
Make/linux64Gcc48DPOpt/parMetisDecomp.o:(.data.rel.ro._ZTIN4Foam14parMetisDecompE[_ZTIN4Foam14parMetisDecompE]+0x10): undefined reference to `typeinfo for Foam::decompositionMethod'
Make/linux64Gcc48DPOpt/parMetisDecomp.o:(.data.rel.ro._ZTVN4Foam14parMetisDecompE[_ZTVN4Foam14parMetisDecompE]+0x30): undefined reference to `Foam::decompositionMethod::decompose(Foam::Field<Foam::Vector<double> > const&)'
Make/linux64Gcc48DPOpt/parMetisDecomp.o:(.data.rel.ro._ZTVN4Foam14parMetisDecompE[_ZTVN4Foam14parMetisDecompE]+0x40): undefined reference to `Foam::decompositionMethod::decompose(Foam::List<int> const&, Foam::Field<Foam::Vector<double> > const&)'
collect2: error: ld returned 1 exit status
make: *** [foam.out] Error 1

Do you know why is it not compiling correctly? Thanks for your help.

Best regards,
Alvaro.

[wyldckat]

Hi Alvaro,

It's a bit strange that the build scripts at the "ThirdParty" folder did not properly handle the build process of "parMetis". And if it ran without problems during so many iterations, it's even more confusing...

Anyway, the correct way for you to point the configuration to the new installation of "parMetis" is as follows:

1. Undo the changes you've made to the "parMetisDecomp/Make/options" file.
2. Edit the file "$WM_PROJECT_DIR/etc/prefs.sh". You can know its path by running:
   

   echo $WM_PROJECT_DIR/etc/prefs.sh

3. Find these lines:
   

   #export PARMETIS_DIR=path_to_system_installed_parmetis
   #export PARMETIS_BIN_DIR=$PARMETIS_DIR/bin
   #export PARMETIS_LIB_DIR=$PARMETIS_DIR/lib
   #export PARMETIS_INCLUDE_DIR=$PARMETIS_DIR/include

4. Remove the comment character "#" and change in the first line to the correct path of where you installed parMetis. For example:
   

   export PARMETIS_DIR=/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64GccDPOpt/export PARMETIS_BIN_DIR=$PARMETIS_DIR/bin
   export PARMETIS_LIB_DIR=$PARMETIS_DIR/lib
   export PARMETIS_INCLUDE_DIR=$PARMETIS_DIR/include

5. Save and close the file.
6. Start a new terminal, to ensure the variables are properly updated/loaded and defined.

The case finally reached the crash point... it took 18750s to reach the same angle 358.4º and then crash.

I'm going to try and diagnose why this happened... but I'm not 100% certain I'll be able to. I'll give more feedback as soon as I can.


edit: This error has also occurred for another person a few days ago: http://www.cfd-online.com/Forums/ope...tml#post488775 - post #91

Best regards,
Bruno

[wyldckat]

Hi Alvaro,

OK, when "dynamicTopoFvMesh" is in play, debugging dynamic meshes is waaaaay over my head

Since fortunately this issue is identical to another one reported recently here: http://www.cfd-online.com/Forums/ope...tml#post488775 post #91 - I ask you to follow these steps below for preparing the case for sending it to Sandeep Menon (see post #92), the author of "dynamicTopoFvMesh":

1. Once your case crashes at 358.4º, edit the file "system/controlDict" and change this line:
   

   startFrom       startTime;

   to this:
   

   startFrom       latestTime;

2. Edit the file "constant/dynamicMeshDict" and find these lines:
   

       // Set the run-time debug level
       debug               0;

   Change to this:
   

       // Set the run-time debug level
       debug               5;

3. Launch the solver again in parallel. It should take a while to run the problem iteration and then crash in the same way, but with a really longer log file. For example:
   

   mpirun -np 4 turbDyMEngineFoam -parallel > log.parallel2 2>&1

4. Make sure the "constant" folders at the processor folders exist, so that post-processing with the native reader in ParaView can work properly with the decomposed case:
   

   for a in processor*; do mkdir $a/constant; done

5. Now, package only the files/folders that matter, for example, like this:
   

   cd ..
   tar -czf turbDyMEngineFoam4subdomains.debug_at_358.3_358.4.tar.gz turbDyMEngineFoam4subdomains/{0,constant,system,log.parallel2} turbDyMEngineFoam4subdomains/processor*/{0,358.3,constant,system,VTK}

   This package has all of the essential files for diagnosing the problem, without having to run any more iterations.

The please send a PM to Sandeep Menon with a link to the package "turbDyMEngineFoam4subdomains.debug_at_358.3_358.4 .tar.gz", with a summary of the issues discussed on this thread, in particular the error message, along with a link to this thread. I believe Sandeep should be interested in having more than one test case where this same issue occurs, specially since this case is small enough to test with.
If you wish, you should also post at that other thread indicating that this same issue has been reproduced with another case and informing that you'll send or have sent the case over private message.

Best regards,
Bruno

[wyldckat]

Hi Alvaro,

I'm going to answer half of your question here on the public forum, if you don't mind

Originally Posted by Mibte

With respect to the post #15, I followed the steps but I still have the same error, even when I reinstalled parMetis again from the Thirdparty/rpmBuild/BUILD/parmetis-4.0.3 by using:

1. make config prefix=/home/alvaro/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64GccDPOpt/

2. make

3. make install

In addition, when running the case I sent to you, it does not arrive anymore to 358.4, it crashes at 309 with a huge Courant number. For these two reasons I have decided to eliminate foam extended-3.0 and installed it again from the website you told me:

http://openfoamwiki.net/index.php/In...oam-extend-3.0

I have been trying so hard to make it work, that some things might have unset and it does not work as it used to do.
When I complete the installation following the webpage steps, the parMetis libraries should be installed directly right? In which directory will it be? So that I can follow your instructions and point it correctly from the prefs.sh file.

When I use my alias command to start up the shell environment for foam-extend 3.0, it lists all of the environment settings regarding 3rd party software. For parMetis, is states something like this:

Sourcing:  /home/ofuser/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64Gcc46DPOpt/etc/parmetis-4.0.3.sh
     PARMETIS_DIR is initialized to:  /home/ofuser/foam/foam-extend-3.0/ThirdParty/packages/parmetis-4.0.3/platforms/linux64Gcc46DPOpt

The special architecture settings "linux64Gcc46DPOpt" is because I'm using Ubuntu 12.04.
I do believe that parMetis is built properly by the build scripts that foam-extend provides, namely those "AllMake*" scripts that are inside the folder "ThirdParty".
When you're finished building foam-extend 3.0, if you run the following command, you should get the full path to where parMetis is installed:

echo $PARMETIS_DIR

The complete list of settings used for building parMetis are in the following files:

• "ThirdParty/AllMake.stage3"
• "ThirdParty/rpmBuild/SPECS/parmetis-4.0.3.spec"

The only difference that I can find, where compared to your steps, is that they use a manual installation procedure, instead of using "make install".


I'll answer to the rest via private message.

Best regards,
Bruno

[Slanth]

Hello, Mibte:

I have read posts above mentioned. You said that you used dynamicTopoMesh (mesquitesolver) to simulate engine. So I wonder if you have successfully run your engine simulation case in parallel.
I really am interested in whether combination of Mesquite solver and enginescorch decomposition method is applicable. Another question I want to ask is that: because I want to perform full cycle simulation, so how to implement attach/detach in mesquite solver?

Looking forward to you reply, thanks very much.