[Hrushi]

Hi,
I am trying to bulid openfoam 2.0.1 on SLES 10 SP1 x86_64 on the lines described in floowing link by Bruno Santos.

http://www.cfd-online.com/Forums/blo...untu-8-04.html

There was an error in buliding cmake-2.8.3. See mkcmake.log file.

However, I went ahead and followed next steps till end as described in above link. I have attached the output of ./Allwmake in two files due to size constraints on file upload.

There is problem with paraView also. See mkPV.log file

Can anybody help me in this?

Thanks

Hrushikesh

[wyldckat]

Hi Hrushikesh,

I've searched a bit and found this - http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43882#c5:

I used -R/path/to/4.5.0/libstdc++.so to fix this library path. But it turns out that -L/path/to/4.5.0/libstdc++.so is also required in addition to this, otherwise the default libstdc++.so is looked up by linker.

This would solve the problem with building CMake. But I'll only be able to check this for myself in about 10h from now

In the mean time, you can check another blog post of mine: Using the official pre-built ParaView 3.10.1 version with OpenFOAM - it's a bit outdated, but a bit of tweaking should fix the differences.

Best regards,
Bruno

[wyldckat]

Hi Hrushikesh,

I've managed to figure the problem! And the issue was already fixed back in September 2011: https://github.com/OpenFOAM/ThirdPar...c22adcab144d5e - I simply didn't pick up on it sooner

Run in the ThirdParty folder the following commands:

rm makeCmake
 wget "https://raw.github.com/OpenFOAM/ThirdParty-2.0.x/master/makeCmake"
chmod +x makeCmake

Then you can properly build CMake and ParaView!

Good luck!
Bruno

[Hrushi]

Hi Bruno,

Thanks a lot. I will try your suggestion and will update you soon on the same.

Again Thanks.

Hrushikesh

[Hrushi]

Hi Bruno,

Yes, that did the trick.

[QUOTE=wyldckat;343460]
Run in the ThirdParty folder the following commands:

rm makeCmake
 wget "https://raw.github.com/OpenFOAM/ThirdParty-2.0.x/master/makeCmake"
chmod +x makeCmake

Then you can properly build CMake and ParaView!

Now both OpenFOAM-2.0.1 and paraView 3.10.1 are succesfully built.

But then I ran into another problem. I can't run my solvers in parallel. See the error file parallel-error.tar.gz

code: mpirun -np 8 'foamExec' interFoam -parallel

However I can run solvers using single processor.

Can you help me in this?

Thanks

Hrushikesh

[wyldckat]

Hi Hrushikesh,

Ouch, that's a crazy crash...
OK, it looks like it is having problems with the Infiniband system... does your system have Infiniband?
Either way, there are at least three options:

1. Re-compile Open-MPI without "openib".
2. Use your system's Open-MPI or any other MPI that it has by default (MPICH, Intel MPI, etc).
3. Try running mpirun with the option for disabling certain NICs:

   Originally Posted by pkr

   The problem is solved if the following command is used:
   mpirun --mca btl_tcp_if_exclude lo,virbr0 -hostfile machines -np 2 /home/rphull/OpenFOAM/OpenFOAM-1.6/bin/foamExec interFoam -parallel
   
   The above command will restrict MPI to use certain networks (lo, vibro in this case).

Best regards,
Bruno

[Hrushi]

[QUOTE=wyldckat;343854]Hi Hrushikesh,

Ouch, that's a crazy crash...
OK, it looks like it is having problems with the Infiniband system... does your system have Infiniband?

Either way, there are at least three options:

1. Re-compile Open-MPI without "openib".
2. Use your system's Open-MPI or any other MPI that it has by default (MPICH, Intel MPI, etc).
   
   I don't know how to do this?
   Can you give some tips on this?
   
   
3. Try running mpirun with the option for disabling certain NICs:

I tried this. But It still gave me same error as previously.

Thanks

Hrushikesh

[wyldckat]

Hi Hrushikesh,

Edit the file "OpenFOAM-2.0.1/etc/config/settings.sh" and you'll see several MPIs available after the following zone:

# Communications library
# ~~~~~~~~~~~~~~~~~~~~~~

unset MPI_ARCH_PATH MPI_HOME FOAM_MPI_LIBBIN

case "$WM_MPLIB" in
OPENMPI)

After you check which MPI you have and/or adapt the one you need, edit the file "$HOME/.bashrc" and change the line:

alias of201='source  $HOME/OpenFOAM/OpenFOAM-2.0.1/etc/bashrc foamCompiler=ThirdParty  WM_COMPILER=Gcc45 WM_NCOMPPROCS=4 WM_MPLIB=MPICH'

Where MPICH is the MPI you want.Then start a new terminal or run "source $HOME/.bashrc"; then run "of201".

Best regards,
Bruno

[Hrushi]

Hi Bruno,
I have tried with suggestions made by you. But still I can not fire my runs in parallel using OpenFoam 2.0.1 on the cluster.

What can be the reason for this? Shall I need to recompile openmpi? If yes , how to do that?

Thanks

Hrushikesh

[wyldckat]

Hi Hrushikesh,

I can't figure it out with so little information...

Anyway, to rebuild your custom Open-MPI, you can remove it by running:

echo $MPI_ARCH_PATH
ls -l $MPI_ARCH_PATH
rm -r $MPI_ARCH_PATH

The first two lines helps you check if you are really erasing the right folder. The third one erases the Open-MPI you built.
Then it's just a matter of running Allwmake once again.

The one thing I don't understand is why the libraries for Infiniband were built!? Did you modify anything in the "ThirdParty-2.1.x/Allwmake" script?

Best regards,
Bruno

[Hrushi]

Hi Bruno,

I tried with all available mpi. Then modified .bshrc accordingly. The machine is giving me error (See mpi-error file).[had forgotten to attach error file in previous post]

Libraries for Infiniband were already built before installing OpenFoam. I have not modified anything in the "ThirdParty-2.1.x/Allwmake" script.

So looking at the error, shall I need to rebuild the MPI and again do Allwmake?

Thanks

Hrushikesh

[wyldckat]

Hi Hrushikesh,

Mmm... I noticed just now that you are running the command in a very bad way. Do not run it like this:

mpirun -np 8 'foamExec' interFoam -parallel

Run it like this:

mpirun -np 8 `which foamExec` interFoam -parallel

There is a simpler way of doing this:

foamJob -s -p interFoam

It (usually) detects the necessary number of cores and the "-p" option launches things in parallel.

If this still doesn't work, then you'll have to do some more tests before trying the solver itself, since this might be a problem with configuring the MPI to work with OpenFOAM and your machine. Here are some old instructions on how to test things:

On how to test if MPI is working: post #4 of "openfoam 1.6 on debian etch", and/or post #19 of "OpenFOAM updates" - Note: As of OpenFOAM 2.0.0, the application "parallelTest" is now called "Test-parallel".

Best regards,
Bruno

[Hrushi]

Hi Bruno,

mpirun -np 8 `which foamExec` interFoam -parallel

This also gives the same error as posted in previous posts.

There is a simpler way of doing this:
Code:
foamJob -s -p interFoam

This gives the following output. (See file foamJob-master).

If this still doesn't work, then you'll have to do some more tests before trying the solver itself, since this might be a problem with configuring the MPI to work with OpenFOAM and your machine. Here are some old instructions on how to test things:
On how to test if MPI is working: post #4 of "openfoam 1.6 on debian etch", and/or post #19 of "OpenFOAM updates" - Note: As of OpenFOAM 2.0.0, the application "parallelTest" is now called "Test-parallel".

I have done the testing as described by you in above posts. The application did got compiled. But foamJob -p -s Test-parallel throws error.The output is in file test-parallel.

Thanks

Hrushikesh

[wyldckat]

Hi Hrushikesh,

You've got a big problem there, mainly because you're not taking into account several important details:

1. You should first do the tests on the machine where you built OpenFOAM. This will confirm to you if things are working or not.
2. You are trying to launch OpenFOAM applications on a cluster where OpenFOAM is not properly installed! When launching OpenFOAM applications, it expects that all machines have the same OpenFOAM installation path.
3. The case folder must be accessible on all machines, or at least properly indicated on "system/decomposeParDict". This includes the need for a writable processor folder from each machine!

Unfortunately I don't have time right now to go into more details, so I'll have to refer you to my Notes about running OpenFOAM in parallel

Best regards,
Bruno

[Hrushi]

Hi Bruno,

Originally Posted by wyldckat

Hi Hrushikesh,

You've got a big problem there, mainly because you're not taking into account several important details:

1. You should first do the tests on the machine where you built OpenFOAM. This will confirm to you if things are working or not.
2. You are trying to launch OpenFOAM applications on a cluster where OpenFOAM is not properly installed! When launching OpenFOAM applications, it expects that all machines have the same OpenFOAM installation path.
3. The case folder must be accessible on all machines, or at least properly indicated on "system/decomposeParDict". This includes the need for a writable processor folder from each machine!

Unfortunately I don't have time right now to go into more details, so I'll have to refer you to my Notes about running OpenFOAM in parallel

Best regards,
Bruno

1.> I did the test on the machine on which OpenFOAM is built.
In .bashrc file, I have this line:
alias of201='source /Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/etc/bashrc foamCompiler=ThirdParty WM_COMPILER=Gcc45 WM_NCOMPPROCS=8'

Then I went to damBreak tutorial and decompose the case into 4 domains. Then I ran foamJob -p -s Test-parallel (parallel.txt) and also foamJob -p -s interFoam (error.txt).

2.> I ran foamInstallationTest. It does not show any error. (foamInst.txt)

3.>Before building openFOAM 2.0.1, I had openFOAM 1.6 installed on the cluster. In OpenFOAM 1.6 (openmpi-1.3.3 included in thirdparty-1.6 package), parallel runs were working smoothly without any error on master as well as on nodes.

So what can be the reason for parallel runs not working in OpenFoam-2.0.1?

Meanwhile, I came across this:
http://www.openfoam.com/mantisbt/view.php?id=296

Thanks

Hrushikesh

[wyldckat]

Indeed! I vaguely remember seeing the same here on the forum... about using Open-MPI 1.4.3 instead... Here we go: http://www.cfd-online.com/Forums/ope...s-openmpi.html

By the way, since you are using the alias method for activating OpenFOAM environments, you might be interested in the following bug report for when using foamJob: http://www.openfoam.com/mantisbt/view.php?id=231 - the pertinent file is this: patch4foamJob_send_settings_var

Good luck!
Bruno

[Hrushi]

Hi Bruno,

I did managed to get the OpenFOAM 2.0.1 and 2.1.0 running in parallel mode (with OpenMPI 1.4.1) on a single node of the cluster using foamJob as well as mpirun command.

But I am not been able to float the runs on multiple nodes.

log file:

/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found
/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/bin/foamExec: line 145: exec: interFoam: not found

Can this be because OpenFOAM environment is not active on other nodes? I usually log onto the node, activate the OF environment, then float the run.

What can be done about this?

Hrushikesh

[wyldckat]

Hi Hrushikesh,

Have you made the changes shown in this link: patch4foamJob_send_settings_var ?

If not or if you don't understand how to do them, run the following commands:

(echo export foamCompiler=ThirdParty; echo export WM_COMPILER=Gcc45; echo export WM_NCOMPPROCS=8)  > /Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/etc/prefs.sh

Or simply edit the file "/Storage1/cfd/OpenFOAM/OpenFOAM-2.0.1/etc/prefs.sh" and place in it the following lines:

export foamCompiler=ThirdParty
export WM_COMPILER=Gcc45
export WM_NCOMPPROCS=8

Best regards,
Bruno

[Hrushi]

Hi Bruno,

Yup. That did the trick (patch4foamJob_send_settings_var).

Thanks for telling me g=how to do those settings.

Now I can run OpenFOAM on multiple nodes.


Hrushikesh

[Hrushi]

Hi Bruno,

Yup. That did the trick (patch4foamJob_send_settings_var).

Thanks for telling me g=how to do those settings.

Now I can run OpenFOAM on multiple nodes.


Hrushikesh