Spdp and two MPI implemetations

bastil · November 20, 2008, 17:41

Hi all,

I have OF installed with both sp and dp. Therefore I change the environment and recompile everything a second time with the other precision. This works but gives me everything twice. Is this essential?
Additionally I am planing to use a second mpi implementation (thinging of MPIGamma). Does this mean four installations now? (sp/dp for both of the mpis) Or is there another way to switch mpi implementation? Thanks.

BastiL

mattijs · November 21, 2008, 04:39

Anything that uses 'scalar' in the source code needs recompilation. There is no need to recompile openmpi if you switch precision.

If you use a second mpi implementation you'll need to recompile the Pstream library so you'll have two (SP and DP) of all OpenFOAM libraries and 4 libPstream.so.

Try Open-MX (http://gforge.inria.fr/projects/open-mx/) instead of MPIGamma. Please post some results if you are successful.

bastil · November 21, 2008, 19:53

I did not get that. So far I recompile ALL aplications for sp. First I compile everything in dp and afterwards in sp. So I get every application twice and have the choice running it in sp or dp.
Obviously this is not esential? Where can I find out what needs scalar?

I will try Open-MX and keep you up. Is this your new favourite instead of GAMMA?

Regards.

mattijs · November 24, 2008, 17:22

Yes it is essential to compile everything OpenFOAM in single and double precision. There is no need though to compile OpenMPI in single or double precision.

We haven't compiled the GAMMA driver for a while. Open-MX should be much easier to install (no kernel patching) and works alongside normal ethernet traffic so you don't need a separate network as you do for GAMMA.

bastil · January 17, 2009, 19:53

Sorry for the delay I just started looking into it.
Building Open-MX is no problem. However I do not really understand how to use it. As far as I understand I need to use OpenMPI as a middleware. I tried to rebuild the OpenFOAM 1.2.6-openmpi with mx-support using the open-mx driver. However, this failed. I am not sure if this is the way it is supposed to run....

Regards BastiL

aunola · January 18, 2009, 06:23

Yes, you will need Open-MX compatible middleware, ie. an MPI implementation with a backend that supports the Open-MX library.

http://open-mx.gforge.inria.fr/

bastil · January 18, 2009, 06:56

Thanks Martin,

as far as I understand OpenMPI should do this, right? However I do not understand how to do? Is it essential to rebuild OpenMPI with OpenMX-Support as it is for MX-Support? (I use the 1.26 delivered with OpenFOAM 1.5). If yes, how to do that. If not, how do I use OpenMX with OpenMPI middleware? I have not found any documentation about that.

Regards.

aunola · January 18, 2009, 09:20

I am an MPICH2 man myself, and have never used OpenMPI, so I cannot give you a definitive answer. However, I would expect you'd have to build/rebuild OpenMPI with support for MX, by specifying where the OpenMX header files and libraries are located.

Most likely this is done with a command line switch when running configure. I.e. something like

./configure --with-mx=<path>

You will have to consult a "how to build OpenMPI" site for the details.

bastil · January 18, 2009, 10:18

Yes thats what I tried to do. However OpenMPI does not build taht way against Open-MX but it does against MX.

aunola · January 18, 2009, 13:18

Look in your configure result file. Does it indicate that it will build with MX support using your Open-MX install?

Maybe this can be of some help also:

http://www.open-mpi.org/faq/?categor...ding#build-p2p

bastil · January 18, 2009, 14:36

Thanks, I was aware of this. It works now. The problem was that "make install" of Open-MX had not been performed. Now everything works fine.

BastiL

November 20, 2008, 17:41	Hi all, I have OF installed	#1
bastil Senior Member BastiL Join Date: Mar 2009 Posts: 530 Rep Power: 20	Hi all, I have OF installed with both sp and dp. Therefore I change the environment and recompile everything a second time with the other precision. This works but gives me everything twice. Is this essential? Additionally I am planing to use a second mpi implementation (thinging of MPIGamma). Does this mean four installations now? (sp/dp for both of the mpis) Or is there another way to switch mpi implementation? Thanks. BastiL

November 21, 2008, 04:39	Anything that uses 'scalar' in	#2
mattijs Senior Member Mattijs Janssens Join Date: Mar 2009 Posts: 1,419 Rep Power: 26	Anything that uses 'scalar' in the source code needs recompilation. There is no need to recompile openmpi if you switch precision. If you use a second mpi implementation you'll need to recompile the Pstream library so you'll have two (SP and DP) of all OpenFOAM libraries and 4 libPstream.so. Try Open-MX (http://gforge.inria.fr/projects/open-mx/) instead of MPIGamma. Please post some results if you are successful.

November 21, 2008, 19:53	I did not get that. So far I r	#3
bastil Senior Member BastiL Join Date: Mar 2009 Posts: 530 Rep Power: 20	I did not get that. So far I recompile ALL aplications for sp. First I compile everything in dp and afterwards in sp. So I get every application twice and have the choice running it in sp or dp. Obviously this is not esential? Where can I find out what needs scalar? I will try Open-MX and keep you up. Is this your new favourite instead of GAMMA? Regards.

November 24, 2008, 17:22	Yes it is essential to compile	#4
mattijs Senior Member Mattijs Janssens Join Date: Mar 2009 Posts: 1,419 Rep Power: 26	Yes it is essential to compile everything OpenFOAM in single and double precision. There is no need though to compile OpenMPI in single or double precision. We haven't compiled the GAMMA driver for a while. Open-MX should be much easier to install (no kernel patching) and works alongside normal ethernet traffic so you don't need a separate network as you do for GAMMA.

January 17, 2009, 19:53	Sorry for the delay I just sta	#5
bastil Senior Member BastiL Join Date: Mar 2009 Posts: 530 Rep Power: 20	Sorry for the delay I just started looking into it. Building Open-MX is no problem. However I do not really understand how to use it. As far as I understand I need to use OpenMPI as a middleware. I tried to rebuild the OpenFOAM 1.2.6-openmpi with mx-support using the open-mx driver. However, this failed. I am not sure if this is the way it is supposed to run.... Regards BastiL

January 18, 2009, 06:23	Yes, you will need Open-MX com	#6
aunola Member Martin Aunskjaer Join Date: Mar 2009 Location: Denmark Posts: 53 Rep Power: 17	Yes, you will need Open-MX compatible middleware, ie. an MPI implementation with a backend that supports the Open-MX library. http://open-mx.gforge.inria.fr/

January 18, 2009, 06:56	Thanks Martin, as far as I	#7
bastil Senior Member BastiL Join Date: Mar 2009 Posts: 530 Rep Power: 20	Thanks Martin, as far as I understand OpenMPI should do this, right? However I do not understand how to do? Is it essential to rebuild OpenMPI with OpenMX-Support as it is for MX-Support? (I use the 1.26 delivered with OpenFOAM 1.5). If yes, how to do that. If not, how do I use OpenMX with OpenMPI middleware? I have not found any documentation about that. Regards.

January 18, 2009, 09:20	I am an MPICH2 man myself, and	#8
aunola Member Martin Aunskjaer Join Date: Mar 2009 Location: Denmark Posts: 53 Rep Power: 17	I am an MPICH2 man myself, and have never used OpenMPI, so I cannot give you a definitive answer. However, I would expect you'd have to build/rebuild OpenMPI with support for MX, by specifying where the OpenMX header files and libraries are located. Most likely this is done with a command line switch when running configure. I.e. something like ./configure --with-mx=<path> You will have to consult a "how to build OpenMPI" site for the details.

January 18, 2009, 10:18	Yes thats what I tried to do.	#9
bastil Senior Member BastiL Join Date: Mar 2009 Posts: 530 Rep Power: 20	Yes thats what I tried to do. However OpenMPI does not build taht way against Open-MX but it does against MX.

January 18, 2009, 13:18	Look in your configure result	#10
aunola Member Martin Aunskjaer Join Date: Mar 2009 Location: Denmark Posts: 53 Rep Power: 17	Look in your configure result file. Does it indicate that it will build with MX support using your Open-MX install? Maybe this can be of some help also: http://www.open-mpi.org/faq/?categor...ding#build-p2p

January 18, 2009, 14:36	Thanks, I was aware of this. I	#11
bastil Senior Member BastiL Join Date: Mar 2009 Posts: 530 Rep Power: 20	Thanks, I was aware of this. It works now. The problem was that "make install" of Open-MX had not been performed. Now everything works fine. BastiL