|
[Sponsors] |
February 24, 2006, 05:35 |
Hello!
I could use some hel
|
#1 |
New Member
Alexander Rudert
Join Date: Mar 2009
Location: Freiberg, Saxony, Germany
Posts: 1
Rep Power: 0 |
Hello!
I could use some help with starting Foam/LAM on a linux cluster with infinband interconnect. So far i can only use the ethernet between the nodes. I tried lamboot -v -ssi rpi ib hostfile, but i only get the message ib module not found. Does somebody know this problem? And another question: How much better is infiniband compared to gigabit ethernet? Has somebody experience with infiniband? |
|
February 24, 2006, 05:56 |
I guess you shouldnt use LAM a
|
#2 |
Senior Member
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 237
Rep Power: 18 |
I guess you shouldnt use LAM as mpi library, but you should use mvapich, the mpi implementation for infiniband.
I'll try it as soon as I can, and I'll let you know. Regarding Infiniband, it's really much better than Gigabit. The transfer rate of infiniband, measured from application, is about 960MB/s (Megabyte/s), and the latency time is very, very low. You can therefore use a high number of processors in a parallel run. As an example, my application saturates gigabit with 12 processes, while you can scale up to 48 and more with infiniband, reducing wallclock computational time. It worths to spend some time in understanding how to use infiniband, really. |
|
February 24, 2006, 06:51 |
I have a similar issue. The la
|
#3 |
Senior Member
Eugene de Villiers
Join Date: Mar 2009
Posts: 725
Rep Power: 21 |
I have a similar issue. The lam compilation shipped with OpenFOAM does not include support for Myrinet, but I know that lam does have a native Myrinet module. Does anyone know which options you need to invoke to compile these modules into lam?
|
|
February 24, 2006, 07:22 |
you have to build LAM from sou
|
#4 |
Member
stefan
Join Date: Mar 2009
Posts: 96
Rep Power: 17 |
you have to build LAM from source for native support of Myrinet.
you can do this with the option gm: ./configure --prefix=/your/lam/dir --with-gm=/path/to/gm ... |
|
February 24, 2006, 07:28 |
We have recently made a test G
|
#5 |
Senior Member
Håkan Nilsson
Join Date: Mar 2009
Location: Gothenburg, Sweden
Posts: 205
Rep Power: 18 |
We have recently made a test Gigabit v.s. Infiniband. I will post a short report on that on the discussion page soon. The test does not show a significant improvement when using Infiniband, at least for the case that was used in the test. The test was a simpleFoam computation with 10^6 cells distributed on 1-16 CPUs. In the higher range of CPU numbers there might be some improvement when using Infiniband, and if using even more CPUs Infiniband might be better relative to Gigabit. For 16 CPU's the parallel efficiency is however only 0.45 for Gigabit and 0.51 for Infiniband, which is not really great.
My collaboration partner at Gridcore (www.gridcore.se) did the technical stuff. He just re-compiled pstream.so and liked it to the correct Infiniband mpi-library. Håkan. |
|
February 24, 2006, 07:35 |
The flag mentioned above is st
|
#6 |
Member
stefan
Join Date: Mar 2009
Posts: 96
Rep Power: 17 |
The flag mentioned above is still usable but deprecated, please use the "--with-rpi-gm=PATH" flag instead!
For further details have a look at the LAM/MPI Installation Guide. |
|
February 24, 2006, 07:51 |
we hare recently made a test G
|
#7 |
Senior Member
Francesco Del Citto
Join Date: Mar 2009
Location: Zürich Area, Switzerland
Posts: 237
Rep Power: 18 |
we hare recently made a test Gigabit vs Infiniband on an AMD Opteron Cluster with another application (a Navier-Stokes software for engine application with structured grids and moving meshes), written in Fortran77/Fortran95 on an mpi base, and the differences between the networks are really relevant.
Depending on the case, we have saturated Gigabit with 12 processors (speedup<1), while with infiniband the speedup grows till 24 processors used, which was the maximum number of processes we have tested. With a bigger test case, with 16 processors the time spent was 113 min with gigabit and 75 with infiniband. We couldn't test more than 16 processors with gigabit, but using Infiniband the speedup always grows, till 48 processors, which is the total number of CPUs of the cluster. With combustion, moreover, the value of the speedup obviously grows, with an efficency of parallelism up to 63% with 30 processors. I've no experience with OpenFOAM over Infiniband, but with some kind of applications, this kind of network is what really makes the difference. |
|
February 27, 2006, 07:00 |
Dear Håkan,
Are you compari
|
#8 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
Dear Håkan,
Are you comparing versus a sequential run? Or is this the %cpu utilization? Is the difference due to the number of sweeps being much larger? Did you try various decompositions? Hierarchical with almost equal number in x,y,z is a good starting point. Did you try scheduledTransfer on/off? |
|
February 27, 2006, 08:23 |
Dear 'Infiniband Interest Grou
|
#9 |
Senior Member
Håkan Nilsson
Join Date: Mar 2009
Location: Gothenburg, Sweden
Posts: 205
Rep Power: 18 |
Dear 'Infiniband Interest Group',
You can find a short report on my test of Infiniband here: Infiniband_vs_Gigabit.pdf.gz I would be very happy to get some comments on how the test was made. I will soon make similar comparisons with InfiniPath, and I would like to make as good a test as possible. I hope that the document answers the questions from Mattijs. Some additional answers follow: The number of iterations for p are slightly changed with the decomposition, typically: 229,251,258,261,271 for 1,2,4,8,16 CPU's. The other equations take 1 iteration for all decompositions. Is this what you are referring to 'number of sweeps'? I don't know if the actual linear solver (ICCG) does more sweeps per iteration. I did not try various decompositions, only load balanced Metis. This test is only a small project on the side of what I should really be doing, and I don't have the time to try all options. There are many ways of decomposing, but I think that the Metis decomposition should be a good starting point. I did not try scheduledTransfer on/off. Can you please tell me what this will affect, and how to do it? Thank you on advance, Mattijs! I have asked before about some hints on how to make the comparisons the best way, and that question is still open. Håkan. |
|
February 27, 2006, 08:53 |
scheduledTransfer makes sure o
|
#10 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
scheduledTransfer makes sure one side (process) of a processor patch is in receive mode while the other one is sending. Means you can use mpi_send instead of mpi_bsend. Sometimes benificial (no buffer, no thread?), sometimes detrimental (more chance of everyone waiting for the slowest).
|
|
February 27, 2006, 09:49 |
Hi Håkan!
I recently got a
|
#11 |
Assistant Moderator
Bernhard Gschaider
Join Date: Mar 2009
Posts: 4,225
Rep Power: 51 |
Hi Håkan!
I recently got a benchmark paper by Cisco where they compared Fluent on GEthernet with Infiniband. For the case that was similar to yours (850k cells) for the number of processors you had available their results are very similar to yours: the speedup with EN and IB are almost the same. For larger numbers of processors (they had 64 nodes) IB performs better. The other results (for smaller runs) suggest in my opinion that for Infiniband to be of use you either have to have small runs and/or a large number (>>10) of nodes. I'd say that the major advantage of IB is not the bandwidth but the low latency. (If you're interested I can send you the PDF)
__________________
Note: I don't use "Friend"-feature on this forum out of principle. Ah. And by the way: I'm not on Facebook either. So don't be offended if I don't accept your invitation/friend request |
|
May 24, 2006, 03:33 |
Hi Infiniband users!
I have
|
#12 |
Senior Member
Jens Klostermann
Join Date: Mar 2009
Posts: 117
Rep Power: 17 |
Hi Infiniband users!
I have problems compiling lam7.1.1 with Infiniband support. What I did (all what is in Allwmake file in the src directory): gmake distclean rm -rf $LAM_ARCH_PATH ./configure \ --prefix=$LAM_ARCH_PATH \ --with-rpi-ib=/usr/ibgd/driver/infinihost \ --enable-shared \ --disable-static \ --without-romio \ --without-mpi2cpp \ --without-profiling \ --without-fc gmake So when I compile it I get the following error gmake[6]: Entering directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src' if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_ack.lo -MD -MP -MF ".deps/ssi_rpi_ib_ack.Tpo" -c -o ssi_rpi_ib_ack.lo ssi_rpi_ib_ack.c; \ then mv -f ".deps/ssi_rpi_ib_ack.Tpo" ".deps/ssi_rpi_ib_ack.Plo"; else rm -f ".deps/ssi_rpi_ib_ack.Tpo"; exit 1; fi mkdir .libs gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_ack.lo -MD -MP -MF .deps/ssi_rpi_ib_ack.Tpo -c ssi_rpi_ib_ack.c -fPIC -DPIC -o .libs/ssi_rpi_ib_ack.o if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_actions.lo -MD -MP -MF ".deps/ssi_rpi_ib_actions.Tpo" -c -o ssi_rpi_ib_actions.lo ssi_rpi_ib_actions.c; \ then mv -f ".deps/ssi_rpi_ib_actions.Tpo" ".deps/ssi_rpi_ib_actions.Plo"; else rm -f ".deps/ssi_rpi_ib_actions.Tpo"; exit 1; fi gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_actions.lo -MD -MP -MF .deps/ssi_rpi_ib_actions.Tpo -c ssi_rpi_ib_actions.c -fPIC -DPIC -o .libs/ssi_rpi_ib_actions.o ssi_rpi_ib_actions.c: In function 'send_peer_fc_info': ssi_rpi_ib_actions.c:1202: warning: right shift count >= width of type ssi_rpi_ib_actions.c:1244: warning: left shift count >= width of type if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_bitmap.lo -MD -MP -MF ".deps/ssi_rpi_ib_bitmap.Tpo" -c -o ssi_rpi_ib_bitmap.lo ssi_rpi_ib_bitmap.c; \ then mv -f ".deps/ssi_rpi_ib_bitmap.Tpo" ".deps/ssi_rpi_ib_bitmap.Plo"; else rm -f ".deps/ssi_rpi_ib_bitmap.Tpo"; exit 1; fi gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_bitmap.lo -MD -MP -MF .deps/ssi_rpi_ib_bitmap.Tpo -c ssi_rpi_ib_bitmap.c -fPIC -DPIC -o .libs/ssi_rpi_ib_bitmap.o if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_dreg.lo -MD -MP -MF ".deps/ssi_rpi_ib_dreg.Tpo" -c -o ssi_rpi_ib_dreg.lo ssi_rpi_ib_dreg.c; \ then mv -f ".deps/ssi_rpi_ib_dreg.Tpo" ".deps/ssi_rpi_ib_dreg.Plo"; else rm -f ".deps/ssi_rpi_ib_dreg.Tpo"; exit 1; fi gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_dreg.lo -MD -MP -MF .deps/ssi_rpi_ib_dreg.Tpo -c ssi_rpi_ib_dreg.c -fPIC -DPIC -o .libs/ssi_rpi_ib_dreg.o ssi_rpi_ib_dreg.c:40: error: static declaration of 'lam_ssi_rpi_ib_env_mempool' follows non-static declaration ./rpi_ib_dreg.h:24: error: previous declaration of 'lam_ssi_rpi_ib_env_mempool' was here gmake[6]: *** [ssi_rpi_ib_dreg.lo] Error 1 gmake[6]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src' gmake[5]: *** [all] Error 2 gmake[5]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src' gmake[4]: *** [all-recursive] Error 1 gmake[4]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib' gmake[3]: *** [all-recursive] Error 1 gmake[3]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi' gmake[2]: *** [all-recursive] Error 1 gmake[2]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share' gmake: *** [all-recursive] Error 1 Any help is apreciated. Thanks Jens |
|
May 24, 2006, 03:34 |
Hi Infiniband users!
I have
|
#13 |
Senior Member
Jens Klostermann
Join Date: Mar 2009
Posts: 117
Rep Power: 17 |
Hi Infiniband users!
I have problems compiling lam7.1.1 with Infiniband support. What I did (all what is in Allwmake file in the src directory): gmake distclean rm -rf $LAM_ARCH_PATH ./configure \ --prefix=$LAM_ARCH_PATH \ --with-rpi-ib=/usr/ibgd/driver/infinihost \ --enable-shared \ --disable-static \ --without-romio \ --without-mpi2cpp \ --without-profiling \ --without-fc gmake So when I compile it I get the following error gmake[6]: Entering directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src' if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_ack.lo -MD -MP -MF ".deps/ssi_rpi_ib_ack.Tpo" -c -o ssi_rpi_ib_ack.lo ssi_rpi_ib_ack.c; \ then mv -f ".deps/ssi_rpi_ib_ack.Tpo" ".deps/ssi_rpi_ib_ack.Plo"; else rm -f ".deps/ssi_rpi_ib_ack.Tpo"; exit 1; fi mkdir .libs gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_ack.lo -MD -MP -MF .deps/ssi_rpi_ib_ack.Tpo -c ssi_rpi_ib_ack.c -fPIC -DPIC -o .libs/ssi_rpi_ib_ack.o if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_actions.lo -MD -MP -MF ".deps/ssi_rpi_ib_actions.Tpo" -c -o ssi_rpi_ib_actions.lo ssi_rpi_ib_actions.c; \ then mv -f ".deps/ssi_rpi_ib_actions.Tpo" ".deps/ssi_rpi_ib_actions.Plo"; else rm -f ".deps/ssi_rpi_ib_actions.Tpo"; exit 1; fi gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_actions.lo -MD -MP -MF .deps/ssi_rpi_ib_actions.Tpo -c ssi_rpi_ib_actions.c -fPIC -DPIC -o .libs/ssi_rpi_ib_actions.o ssi_rpi_ib_actions.c: In function 'send_peer_fc_info': ssi_rpi_ib_actions.c:1202: warning: right shift count >= width of type ssi_rpi_ib_actions.c:1244: warning: left shift count >= width of type if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_bitmap.lo -MD -MP -MF ".deps/ssi_rpi_ib_bitmap.Tpo" -c -o ssi_rpi_ib_bitmap.lo ssi_rpi_ib_bitmap.c; \ then mv -f ".deps/ssi_rpi_ib_bitmap.Tpo" ".deps/ssi_rpi_ib_bitmap.Plo"; else rm -f ".deps/ssi_rpi_ib_bitmap.Tpo"; exit 1; fi gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_bitmap.lo -MD -MP -MF .deps/ssi_rpi_ib_bitmap.Tpo -c ssi_rpi_ib_bitmap.c -fPIC -DPIC -o .libs/ssi_rpi_ib_bitmap.o if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID="" -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_dreg.lo -MD -MP -MF ".deps/ssi_rpi_ib_dreg.Tpo" -c -o ssi_rpi_ib_dreg.lo ssi_rpi_ib_dreg.c; \ then mv -f ".deps/ssi_rpi_ib_dreg.Tpo" ".deps/ssi_rpi_ib_dreg.Plo"; else rm -f ".deps/ssi_rpi_ib_dreg.Tpo"; exit 1; fi gcc -DHAVE_CONFIG_H -I. -I. -I. -DLAM_SSI_RPI_IB_TINYMSGLEN=32768 -DLAM_SSI_RPI_IB_PORT=-1 -DLAM_SSI_RPI_IB_HCA_ID= -DLAM_SSI_RPI_IB_NUM_ENVELOPES=64 -I../../../../../share/include -I../../../../../share/include -I../../../../../share/include -DLAM_BUILDING=1 -DLAM_BUILDING=1 -I/usr/ibgd/driver/infinihost/include -O3 -m64 -fPIC -MT ssi_rpi_ib_dreg.lo -MD -MP -MF .deps/ssi_rpi_ib_dreg.Tpo -c ssi_rpi_ib_dreg.c -fPIC -DPIC -o .libs/ssi_rpi_ib_dreg.o ssi_rpi_ib_dreg.c:40: error: static declaration of 'lam_ssi_rpi_ib_env_mempool' follows non-static declaration ./rpi_ib_dreg.h:24: error: previous declaration of 'lam_ssi_rpi_ib_env_mempool' was here gmake[6]: *** [ssi_rpi_ib_dreg.lo] Error 1 gmake[6]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src' gmake[5]: *** [all] Error 2 gmake[5]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib/src' gmake[4]: *** [all-recursive] Error 1 gmake[4]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi/ib' gmake[3]: *** [all-recursive] Error 1 gmake[3]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi/rpi' gmake[2]: *** [all-recursive] Error 1 gmake[2]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share/ssi' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/home/klosterm/OpenFOAM/OpenFOAM-1.3/src/lam-7.1.1/share' gmake: *** [all-recursive] Error 1 Any help is appreciated. Thanks Jens |
|
May 24, 2006, 05:46 |
How about using openmpi? This
|
#14 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
How about using openmpi? This might have more uptodate infiniband. We include 1.0.2a7.
|
|
May 24, 2006, 08:08 |
When I compile openmpi everyth
|
#15 |
Senior Member
Jens Klostermann
Join Date: Mar 2009
Posts: 117
Rep Power: 17 |
When I compile openmpi everything works fine.
When I recompile the the Pstream libs I get no errors, but the following warnings OPwrite.C: In static member function ‘static bool Foam:Pstream::write(int, const char*, std::streamsize, bool)’: OPwrite.C:77: warning: use of old-style cast OPwrite.C:89: warning: use of old-style cast 1. Are these warnings problematic? However I ignored them and did: mpirun -v --mca btl mvapi,self -np 4 --hostfile ompimachinefile -ssh "/home/klosterm/OpenFOAM/OpenFOAM-1.3/applications/bin/linuxAMD64Gcc4DPOpt/inter Foam . dambreak -parallel" with the following error as result: [stokes:25010] [0,0,0] ORTE_ERROR_LOG: Not implemented in file rmgr_urm.c at line 177 [stokes:25010] [0,0,0] ORTE_ERROR_LOG: Not implemented in file rmgr_urm.c at line 365 [stokes:25010] mpirun: spawn failed with errno=-7 Thank you for helping! Jens |
|
May 25, 2006, 06:40 |
Those warnings are not problem
|
#16 |
Senior Member
Mattijs Janssens
Join Date: Mar 2009
Posts: 1,419
Rep Power: 26 |
Those warnings are not problematic.
Never ran openmpi+infiniband so cannot help you there. Can you post the solution if you find it out? |
|
June 12, 2006, 06:00 |
So the problem is solved. Open
|
#17 |
Senior Member
Jens Klostermann
Join Date: Mar 2009
Posts: 117
Rep Power: 17 |
So the problem is solved. OpenFOAM 1.3 is running with infiniband and openmpi. I use the openmpi version 1.2a1r10111, but I think the one which is shipped with OpenFOAM 1.3 is also working.
The problem was that it was not possible to run different versions of mpirun (lam, mpich or openmpi) for different users at the same time. So we hat a lam job running over ethernet which was somehow blocking the mpirun of openmpi. |
|
March 7, 2007, 13:32 |
Hello Jens,
I am very inter
|
#18 |
Member
anne dejoan
Join Date: Mar 2009
Location: madrid, spain
Posts: 66
Rep Power: 17 |
Hello Jens,
I am very interested in having an idea about your compilation of openmpi with infiniband. I am trying several mpi on my infiniband (and let already a message about mvapich) but after having read this threat I tried to compile openmpi included in openfoam distribution and did not succeed. I have some errors put at the end of my present mail. These errors are independant of the option inifiniband in my configure file. How does your openmpi version work? Thanks Anne ------------------------------------ gmake[5]: *** No hay ninguna regla para construir el objetivo `distclean'. Alto. gmake[5]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca/i o/romio/romio' gmake[4]: *** [distclean-recursive] Error 1 gmake[4]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca/i o/romio' gmake[3]: *** [distclean-recursive] Error 1 gmake[3]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca/i o' gmake[2]: *** [distclean-recursive] Error 1 gmake[2]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi/mca' gmake[1]: *** [distclean-recursive] Error 1 gmake[1]: Leaving directory `/afs/ciemat.es/users/u5303/OpenFOAM/OpenFOAM-1.3/src/openmpi-1.0.2a7/ompi' gmake: *** [distclean-recursive] Error 1 -------------------------------------------------- |
|
March 11, 2007, 17:40 |
Hi Anne,
I had a lot of tro
|
#19 |
Senior Member
Jens Klostermann
Join Date: Mar 2009
Posts: 117
Rep Power: 17 |
Hi Anne,
I had a lot of troubles setting up OpenFOAM for infiniband communication. So far it works with openmpi for almost a year now. From my experience I suggest to use always the most recent version of openmpi. Regards Jens |
|
February 8, 2008, 14:43 |
Hi - I'm new to OpenFOAM.
I
|
#20 |
Guest
Posts: n/a
|
Hi - I'm new to OpenFOAM.
I ran damBreak3d using openMPI over Infiniband with the "runnproc" scripts 1-8. Now I want to try 16 processes and more. My questions is: 1) Should I clone the case = nproc8 and create a "nproc16" with FoamX (FoamX won't start on my system) ? 2) Is it best to create a new case? Is there a way from command line to create a new case? 3) I copied the case "nproc8" and changed "8" to 16 but Foam IO had trouble opening the controlDict file |
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
HPMPI Infiniband problem | carsten | OpenFOAM Bugs | 2 | January 25, 2009 16:36 |
OpenFOAM and infiniband | mrangitschdowcom | OpenFOAM Installation | 5 | October 30, 2008 08:47 |
infiniband interconnect help | rjh | FLUENT | 0 | February 4, 2008 12:41 |
OpenFOAM 141 parallel results infiniband vs gigabit vs SMP | msrinath80 | OpenFOAM Running, Solving & CFD | 10 | November 30, 2007 19:11 |
Interesting behavior when using LAM without ib rpi on an infiniband cluster OF 141 | msrinath80 | OpenFOAM Running, Solving & CFD | 0 | October 2, 2007 17:31 |