[Tobi]

Dear all,

like in this thread: http://www.cfd-online.com/Forums/har...ec-memory.html

Bruno mentioned that you will have a very nice performance if you set up OpenFOAM in an optimal way.

I always build OpenFOAM out of git repository and with ./Allwmake
So my question is, if that is a optimal way or do I have to make changes in some files to build it optimal?

Or do I just have to use the actual (here mentioned the gcc 4.7 version) versions of compiler and other packages?


Thanks in advance
Tobi

[wyldckat]

Hi Tobi,

The quick instructions are (after you've cloned the repo and have the ThirdParty folder in place):

1. Create an alias in your "~/.bashrc" file, with the following code and make sure you haven't got other OpenFOAM environments being sourced (use aliases instead):
   

   alias of22x='source $HOME/OpenFOAM/OpenFOAM-2.2.x/etc/bashrc WM_COMPILER=Gcc47 WM_MPLIB=SYSTEMOPENMPI WM_NCOMPPROCS=4'

   Save and close the file "~/.bashrc".
   "4" is the number of cores to be used for compiling.
2. Start a new terminal and run the command of22x to start the environment and go into the main folder:
   

   of22x
   foam

3. Now edit these files:
   

   wmake/rules/linux64Gcc47/cOpt
   wmake/rules/linux64Gcc47/c++Opt

4. The lines "cOPT" and "c++OPT" are the ones you'll append the dedicated tuning options for your machine.
   • For example, on my machine, I used what I mentioned here http://www.cfd-online.com/Forums/blo...-amd-cpus.html :
     

     cOPT = -march=amdfam10 -O2 -pipe
     c++OPT = -march=amdfam10 -O2 -pipe

     The "-O2" option focuses on speed (doesn't care about how big the binaries get) and "-O3" focuses on a balancing of size and speed (smaller sometimes is faster).
   • On your case, with the new machines, according to the blog post I mentioned on that other thread you wrote about, it should be something like:
     

     cOPT = -O2 -march=bdver1 -mtune=bdver1
     c++OPT = -O2 -march=bdver1 -mtune=bdver1

     You'll have to try which one of "-O3" or "-O2" is best for you.

Now, experimenting can get a bit annoying, but essential. You can do several builds for the same installation, by doing these steps:

1. Having more than one alias, depending on options:
   

   alias of22x='source $HOME/OpenFOAM/OpenFOAM-2.2.x/etc/bashrc WM_COMPILER=Gcc47 WM_MPLIB=SYSTEMOPENMPI WM_NCOMPPROCS=4'
   alias of22x_v2='source $HOME/OpenFOAM/OpenFOAM-2.2.x/etc/bashrc WM_COMPILER=Gcc47_v2 WM_MPLIB=SYSTEMOPENMPI WM_NCOMPPROCS=4'
   alias of22x_v3='source $HOME/OpenFOAM/OpenFOAM-2.2.x/etc/bashrc WM_COMPILER=Gcc47_v3 WM_MPLIB=SYSTEMOPENMPI WM_NCOMPPROCS=4'

2. Then clone the rule folder:
   

   foam
   cp -r wmake/rules/linux64Gcc47 wmake/rules/linux64Gcc47_v2
   cp -r wmake/rules/linux64Gcc47 wmake/rules/linux64Gcc47_v3

3. Then start a new terminal for each environment and run the respective alias command on each terminal, namely:
   

   of22x
   of22x_v2
   of22x_v3

There are several more tuning options you can try. For the Opteron 6234 you mentioned on the other thread: http://developer.amd.com/wordpress/m...f-62004200.pdf

And yes, you'll have to do a lot of testing, as to ascertain which flags work the best for you, specially when running in parallel!


By the way, I specified in the aliases to use the system's Open-MPI, because allegedly it's the one properly built for the system... assuming of course you've already installed the one for your system

Best regards,
Bruno

[Tobi]

Originally Posted by wyldckat

1. Now edit these files:
   

   wmake/rules/linux64Gcc47/cOpt
   wmake/rules/linux64Gcc47/c++Opt

I think these lines are telling OF that the binaries should be written in the different folders

Originally Posted by wyldckat

And why to go through all of this trouble? Because I want to know!

Thats the best answer!
So I think a good option to know which flag options are the best is to test it with snappyHexMesh like you did or would it be better to make a calculation?


Flags for compiler are new for me but I realize that with the best settings it would be a very greate deal.

At least, where do you know the correct flags like:

cOPT = -O2 -march=bdver1 -mtune=bdver1 c++OPT = -O2 -march=bdver1 -mtune=bdver1

Thanks in advance Bruno (you are my ideal)

[Tobi]

Additionally - I think you have the gcc flags find out here: http://developer.amd.com/wordpress/m...f-62004200.pdf

Do you know if I need OpenMPI? Or vectorization, maybe -ffast-math?

Or do I have to figure it out by my self?

Thanks bruno!

[wyldckat]

Hi Tobi,

The flags I mentioned are from this blog post: http://developer.amd.com/community/b...es-processors/
I had mentioned this at the end of this post: http://www.cfd-online.com/Forums/har...tml#post451659

Personally, I do not advise you to use the "-ffast-math". You should only use it if you are sure about what you're simulating, namely what numeric sensitivity it has got.
Beyond that, you'll have to try yourself. You can load all of the flags mentioned on that other optimizing document and then start pruning out the ones that might not matter.

Open-MPI is the usual MPI toolbox people are using on Linux Distributions and that those Distributions usually offer without the need for building one yourself. And you'll need an MPI for running in parallel, because OpenFOAM doesn't use OpenMP.

As for benchmarking: snappyHexMesh is only one example. You should try with all of the kinds of simulations you'll be doing. Meshing and simulation are both essential.

Best regards,
Bruno

edit: As for MPI tuning (not building), check my blog post Notes about running OpenFOAM in parallel

[Tobi]

Further more, if I compile I get the following lines:

SOURCE=fields/fvPatchFields/derived/phaseHydrostaticPressure/phaseHydrostaticPressureFvPatchScalarField.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -O3  -DNoRepository -ftemplate-depth-100 -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/triSurface/lnInclude -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/meshTools/lnInclude  -IlnInclude -I. -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64Gcc47DPOpt/phaseHydrostaticPressureFvPatchScalarField.o

I compile with -O3 ?

Tobi

[wyldckat]

Hi Tobi,

Did you modify any of the "wmake" rule files? And if you did, what exactly did you do?

Best regards,
Bruno

[Tobi]

Originally Posted by wyldckat

Hi Tobi,

Did you modify any of the "wmake" rule files? And if you did, what exactly did you do?

Best regards,
Bruno

Problem solved.
In the bashrc I used in v2 the Gcc47 instead of the Gcc47_v2.
Now its working!

I see the flags in the compilation lines!
Thanks for all your support bruno.

[Tobi]

Hi bruno,

~: gcc --version

gcc (OpenFOAM) 4.7.2
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I think the first step for compiling in optimal way is done.
Now I am going to test it out.

[Tobi]

Okay - OpenFOAM is now compiled with the normal standard settings!

Now I made the changes you told befor and I get the following message:

shor-ty@cfd:~$ of22_v3

Warning in /home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/etc/config/settings.sh:
    Unknown OpenFOAM compiler type 'Gcc47_v3'
    Please check your settings

Can you help me?
I think it depend on the fact, that I am using 4.7.2 now instead of the standard compiler of ubuntu.

[wyldckat]

Ooops! I forgot about this line of code in "etc/config/settings.sh": https://github.com/OpenFOAM/OpenFOAM...ttings.sh#L249

Gcc47 | Gcc47++0x)

change it to:

Gcc47* | Gcc47++0x)

and run the respective alias command again.

[Tobi]

Originally Posted by wyldckat

Ooops! I forgot about this line of code in "etc/config/settings.sh": https://github.com/OpenFOAM/OpenFOAM...ttings.sh#L249

Gcc47 | Gcc47++0x)

change it to:

Gcc47* | Gcc47++0x)

and run the respective alias command again.

Well thats a very nice solution ... Thanks bruno.

Everything is working now ... Sometimes I should use my head but in that topic its hard for me to be sure that everything is correct then. . .

[Tobi]

So now I think its working good.
Still one question is remaining:

MPI is compiled with other flags:

gcc -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -I../../include -I../libscotch acpl.c -o acpl -L../../lib -lscotch -lscotch -lscotcherrexit -lz -lm -lrt
gcc -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -I../../include -I../libscotch amk_ccc.c -o amk_ccc -L../../lib -lscotch -lscotch -lscotcherrexit -lz -lm -lrt
gcc -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -I../../include -I../libscotch amk_fft2.c -o amk_fft2 -L../../lib -lscotch -lscotch -lscotcherrexit -lz -lm -lrt
gcc -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -I../../include -I../libscotch amk_grf.c -o amk_grf -L../../lib -lscotch -lscotch -lscotcherrexit -lz -lm -lrt

But OF code is compiled with the additional flags:

SOURCE=meshes/polyMesh/globalMeshData/globalPoints.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -O2 -march=bdver1 -pipe -mtune=bdver1  -DNoRepository -ftemplate-depth-100 -IMake/linux64Gcc47_v3DPOpt -IlnInclude -I. -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64Gcc47_v3DPOpt/globalPoints.o

Should I make changes that mpi will compile with the optimization flags too?

Thanks in advance
Tobi

[wyldckat]

Hi Tobi,

Actually, the first block of output is regarding the build of the Scotch libraries and applications. Honestly, I don't think you need to bother much about the Scotch libraries, since it's only used in the decomposition processes.

Best regards,
Bruno

[Tobi]

Originally Posted by wyldckat

Hi Tobi,

Actually, the first block of output is regarding the build of the Scotch libraries and applications. Honestly, I don't think you need to bother much about the Scotch libraries, since it's only used in the decomposition processes.

Best regards,
Bruno

Oh yes you are right, the MPI is build with the flags I looked in the wrong lines. What I found in the building progress were missing files ...

Note: ignore spurious warnings about missing mpicxx.h headers

wclean mpi
wmake libso mpi
wmakeLnInclude: linking include files to ./lnInclude
Making dependency list for source file UOPwrite.C
Making dependency list for source file UIPread.C
could not open file ompi/mpi/cxx/pmpicxx.h for source file UOPwrite.C
Making dependency list for source file UPstream.C
could not open file ompi/mpi/cxx/pop_inln.h for source file UOPwrite.C
could not open file ompi/mpi/cxx/pgroup_inln.h for source file UOPwrite.C
could not open file ompi/mpi/cxx/pstatus_inln.h for source file UOPwrite.C
could not open file ompi/mpi/cxx/prequest_inln.h for source file UOPwrite.C
could not open file ompi/mpi/cxx/pmpicxx.h for source file UIPread.C
Making dependency list for source file PstreamGlobals.C
could not open file ompi/mpi/cxx/pmpicxx.h for source file PstreamGlobals.C
could not open file ompi/mpi/cxx/pmpicxx.h for source file UPstream.C
could not open file ompi/mpi/cxx/pop_inln.h for source file UIPread.C
could not open file ompi/mpi/cxx/pgroup_inln.h for source file UIPread.C
could not open file ompi/mpi/cxx/pstatus_inln.h for source file UIPread.C
could not open file ompi/mpi/cxx/prequest_inln.h for source file UIPread.C
could not open file ompi/mpi/cxx/pop_inln.h for source file PstreamGlobals.C
could not open file ompi/mpi/cxx/pgroup_inln.h for source file PstreamGlobals.C
could not open file ompi/mpi/cxx/pstatus_inln.h for source file PstreamGlobals.C
could not open file ompi/mpi/cxx/prequest_inln.h for source file PstreamGlobals.C
could not open file ompi/mpi/cxx/pop_inln.h for source file UPstream.C
could not open file ompi/mpi/cxx/pgroup_inln.h for source file UPstream.C
could not open file ompi/mpi/cxx/pstatus_inln.h for source file UPstream.C
could not open file ompi/mpi/cxx/prequest_inln.h for source file UPstream.C
SOURCE=UOPwrite.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -O2 -march=bdver1 -pipe -mtune=bdver1  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -IlnInclude -I. -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64Gcc47_v3DPOptSYSTEMOPENMPI/UOPwrite.o
SOURCE=UIPread.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -O2 -march=bdver1 -pipe -mtune=bdver1  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -IlnInclude -I. -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64Gcc47_v3DPOptSYSTEMOPENMPI/UIPread.o
SOURCE=UPstream.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -O2 -march=bdver1 -pipe -mtune=bdver1  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -IlnInclude -I. -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64Gcc47_v3DPOptSYSTEMOPENMPI/UPstream.o
SOURCE=PstreamGlobals.C ;  g++ -m64 -Dlinux64 -DWM_DP -Wall -Wextra -Wno-unused-parameter -Wold-style-cast -O2 -march=bdver1 -pipe -mtune=bdver1  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi -pthread -IlnInclude -I. -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OpenFOAM/lnInclude -I/home/shor-ty/OpenFOAM/OpenFOAM-2.2.x/src/OSspecific/POSIX/lnInclude   -fPIC -c $SOURCE -o Make/linux64Gcc47_v3DPOptSYSTEMOPENMPI/PstreamGlobals.o

but it seems that MPI is compiling the missing files after the lines, isn't it?

[Tobi]

Just a feedback:


FX-8150 (Normal tuning - not overclocked) and OpenFOAM-2.2.x:

- Tutorial motorbike with 1 core: 292s 290s 290s | gcc 4.7.2 -O3 (standard)
- Tutorial motorbike with 1 core: 271s 270s 270s | gcc 4.7.2 -O2 -march=bd1ver -mtune=bd1ver

- Tutorial motorbike with 6 cores: 113s 111s | gcc 4.7.2 -O3 (standard)
- Tutorial motorbike with 6 cores: 108s 108s | gcc 4.7.2 -O2 -march=bd1ver -mtune=bd1ver




Bruno - at the end!
Thanks a lot for your common help. Without you I would never think about the option of a other compilation

Next week I will get my new server. After I set up everything I will build a more complex test case as the motorbike and make some test with different settings and give a feedback.

[Tobi]

Hi all,

I just want let you know that my test benchmark geometry is ready for meshing

In the attachment you find the geometry...

I keep you posted.

Tobi