[wildfire230]

Hi All,

I am trying to compile OpenFOAM 5.0 on the Titan supercomputer, and I am having lots of problems figuring out what compile options to set, etc... I do not have much of a background in architecture specific things, and the help desk has limited experience with OpenFOAM, so I am trying to DIY it.

Titan is a Cray XK7 supercomputer. This is what I have from the system user guide:

Compiling code on Titan (and other Cray machines) is different than compiling code for commodity or beowulf-style HPC linux clusters. Among the most prominent differences:

• Cray provides a sophisticated set of compiler wrappers to ensure that the compile environment is setup correctly. Their use is highly encourged.
• In general, linking/using shared object libraries on compute partitions is not supported.
• Cray systems include many different types of nodes, so some compiles are, in fact, cross-compiles.

Available Compilers

The following compilers are available on Titan:

• PGI, the Portland Group Compiler Suite (default)
• GCC, the GNU Compiler Collection
• CCE, the Cray Compiling Environment
• Intel, Intel Composer XE

7.1. Cray Compiler Wrappers

• (Back to Top) Cray provides a number of compiler wrappers that substitute for the traditional compiler invocation commands. The wrappers call the appropriate compiler, add the appropriate header files, and link against the appropriate libraries based on the currently loaded programming environment module. To build codes for the compute nodes, you should invoke the Cray wrappers via:

• cc To use the C compiler
• CC To use the C++ compiler
• ftn To use the FORTRAN 90 compiler

So, first question, what should these flags be set to:

export WM_NCOMPROCS=8
export WM_COMPILER=CC                                                                        
export WM_CC=cc
export WM_CXX=CC                                                                                 
export WM_MPLIB=CRAYMPI                                                                        
export WM_MPI_CC=cc
export WM_MPI_CXX=CC
export WM_MPIRUN_PROG=aprun
export WM_LDFLAGS=-dynamic
export LDFLAGS=-dynamic
export LD=CC

and, what, if any, modules do I need to have loaded? I believe I have to use cc and CC as the c and c++ compiler options, since they are the compiler wrappers provided. I am not sure at all what to set WM_MPLIB to, but I tried CRAYMPI. As far as I know, it uses MPICH. The system has a default loaded module of cray-mpich/7.5.2 and it also has the pgi compiler pgi/16.10.0 loaded as default.


Just as an example of the errors I am having, when I run ./Allwmake in ThirdParty-5.0, I get the following (truncated) output:

========================================
Start ThirdParty Allwmake
========================================

========================================
Build MPI libraries if required

========================================
Build Scotch decomposition library scotch_6.0.3
    /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3
    scotch header in /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3/include
    scotch libs   in /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib

========================================
Build PTScotch decomposition library scotch_6.0.3 (uses MPI)
    /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3

+ cd scotch_6.0.3/src
+ prefixDIR=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3
+ libDIR=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib/mpich2-1.1.1p1
+ incDIR=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3/include/mpich2-1.1.1p1
+ mkdir -p /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3
+ mkdir -p /autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib/mpich2-1.1.1p1
+ configOpt='prefix=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3 libdir=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64ccDPInt32/lib/mpich2-1.1.1p1 includedir=/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3/include/mpich2-1.1.1p1'
+ '[' -f ../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM ']'
+ rm -f Makefile.inc
+ ln -s ../../etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM Makefile.inc
+ '[' -f Makefile.inc ']'
+ unset configEnv
+ '[' cc '!=' gcc ']'
+ configEnv='CC=cc CCS=cc'
+ make realclean
(cd libscotch ;      make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
rm -f *~ *.o lib*.so parser_yy.c parser_ly.h parser_ll.c *scotch.h *scotchf.h y.output *dummysizes
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
(cd scotch ;         make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
rm -f *~ *.o acpl amk_ccc amk_fft2 amk_grf amk_hy amk_m2 amk_p2 atst gbase gcv *ggath *gmap gmk_hy gmk_m2 gmk_m3 gmk_msh gmk_ub2 gmtst *gord gotst gout *gpart *gscat *gtst mcv mmk_m2 mmk_m3 mord mtst
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
(cd libscotchmetis ; make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
rm -f *~ *.o lib*.so
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
(cd check ;          make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
rm -f *~ *.o
rm -f                 \
                    test_common_thread        \
                    test_scotch_graph_color        \
                    test_scotch_graph_map        \
                    test_scotch_graph_order        \
                    test_scotch_graph_part_ovl    \
                    test_scotch_dgraph_band        \
                    test_scotch_dgraph_check    \
                    test_scotch_dgraph_coarsen    \
                    test_scotch_dgraph_grow        \
                    test_scotch_dgraph_redist    \
                    test_strat_par            \
                    test_strat_seq
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
(cd esmumps ;        make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f *~ common.h *.o lib*.so main_esmumps
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f ../bin/* ../include/* ../lib/*
+ make -j 8 CC=cc CCS=cc ptscotch
(cd libscotch ;      make VERSION=6 RELEASE=0 PATCHLEVEL=3 ptscotch && make ptinstall)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
make CC="cc" CCD="cc"    \
                    scotch.h                \
                    scotchf.h                \
                    libscotch.so                \
                    libscotcherr.so            \
                    libscotcherrexit.so
make[2]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
cc -m64 -fPIC -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -shared -c arch.c -o arch.o
cc -m64 -fPIC -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -shared -c arch_cmplt.c -o arch_cmplt.o



...
...


cc -m64 -fPIC -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict -shared -c bgraph_bipart_df.c -o bgraph_bipart_df.o
pgcc-Error-Unknown switch: -Xlinker
make[2]: *** [dummysizes] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
make[1]: *** [scotch] Error 2
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
make: *** [ptscotch] Error 2
+ make realclean
(cd libscotch ;      make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
rm -f *~ *.o lib*.so parser_yy.c parser_ly.h parser_ll.c *scotch.h *scotchf.h y.output *dummysizes
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotch'
(cd scotch ;         make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
rm -f *~ *.o acpl amk_ccc amk_fft2 amk_grf amk_hy amk_m2 amk_p2 atst gbase gcv *ggath *gmap gmk_hy gmk_m2 gmk_m3 gmk_msh gmk_ub2 gmtst *gord gotst gout *gpart *gscat *gtst mcv mmk_m2 mmk_m3 mord mtst
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/scotch'
(cd libscotchmetis ; make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
rm -f *~ *.o lib*.so
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/libscotchmetis'
(cd check ;          make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
rm -f *~ *.o
rm -f                 \
                    test_common_thread        \
                    test_scotch_graph_color        \
                    test_scotch_graph_map        \
                    test_scotch_graph_order        \
                    test_scotch_graph_part_ovl    \
                    test_scotch_dgraph_band        \
                    test_scotch_dgraph_check    \
                    test_scotch_dgraph_coarsen    \
                    test_scotch_dgraph_grow        \
                    test_scotch_dgraph_redist    \
                    test_strat_par            \
                    test_strat_seq
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
(cd esmumps ;        make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f *~ common.h *.o lib*.so main_esmumps
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f ../bin/* ../include/* ../lib/*

    WARNING: required include file 'ptscotch.h' not found!

========================================
Build Metis decomposition
    optional component Metis was not found
========================================
Build CGAL
    optional component was not found

========================================
Done ThirdParty Allwmake
========================================

I would HUGELY appreciate any help or advice.


Thanks!

[wyldckat]

Greetings wildfire230,

I'll have to be quick, but here is what I'm able to quickly suggest:

1. Check the folders located inside "wmake/rules". The ones I suggest you compare with Meld or Kompare or some other file comparison GUI, are the following folders:
   • "linux64Gcc"
   • "linux64Icc"
   • "linuxPPC64Gcc"
   
   
   1. The reason for this is because it's in there that you should define the rules for compiling OpenFOAM.
   2. And you can choose your own folder name if you change, for example, the value for the "WM_COMPILER" variable in "OpenFOAM-5.0/etc/bashrc".
2. Forget about trying to compile the content of the folder "ThirdParty". Scotch is useless inside the Titan, unless you are planning on meshing and post-processing the cases directly inside the Titan.
3. I haven't tested it properly, but at least theoretically, it is possible to build OpenFOAM as static binaries (merging all of the necessary libraries into each application) simply by running Allwmake with the additional argument "lib", for example:
   

   ./Allwmake lib > log.make 2>&1

4. Do not set the environment variable "WM_NCOMPROCS", due to the following reasons:
   1. The name is incorrect, it's missing one 'P': "WM_NCOMPPROCS"
   2. You don't need to define the number of cores for building OpenFOAM in this way, you can do it by adding the option "-j 8" on the Allwmake command line, e.g.:
      

      ./Allwmake lib -j 8 > log.make 2>&1

   3. I have no idea if the Titan will like your idea of trying to compile with 8 cores... unless it's done within the login node?
5. You do not need to redefine "WM_CC", "WM_CXX", "WM_MPI_CC" and "WM_MPI_CXX", because those would only be used for building Scotch.
6. Using this:
   

   export WM_MPLIB=CRAYMPI

   will only work if you properly configure an entry for it in the file "etc/config.sh/settings", as well as in the "wmake/rules" folder.
7. "WM_LDFLAGS" and similar: those are not used. You should configure directly in the "rules" folder that you will use.

Beyond this, the last time I tried looking into this topic... let me quote myself from a few months ago:

Originally Posted by wyldckat

last time I looked into this was back in 2011 and is documented here: Anyone succesful on static executable solver?

Best regards,
Bruno

[wildfire230]

Bruno,

Thanks so much for your thoughtful response. I know I'm not the only one would say your contributions to this platform are really incredible.

I believe I have made some progress with the compilation based on your advice. I believe I have successfully compiled the third party software, although I'm not 100% sure. At least the only error that appears seems to be unimportant:

cp: cannot stat `../bin/d[agm]*': No such file or directory
make: [install] Error 1 (ignored)

Aside from that, the ThirdParty compilation seems to successfully reach completion:

make[1]: Entering directory `/autofs/nccs-svm1_home1/vza/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
rm -f *~ *.o
rm -f                 \
                    test_common_thread        \
                    test_scotch_graph_color        \
                    test_scotch_graph_map        \
                    test_scotch_graph_order        \
                    test_scotch_graph_part_ovl    \
                    test_scotch_dgraph_band        \
                    test_scotch_dgraph_check    \
                    test_scotch_dgraph_coarsen    \
                    test_scotch_dgraph_grow        \
                    test_scotch_dgraph_redist    \
                    test_strat_par            \
                    test_strat_seq
make[1]: Leaving directory `/autofs/nccs-svm1_home1/vza/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/check'
(cd esmumps ;        make realclean)
make[1]: Entering directory `/autofs/nccs-svm1_home1/vza/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f *~ common.h *.o lib*.so main_esmumps
make[1]: Leaving directory `/autofs/nccs-svm1_home1/vza/OpenFOAM/ThirdParty-5.0/scotch_6.0.3/src/esmumps'
rm -f ../bin/* ../include/* ../lib/*
========================================
Build Metis decomposition
    optional component Metis was not found
========================================
Build CGAL
    optional component was not found

========================================
Done ThirdParty Allwmake
========================================

Compiling OpenFOAM-5.0 with the same settings then seems to start correctly, but terminates with a fatal error: labelList.H: No such file or directory. Is that a common error that you know about? I haven't had much luck looking for other relevant mentions of this error. The full output is here if that is helpful:

Allwmake /ccs/home/vza/OpenFOAM/OpenFOAM-5.0
make: Nothing to be done for `all'.

========================================
Start ThirdParty Allwmake
========================================

========================================
Build MPI libraries if required

========================================
Build Scotch decomposition library scotch_6.0.3
    /autofs/nccs-svm1_home1/vza/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3
    scotch header in /autofs/nccs-svm1_home1/vza/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/scotch_6.0.3/include
    scotch libs   in /autofs/nccs-svm1_home1/vza/OpenFOAM/ThirdParty-5.0/platforms/linux64CCDPInt32/lib

========================================
Build Metis decomposition
    optional component Metis was not found
========================================
Build CGAL
    optional component was not found

========================================
Done ThirdParty Allwmake
========================================

Allwmake /ccs/home/vza/OpenFOAM/OpenFOAM-5.0/src
no git description found
Allwmake /ccs/home/vza/OpenFOAM/OpenFOAM-5.0/src/Pstream
wmake dummy
CC -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -O3  -DNoRepository -ftemplate-depth-100  -IlnInclude -I. -I/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude -I/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OSspecific/POSIX/lnInclude   -fPIC -c UPstream.C -o Make/linux64CCDPInt32Opt/UPstream.o
CC -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -O3  -DNoRepository -ftemplate-depth-100  -IlnInclude -I. -I/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude -I/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OSspecific/POSIX/lnInclude   -fPIC -c UIPread.C -o Make/linux64CCDPInt32Opt/UIPread.o
CC -Dlinux64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -O3  -DNoRepository -ftemplate-depth-100  -IlnInclude -I. -I/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude -I/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OSspecific/POSIX/lnInclude   -fPIC -c UOPwrite.C -o Make/linux64CCDPInt32Opt/UOPwrite.o
In file included from UPstream.C:26:0:
/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude/UPstream.H:42:23: fatal error: labelList.H: No such file or directory
 #include "labelList.H"
                       ^
compilation terminated.
make: *** [Make/linux64CCDPInt32Opt/UPstream.o] Error 1
make: *** Waiting for unfinished jobs....
In file included from /autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude/Pstream.H:42:0,
                 from /autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude/UOPstream.H:36,
                 from UOPwrite.C:29:
/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude/UPstream.H:42:23: fatal error: labelList.H: No such file or directory
 #include "labelList.H"
                       ^
compilation terminated.
In file included from /autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude/Pstream.H:42:0,
                 from /autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude/UIPstream.H:36,
                 from UIPread.C:29:
/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/OpenFOAM/lnInclude/UPstream.H:42:23: fatal error: labelList.H: No such file or directory
 #include "labelList.H"
                       ^
compilation terminated.
make: *** [Make/linux64CCDPInt32Opt/UOPwrite.o] Error 1
make: *** [Make/linux64CCDPInt32Opt/UIPread.o] Error 1

Thanks again!

[ykanani]

Hi,
Installing OpenFoam on Cray machines is a bit tricky. Sometimes you have to compile the code on compute nodes. To solve the problem, I would first contact the technician support.
Also, some of my colleagues are using OpenFOAM on Titan (Not 5.0 though).
If contacting the Titan support did not help ( which I am sure it would), please let me know and I can ask them to provide you with their complication procedures and environment variables.

Regards,
Yousef

[wildfire230]

Hi Yousef,

That would be EXTREMELY helpful! The technical support is likewise having trouble making progress for compiling OpenFOAM 5.0. I spoke to someone who was using OF 2.4 on Titan, and they gave me their installation scripts, but to be honest I am having trouble understanding exactly what they did, and the file structures have changed since 2.4, so with my level of understanding I don't really have the ability to modify the scripts appropriately.

Anything you could provide would be helpful.

All the best

[ykanani]

Originally Posted by wildfire230

Hi Yousef,

That would be EXTREMELY helpful! The technical support is likewise having trouble making progress for compiling OpenFOAM 5.0. I spoke to someone who was using OF 2.4 on Titan, and they gave me their installation scripts, but to be honest I am having trouble understanding exactly what they did, and the file structures have changed since 2.4, so with my level of understanding I don't really have the ability to modify the scripts appropriately.

Anything you could provide would be helpful.

All the best

My colleagues are using 2.4 version as well and I think they used the same instruction that you are following. Here, I will give you some links which are the instruction for OpenFOAM installation on other Cray machines which perhaps can give you hint to resolve your issues:
https://support.pawsey.org.au/docume...ay/US/OpenFOAM
http://www.archer.ac.uk/documentatio...2.2_archer.php

See attached file as well.

More importantly, it seems your environment variables are not set properly (perhaps since Titan uses two different paths to point to the same location).

you are compiling from this directory:
Allwmake /ccs/home/vza/OpenFOAM/OpenFOAM-5.0/src , but OpenFoam is looking under:

/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/

Please check your OpenFoam bashrc (perhaps line 45,46):

export FOAM_INST_DIR=$(cd $(dirname ${BASH_SOURCE:-$0})/../.. && pwd -P) || \

export FOAM_INST_DIR=$HOME/$WM_PROJECT

comment the first line. The FOAM_INST_DIR should point to /home/vza/OpenFOAM assuming that your $HOME is set to /home/vza/

I hope this helps.


Regards,
Yousef

Building OpenFOAM On Cray (1).docx

[wildfire230]

Hi Yousef,

Thanks so much! It's very helpful. I was able to follow all of the instructions in your attachment. Unfortunately, I am still running into a fatal error:labelList.H: No such file or directory during compilation. It seems to compile ThirdParty successfully, and then in the third or fourth compilation step for OpenFOAM it fails with that error.

I was curious if this is just some linking error or file path error because of your message about the two different directories:
/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/src/
and/ccs/home/vza/OpenFOAM/OpenFOAM-5.0/src
but these directories both point to exactly the same place as far as I can tell. I think one is just an alias for the other. Honestly I don't really understand how that works or whether that could be related to the error.

Even when I have $FOAM_INST_DIR set to /ccs/home/vza/OpenFOAM, the compiler still seems to do all of its work in /autofs/nccs-svm1_home1/vza/OpenFOAM, and ThirdParty compiles in this manner, so I guess maybe the two different paths are not related to the error?

One thing which I did not fully understand about your instructions, it specified to modify the c and c++ files in wmake/rules/linux64Gcc, but when I try to compile like that, it seems like it was looking for wmake/rules/linux64CC instead, and failed with an error about that. I copied linux64Gcc to linux64CC and then it was happy, but failed with the labelList.H error.

This might be a dumb question, but do I need to compile through a job submission? Titan uses srun, and I was just wondering if that would somehow be related to compiling for the compute nodes, etc... I feel like the help staff would have mentioned that if it was an issue.

I just tried the instructions you attached again on a clean download, and again it seems like ThirdParty compiles, but OpenFOAM compilation now fails with this:

touch: cannot touch `/autofs/nccs-svm1_home1/vza/OpenFOAM/OpenFOAM-5.0/platforms/linux64GccDPInt32OptCRAY-MPICH/src/Pstream/mpi/using:CRAY-MPICH': No such file or directory

Is that somehow helpful for debugging?

Thanks again

[ykanani]

To remember what I had done, I have looked at OpenFOAM 5.0 that I have installed on a Cray machine. It seems I mostly followed this procedure with small modifications for OpenFOAM 5:
https://support.pawsey.org.au/docume...ay/US/OpenFOAM

But before getting into details, it helps if you know some aspects of config files in the OpenFOAM installation. As you know, setting the environment variables starts with OpenFOAM bashrc which determines almost everything. For example, the last part of linux64CC comes from the WM_COMPILER option in the bashrc file. So if you set it as Gcc, it will become linux64Gcc for example.

For mpi, we set WM_MPLIB in the bashrc, and then, in config.sh/mpi, we provide appropriate path to point to mpi for that specific WM_MPILIB. You can read bashrc, config.sh/setting, config.sh/mpi to realize what is going on. I am not expert as well, but understanding these files is really helpful in OpenFOAM installation.

OK, let get to the main issue. Please follow the instruction below (this is mostly based on the above link with modifications for OpenFOAM 5):

- download and extract OpenFOAM and ThirrdParty (you have already done this)
-load appropriate modules

module swap PrgEnv-cray PrgEnv-gnu
module add cmake
module add boost
module list

-Modify etc/bashrc:
1- change

export WM_ARCH_OPTION=64

to:

export WM_ARCH_OPTION=cray

2- change

WM_MPLIB=OPENMPI

to

WM_MPLIB=MPICH2

3- comment these lines:

[ ${BASH_SOURCE:-$0} ] && \
export FOAM_INST_DIR=$(cd $(dirname ${BASH_SOURCE:-$0})/../.. && pwd -P) || \

you should have

export FOAM_INST_DIR=$HOME/$WM_PROJECT

4-Add (optional)

export WM_NCOMPPROCS=4

-Modify etc/config.sh/setting
1- add cray option to the architecture section, since we set WM_ARCH_OPTION to "cray". it should look like:

Linux)
    WM_ARCH=linux

    # Compiler specifics
    case `uname -m` in
        i686)
            export WM_ARCH_OPTION=32
            export WM_CC='gcc'
            export WM_CXX='g++'
            export WM_CFLAGS='-fPIC'
            export WM_CXXFLAGS='-fPIC -std=c++0x'
            export WM_LDFLAGS=
        ;;

    x86_64)
        case "$WM_ARCH_OPTION" in
        32)
            export WM_COMPILER_ARCH=64
            export WM_CC='gcc'
            export WM_CXX='g++'
            export WM_CFLAGS='-m32 -fPIC'
            export WM_CXXFLAGS='-m32 -fPIC -std=c++0x'
            export WM_LDFLAGS='-m32'
            ;;
        64)
            WM_ARCH=linux64
            export WM_COMPILER_LIB_ARCH=64
            export WM_CC='gcc'
            export WM_CXX='g++'
            export WM_CFLAGS='-m64 -fPIC'
            export WM_CXXFLAGS='-m64 -fPIC -std=c++0x'
            export WM_LDFLAGS='-m64'
            ;;
        cray)
            WM_ARCH=crayxc
            export WM_COMPILER_LIB_ARCH=64
            export WM_CC='cc'
            export WM_CXX='CC'
            export WM_CFLAGS='-fPIC'
            export WM_CXXFLAGS='-fPIC'
            ;;
        *)

-Modify etc/config.sh/mpi
1-Add MPICH2 option (since we set WM_MPLIB=MPICH2 ). It should looks like:

case "$WM_MPLIB" in
SYSTEMOPENMPI)
    # Use the system installed openmpi, get library directory via mpicc
    export FOAM_MPI=openmpi-system

    # Undefine OPAL_PREFIX if set to one of the paths on foamOldDirs
    if [ -z "$($foamClean "$OPAL_PREFIX" "$foamOldDirs")" ]
    then
        unset OPAL_PREFIX
    fi

    libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    export MPI_ARCH_PATH="${libDir%/*}"

    _foamAddLib     $libDir
    unset libDir
    ;;

MPICH2)
    export FOAM_MPI=mpich2
#    export FOAM_MPI_LIBBIN=$FOAM_LIBBIN/mpich2 
    export MPI_ARCH_PATH=$MPICH_DIR
    ;;
...

IMPORTANT NOTE: you need to make sure that MPICH_DIR is properly set and points to mpi. You can do "echo $MPICH_DIR" in the terminal, it should be something like:

/opt/cray/mpt/7.5.0/gni/mpich-gnu/4.9

-create or modifty wamke/rules
1- create crayxcGcc folder under wmake (since we set WM_ARCH=crayxc in setting and WM_COMPILER is set to Gcc in bashrc ) by copying linux64Gcc folder

cp -r  wmake/rules/linux64Gcc wmake/rules/crayxcGcc

2- modify wmake/rules/crayxcGcc/c to:

SUFFIXES += .c

cWARN        = -Wall

cc          = cc -m64

include $(DEFAULT_RULES)/c$(WM_COMPILE_OPTION)

cFLAGS      = $(GFLAGS) $(cWARN) $(cOPT) $(cDBUG) $(LIB_HEADER_DIRS) -fPIC

ctoo        = $(WM_SCHEDULER) $(cc) $(cFLAGS) -c $< -o $@

LINK_LIBS   = $(cDBUG)

LINKLIBSO   = $(cc) -shared
LINKEXE     = $(cc) -Xlinker --add-needed -Xlinker -z -Xlinker nodefs

3- modify wmake/rules/crayxcGcc/c++ to:

SUFFIXES += .C

c++WARN     = -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof

# Suppress some warnings for flex++ and CGAL
c++LESSWARN = -Wno-old-style-cast -Wno-unused-local-typedefs -Wno-array-bounds

CC          = CC -std=c++11 -m64

include $(DEFAULT_RULES)/c++$(WM_COMPILE_OPTION)

ptFLAGS     = -DNoRepository -ftemplate-depth-100

c++FLAGS    = $(GFLAGS) $(c++WARN) $(c++OPT) $(c++DBUG) $(ptFLAGS) $(LIB_HEADER_DIRS) -fPIC

Ctoo        = $(WM_SCHEDULER) $(CC) $(c++FLAGS) -c $< -o $@
cxxtoo      = $(Ctoo)
cctoo       = $(Ctoo)
cpptoo      = $(Ctoo)

LINK_LIBS   = $(c++DBUG)

LINKLIBSO   = $(CC) $(c++FLAGS) -shared -Xlinker --add-needed -Xlinker --no-as-needed
LINKEXE     = $(CC) $(c++FLAGS) -Xlinker --add-needed -Xlinker --no-as-needed

4-create wmake/rules/crayxcGcc/mplibMPICH2 file containing:

PFLAGS = -DMPICH_SKIP_MPICXX
PINC =
PLIBS =

Modify ThirdParty files
1- change ThirdParty-5.0/etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM to:

EXE =
LIB = .so
OBJ = .o
MAKE = make
AR = cc -v
ARFLAGS = -shared -o
CAT = cat
CCS = cc
CCP = cc
CCD = cc
CFLAGS = -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -Drestrict=__restrict
CLIBFLAGS = -shared -fPIC
LDFLAGS = -lz -lm -lrt
CP = cp
LEX = flex -Pscotchyy -olex.yy.c
LN = ln
MKDIR = mkdir
MV = mv
RANLIB = echo
YACC = bison -pscotchyy -y -b y

2-modify ThirdParty-5.0/scotch_6.0.3/src/libscotch/Makefile. look near the end of the Makefile for the section:

scotch.h : dummysizes$(EXE) \
library.h
./dummysizes$(EXE) library.h scotch.h
scotchf.h : dummysizes$(EXE) \
library_f.h
./dummysizes$(EXE) library_f.h scotchf.h
ptscotch.h : ptdummysizes$(EXE) \
library_pt.h
./ptdummysizes$(EXE) library_pt.h ptscotch.h
ptscotchf.h : ptdummysizes$(EXE) \
library_pt_f.h
./ptdummysizes$(EXE) library_pt_f.h ptscotchf.h

and modify it as follows:

scotch.h : dummysizes$(EXE) \
library.h
aprun -q ./dummysizes$(EXE) library.h scotch.h
scotchf.h : dummysizes$(EXE) \
library_f.h
aprun -q ./dummysizes$(EXE) library_f.h scotchf.h
ptscotch.h : ptdummysizes$(EXE) \
library_pt.h
aprun -q ./ptdummysizes$(EXE) library_pt.h ptscotch.h
ptscotchf.h : ptdummysizes$(EXE) \
library_pt_f.h
aprun -q ./ptdummysizes$(EXE) library_pt_f.h ptscotchf.h

Now you are ready to compile and hopefully it works . Before compilation, make sure to:

export CRAYPE_LINK_TYPE=dynamic
export CRAY_ADD_RPATH=yes

I think you should be able to compile it on the login node, but if it didn't work, it does not hurt trying it by submitting a job.

[wildfire230]

Amazing! It looks like it's working!

Thank you so much! I owe you a coffee.

Just fyi, if anyone comes across this forum looking to compile on Titan, the instructions to add the aprun commands in the libscotch makefile are unnecessary, since we are compiling on the head node and not through a job submission. It actually caused an error for me including that change. Also don't forget to source your OpenFOAM-5.0/etc/bashrc file before you run ./Allwmake.

[ykanani]

Great, I am glad that it worked.

[wildfire230]

Hi Yousef,

Sorry to bother you again! It seems like compilation was successful, at least it finished with no errors. Now, when I try to run a simple test command, for example 'blockMesh' in the cavity tutorial case, I receive an 'Illegal instruction' error, with no other helpful context. I thought this might be because I was trying to run the command through the shell, so I tried the same thing with a job submission (properly sourcing the etc/bashrc and switching to the proper directory) and I still receive a single line of output which is 'Illegal instruction'.

From what I gather, this might be related to building on the wrong architecture or using incompatible optimization flags or something? I was just wondering if you had any experience with this or other ideas?

Thanks again!

[ykanani]

Originally Posted by wildfire230

Hi Yousef,

Sorry to bother you again! It seems like compilation was successful, at least it finished with no errors. Now, when I try to run a simple test command, for example 'blockMesh' in the cavity tutorial case, I receive an 'Illegal instruction' error, with no other helpful context. I thought this might be because I was trying to run the command through the shell, so I tried the same thing with a job submission (properly sourcing the etc/bashrc and switching to the proper directory) and I still receive a single line of output which is 'Illegal instruction'.

From what I gather, this might be related to building on the wrong architecture or using incompatible optimization flags or something? I was just wondering if you had any experience with this or other ideas?

Thanks again!

No problem. As far as I now, the login node on Titan is not Cray. But you just compiled OpenFOAM for cray system which are compute nodes. You cannot use this installation on login node.
I think you are running blockMesh on the login nodes now, is that right?

Submit a job and check if it works or not. I think it will work.

I am not sure about this, but you might need to use aprun for running even serial commands on Titan (e.g. aprun - n 1 blockMesh)

Yousef

Edit: just noticed that you said you tried job submission as well. Try running it with aprun command. If it did not work, provide the compilation instructions along with necessary files (wmake/rules/crayxc...,bashrc ,etc) to the technical support. They should be able to help in this regard. I cannot think of something else to solve the issue

[ykanani]

Originally Posted by wildfire230

Hi Yousef,

Sorry to bother you again! It seems like compilation was successful, at least it finished with no errors. Now, when I try to run a simple test command, for example 'blockMesh' in the cavity tutorial case, I receive an 'Illegal instruction' error, with no other helpful context. I thought this might be because I was trying to run the command through the shell, so I tried the same thing with a job submission (properly sourcing the etc/bashrc and switching to the proper directory) and I still receive a single line of output which is 'Illegal instruction'.

From what I gather, this might be related to building on the wrong architecture or using incompatible optimization flags or something? I was just wondering if you had any experience with this or other ideas?

Thanks again!

I checked their user guide. I was right that you have to run command with aprun to execute them on compute nodes. Please see (Login, Service and compute node section of their userguide). This is from their userguide:

“Note: On Cray machines, the only way to access the compute nodes is via the aprun command.”

Please update me as well, good luck.

Yousef

[wildfire230]

Thanks! I think you are right about aprun. That didn't fully solve the problem for me though because it gives a 'not a directory' error. Anyway it turns out that the compute nodes do not have access to our home directory, which is where I compiled, so I'm in the process of copying over to my project directory, and hopefully that will fix everything. I will update once I'm sure.

Thanks again!

[wildfire230]

Ok, it works!

Here are some more details, if anyone comes to this forum looking for answers:

On Titan there are a lot of different file systems, including your home directory, which is something like /ccs/home/*** where *** is your three character username, there is your project specific directory /ccs/proj/****** where ****** is your specific project identifier. Those are both NFS file systems. Of them, you should compile in /ccs/proj/****** because the compute nodes have read access to that file system, but not to your home directory. However, you can't execute any commands in that directory, because the compute nodes do not have write access there. So you should put your specific case files in /lustre/atlas/proj-shared/******, which the compute nodes have read and write access to. Also fyi, that you may want to get a purge exemption on your /ccs/proj/****** directory once you compile there, because it would suck to have your installation deleted.

Here is an example of a batch script to run the cavity case on Titan, which I have copied to /lustre/atlas/proj-shared/******/cavity:

#!/bin/bash
# Begin PBS directives
#PBS -A ******
#PBS -N test
#PBS -j oe
#PBS -l walltime=0:01:00,nodes=1
#PBS -l gres=atlas1%atlas2
# End PBS directives and begin shell commands

source /ccs/proj/******/OpenFOAM/OpenFOAM-5.0/etc/bashrc

cd /lustre/atlas/proj-shared/******/cavity

aprun -n 1 blockMesh >> logfile
aprun -n 1 icoFoam >> logfile

Thanks again Yousef!

[olesen]

Originally Posted by wildfire230

Hi All,

I am trying to compile OpenFOAM 5.0 on the Titan supercomputer, and I am having lots of problems figuring out what compile options to set, etc... I do not have much of a background in architecture specific things, and the help desk has limited experience with OpenFOAM, so I am trying to DIY it.

Titan is a Cray XK7 supercomputer.

Yes, as I understand it, Cray has various special things. In OpenFOAM-1712 https://www.openfoam.com/releases/openfoam-v1712/
there is the possibility to have WM_COMPILER=Cray and WM_MPLIB=CRAY-MPICH settings.

The wmake rules for linux64Cray use 'cc' and 'CC' with flags corresponding to gcc.
The mplibCRAY-MPICH rules simply map to mplibMPICH at the moment.
If you are building ThirdParty ptscotch, it should properly pick up
'cc' instead of 'mpicc' for its build.

However, since I believe Titan is a site that is also using spack https://github.com/spack/spack,
you might wish to give that a try (it might also get you more support from the help desk).
The spack OpenFOAM-1712 is available via the openfoam-com package, which I maintain.

If you go that route, any issues are probably better to be raised
either via spack (there are presumably several cray/openfoam people
there) or via https://develop.openfoam.com/Development/OpenFOAM-plus

/mark

[wildfire230]

Hi all,


I am sorry to revise this thread, but I am now trying to compile OF 3.0.1 on the Titan supercomputer using the same general approach as outlined above. However, when trying to build CGAL-4.7, I receive an error.


First of all, I run:


cmake . -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC
make


This gives the following output:
[ 33%] Building CXX object src/CGAL/CMakeFiles/CGAL.dir/all_files.cpp.o
In file included from /ccs/proj/cfd108/OpenFOAM/ThirdParty-3.0.1/CGAL-4.7/include/CGAL/number_type_basic.h:60:0,
from /ccs/proj/cfd108/OpenFOAM/ThirdParty-3.0.1/CGAL-4.7/include/CGAL/basic.h:44,
from /ccs/proj/cfd108/OpenFOAM/ThirdParty-3.0.1/CGAL-4.7/src/CGAL/test_FPU_rounding_mode.cpp:23,
from /ccs/proj/cfd108/OpenFOAM/ThirdParty-3.0.1/CGAL-4.7/src/CGAL/all_files.cpp:1:
/ccs/proj/cfd108/OpenFOAM/ThirdParty-3.0.1/CGAL-4.7/include/CGAL/double.h: In function 'double CGAL::nextafter(double, double)':
/ccs/proj/cfd108/OpenFOAM/ThirdParty-3.0.1/CGAL-4.7/include/CGAL/double.h:191:27: error: '_nextafter' was not declared in this scope
return _nextafter(d1, d2); // works at least for VC++-7.1
^
make[2]: *** [src/CGAL/CMakeFiles/CGAL.dir/all_files.cpp.o] Error 1
make[1]: *** [src/CGAL/CMakeFiles/CGAL.dir/all] Error 2
make: *** [all] Error 2



Does anyone know how to solve this error? I would really appreciate it.


Also, when I try to compile scotch_6.0.3 using make, I receive the following errors:


mkdir ../bin
mkdir ../include
mkdir ../lib
(cd libscotch ; make VERSION=6 RELEASE=0 PATCHLEVEL=3 scotch && make install)
make[1]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-3.0.1/scotch_6.0.3/src/libscotch'
make CC="cc" CCD="cc" \
scotch.h \
scotchf.h \
libscotch.a \
libscotcherr.a \
libscotcherrexit.a
make[2]: Entering directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-3.0.1/scotch_6.0.3/src/libscotch'
cc -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -DSCOTCH_PTHREAD -Drestrict=__restrict -DIDXSIZE64 -c arch.c -o arch.o
cc -O3 -DCOMMON_FILE_COMPRESS_GZ -DCOMMON_PTHREAD -DCOMMON_RANDOM_FIXED_SEED -DSCOTCH_RENAME -DSCOTCH_PTHREAD -Drestrict=__restrict -DIDXSIZE64 -DSCOTCH_VERSION=6 -DSCOTCH_RELEASE=0 -DSCOTCH_PATCHLEVEL=3 dummysizes.c -o dummysizes -lz -lm -lrt -pthread
./dummysizes library.h scotch.h
make[2]: *** [scotch.h] Illegal instruction
make[2]: *** Deleting file `scotch.h'
make[2]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-3.0.1/scotch_6.0.3/src/libscotch'
make[1]: *** [scotch] Error 2
make[1]: Leaving directory `/autofs/nccs-svm1_proj/cfd108/OpenFOAM/ThirdParty-3.0.1/scotch_6.0.3/src/libscotch'
make: *** [scotch] Error 2



I would really appreciate any help. It is really holding me back from my research.


All the best.

[wyldckat]

Quick answer @wildfire230:

1. If Scotch was built successfully when you built OpenFOAM 5 a few months ago, then you can use that installation with OpenFOAM 3.0.1.
   • Take a look into the file "ThirdParty-3.0.1/Allwmake" to see the paths where Scotch is installed into.
2. As for CGAL, it might also work just fine if you use the CGAL build that you have from OpenFOAM 5.

That said, CGAL is the least important, because it's only necessary for foamyHexMesh and foamyQuadMesh, which I think they are not very commonly used meshers.

[cperalta]

Hello

I tried to following the guide above to compile OpenFOAM 7 in a Cray XC50.
The instructions above are very similar to what is described here:
https://bluewaters.ncsa.illinois.edu/openfoam

I am stuck in the compilation of the scotch library, which I think it is similar
to the "dummysizes" compilation mentioned here

https://www.epcc.ed.ac.uk/blog/2017/...iling-openfoam

I get this error
COTCH_RENAME -DSCOTCH_PTHREAD -Drestrict=__restrict -DIDXSIZE64 -shared -fPIC -c parser_yy.c -o parser_yy.o -DSCOTCH_VERSION_NUM=6 -DSCOTCH_RELEASE_NUM=0 -DSCOTCH_PATCHLEVEL_NUM=6
/usr/bin/ld: unrecognized option '-Wl,-rpath=/opt/cray/pe/gcc-libs'
/usr/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status
Makefile:3042: recipe for target 'dummysizes' failed
make[2]: *** [dummysizes] Error 1
make[2]: *** Waiting for unfinished jobs....

and I tried to compile dummysizes directly, as explained in the link above,
but I keep getting the error message

unrecognized option '-Wl,-rpath=/opt/cray/pe/gcc-libs'

I tried unloading the craype-mic-knl module as explained above, but
keep getting the same error.

Could anyone tell me what this error mean?

Thank you

Carlos

[wildfire230]

Hi Carlos,


My best guess is to see if you can follow the guidelines from ykanani posted above and adapt them to your current case. It depends on getting the right programming environment and modules loaded. It is possibly an issue with the MPLIB, ARCH, ARCH_OPTION definitions.


Best