[kokizzu]

symlinking ld to the correct one solves that problem:

# ll /usr/bin/ld*
lrwxrwxrwx. 1 root root      6 May 28 11:25 /usr/bin/ld -> ld_mic
-rwxr-xr-x. 1 root root   5325 May 12 16:00 /usr/bin/ldd
lrwxrwxrwx. 2 root root     88 Apr 11 12:18 /usr/bin/ld_mic -> /opt/mpss/3.2.1/sysroots/x86_64-mpsssdk-linux/usr/bin/k1om-mpss-linux/k1om-mpss-linux-ld
-rwxr-xr-x. 1 root root 594968 Feb 22  2013 /usr/bin/ld_org

[akun646]

Originally Posted by kostyfisik

May be I am wrong, by I will try compare results. There is some data
http://www.hector.ac.uk/cse/distribu...am/node17.html and from topic starter

Xeon Phi for 50 time steps
grid size - 90^3 - 175^3
best time - 200s - 1500 s

Hectors 4 core of AMD 2.8GHz dual-core Opteron 5 time steps
grid size - 100^3 - 200^3
time - 795s - 8800 s

Hectors 1024 core of AMD 2.8GHz dual-core Opteron 40 time steps
grid size - 200^3
time - 1490 s

So, single Xeon Phi card for OpenFOAM is compatible with 1024 core cluster (for this benchmark)

Hello kostyfisik! Have you modified the source code of OpenFOAM to get this performance improvement? Or you just compiled OpenFOAM on Xeon Phi? I am intrested because there may be a chance for me to access Xeon Phi but I really want to know if it is worth. Thank you very much!!!

Facundo

[matthias]

Hello,

currently, I'm testing OF on two Intel Xeon PHI cards using a native OF build
but I cannot reproduce the results of the benchmark obtained by asaijo. I get really bad results for PHI architecture.

Did anyone know what code modifications should be done to get the performance improvements described in this thread?


Best regards

Matthias

[waynezw0618]

Hi
I followed the setup here as shown from kokizzu but still lots of error messages as :
~OpenFOAM/OpenFOAM-2.2.2/wmake/platforms/linux64Icc/wmkdep: cannot execute binary file ..
can any one tell me how to fix that?

[waynezw0618]

Hello:
I also met this problem, but here my "compilervars_arch.sh" don't have option "mic", I am very sure we have several xeon Phi cards on each node here. is it because of installation of compiler?

thanks!

Originally Posted by kokizzu

Hi, I'm building OpenFOAM for Intel MIC (Xeon Phi), but i got this error:

icpc -std=c++0x -Dlinux64 -DWM_DP -wd327,525,654,819,1125,1476,1505,1572 -xHost -O2 -no-prec-div  -DNoRepository -I.. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/twoPhaseMixture/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/incompressible/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/transportModels/interfaceProperties/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/turbulenceModels/incompressible/turbulenceModel -I/root/OpenFOAM/OpenFOAM-2.3.0/src/finiteVolume/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/meshTools/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/fvOptions/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/sampling/lnInclude -IlnInclude -I. -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OpenFOAM/lnInclude -I/root/OpenFOAM/OpenFOAM-2.3.0/src/OSspecific/POSIX/lnInclude   -fPIC -Xlinker --add-needed -Xlinker --no-as-needed Make/linux64IccDPOpt/porousInterFoam.o -L/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib \
             -ltwoPhaseMixture -linterfaceProperties -ltwoPhaseProperties -lincompressibleTransportModels -lincompressibleTurbulenceModel -lincompressibleRASModels -lincompressibleLESModels -lfiniteVolume -lmeshTools -lfvOptions -lsampling -lOpenFOAM -ldl   -L/lib -lm -o /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/bin/porousInterFoam
ld: warning: libimf.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
ld: warning: libsvml.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
ld: warning: libirng.so, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
ld: warning: libintlc.so.5, needed by /root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtwoPhaseMixture.so, not found (try using -rpath or -rpath-link)
/root/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/lib/libtriSurface.so: undefined reference to `__intel_sse4_atol'

My setup:

vim etc/bashrc
export WM_COMPILER=Icc
export WM_MPLIB=

vim wmake/rules/linuxIcc/c
cFLAGS      = $(GFLAGS) $(cWARN) $(cOPT) $(cDBUG) $(LIB_HEADER_DIRS) -KPIC -mmic

vim wmake/rules/linuxIcc/c++
c++FLAGS    = $(GFLAGS) $(c++WARN) $(c++OPT) $(c++DBUG) $(ptFLAGS) $(LIB_HEADER_DIRS) -KPIC -mmic

vim wmake/rules/linuxIcc/c++Opt
c++OPT      = -O2 -no-prec-div

source etc/bashrc
./Allwmake

Is it because I don't have Intel MKL installed?

the libimf.so was found here

/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/mic/libimf.so
/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/ia32/libimf.so
/opt/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64/libimf.so
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/mic/libimf.so
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/ia32/libimf.so
/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64/libimf.so

where should I put the -rpath or -rpath-link flags?

or should i run

/opt/intel/composer_xe_2013_sp1.2.144/bin/compilervars_arch.sh mic

first?

[davidtimide]

Hi everybody, thank you so much for sharing your Phi/OpenFoam sucesses and difficulties.

I'd like to try the same for naval hydrodynamics and have a 8*Xeon Phi machine at hand (Supermicro 4028, Dual Haswell, eight 31S1P mic's, accessible via Teamviewer). I want the solution to run on the Phi's only, cluster-in-a-box-style: The host OS and processors would only care about the virtual networking and storage then. No need to care about the complicated offloading/coprocessing stuff.

I'm both an OpenFOAM and C++ nOOb, and already overloaded with other parts of the project, so it's probably hopeless for me to get this setup to work all by myself. So i'm looking for someone with significant experience who would be interested in a small consulting mission.

First step would be to compile OpenFOAM for the mic's, maybe perform some tuning along what others have tried (hints here: https://www.nersc.gov/assets/Uploads...5-OpenFOAM.pdf) and then benchmark a reference case (~20 million cells) against the same load running on the two Haswell host processors. I guess this is only a few days of work for someone who knows his stuff, and it would allow to decide if it's worth pursuing in terms of raw machine performance.

(I'd be equally interrested in that kind of test for SU2 if you are familiar with that.)

If that succeeds (success meaning that one phi card can more or less do the work of a Haswell CPU), the second step would be to engage in workflow setup. Between the discussions about requirements, installation, testing, writing up (nothing formal needed, i just need to memorize what to do and what to keep in mind) and explaining i'd expect this to take around a couple weeks of work. A typical marine case would be 2DOF or 6DOF with or without waves, including some kind of turbulence modelling. Apart from the large hull 15m there will also be small apendages (think .5 to 3m) behaving like airfoils. Results will be graphical pressures, velocities, flowlines on the hull, and tables with resistance, position, accelerations. I don't necessarily need absolute data because i can calibrate against known stuff.

Context: I'm a lonesome entrepreneur doing the feasibility homework of my startup project. At this stage i have more or less proven my design (on a commercial cfd package) and am now into optimizing it. Later on i'll have it cross-checked by more knowledgable/better equipped hydrodynamics consultants. So there is no big money involved at this stage, and even when actually launched my project would become a small business with a handful of people. However the months of sheer cost of living and computing expenses start to pile up to significant amounts, and since you could save me months i'd be happy to honor that. We could of course also talk about machine time if you have heavy calculations to do yourself. Also i'll probably have other stuff to do later on if we get on well.

The thread is probably not the best place to discuss details so please private message me or drop me a mail at davidtimide <at> gmail <dot> com if interested!

Thanks for your kind attention and a nice day to all!

[shivaramakrishna]

Hi All,
When i build openfoam for xeonphi, I am getting the following error. Can some one please tell me what could be the problem.

ld: Relocatable linking with relocations from format elf64-k1om (Make/linux64IccDPOpt/sigFpe.o) to format elf64-x86-64 (/share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/lib/linux64IccDPOpt/libOSspecific.o) is not supported
make: *** [/share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/lib/linux64IccDPOpt/libOSspecific.o] Error 1
+ wmake libso OpenFOAM
wmakeLnInclude: linking include files to ./lnInclude
/bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file
/bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file
/bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file
/bin/sh: /share_nfs/openfoam-xeonphi/OpenFOAM/OpenFOAM-1.7.x/wmake/bin/linux64Icc/wmkdep: cannot execute binary file

[shivaramakrishna]

Hi All,
The above linking problem resolved after adding LD flags(-melf_k1om) in make file.


Thanks
Sivaramakrishna

[mesh-monkey]

Originally Posted by shivaramakrishna

Hi All,
The above linking problem resolved after adding LD flags(-melf_k1om) in make file.


Thanks
Sivaramakrishna

Hi Sivaramakrishna,

I appear to be having the same problem (error linking from format elf64-k1om to elf64-x86-64).
Are you able to elaborate on how exactly you added that flag? For example, what file it's added to and the syntax?

Thanks, Tom

[shivaramakrishna]

Hi,
Add -melf_k1om flag to LD_FLAGS in make files c, c++.
If that does not work change symbolic link of ld to ld of mic architecture to resolve linking errors.

Thanks
Sivaramakrishna

[ma-tri-x]

Add -melf_k1om flag to LD_FLAGS in make files c, c++.

What makefiles? in

wmake/rules/linux64Icc

???
Doesn't work for me. Neither does it work when I say in

etc/bashrc

            export WM_CXXFLAGS=$WM_CFLAGS
            export WM_LDFLAGS='-melf_k1om '$WM_CFLAGS

Or do I have to do

./Allclean
./Allwmake.firstInstall

again??

[ma-tri-x]

Hi Guys,

Building OpenFOAM for Xeon Phi is bit tricky, but not difficult.

1. Make Zlib and Flex for Xeon Phi
2. Set OpenFOAM bashrc/setting.sh to linux64Icc and INTEL MPI
3. Make wmake only
4. Add -mmic to wmake/rules/linux64ICC/{c, c++}
5. Compile OpenFOAM libraries. If compilation fails, add Zlib/Flex path to Make/options
6. Compile solver applicaitions. You'll get many undefined function errors. Then use mpiicpc compiler (not icpc) and add -Wl,rpath-link flag for appropirate library's path by manual.

Finally, I got icoFoam for Xeon Phi. Perhaps there is an easier and canonical way.

erm... If it was that easy, I wonder why I'm still working on this issue since January.
Question:

Do I need to compile this freaky thirdParty Metis, ParMGridGen, Libccmio and stuff in order to make OpenFoam or Foam-Extend usable? For me, I solved many issues for the compiling procedure (Cross-compilation for Xeon Phi Co-Processor 3110) resulting in a super long manual, but with this Metis and libccmio Stuff I run into strange linker issues that seem to stem from the insufficient configure scripts for qmake that do not implement my -mmic option but use their own with -m64. When I try to compile Foam-Extend without this third-party stuff, libfoam.so compiles incomplete and so does the rest of course. I would like to leave all this thirdparty stuff out because fixing the rpm_make routine --- ahem --- s*cks. Or do I do a mistake?

--- Edit:
Ok, I left out the ThirdParty stuff and I was able to compile at least the sources /src without scotch, metis and parmidgren.
But going for the applications, I also get this error:

k1om-mpss-linux-ld: warning: libimf.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libinterfaceProperties.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libsvml.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libinterfaceProperties.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libirng.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libinterfaceProperties.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libintlc.so.5, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libinterfaceProperties.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libmeshTools.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libincompressibleRASModels.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libLESdeltas.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libincompressibleLESModels.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libLESfilters.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libincompressibleLESModels.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libmpi.so.12, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libfoam.so, not found (try using -rpath or -rpath-link)
k1om-mpss-linux-ld: warning: libz.so, needed by /home/max/foam/foam-extend-4.0/lib/linux64IccDPOpt/libfoam.so, not found (try using -rpath or -rpath-link)

my ld variable looks like:

#echo $LD_LIBRARY_PATH
/lib64:
/opt/intel/compilers_and_libraries_2017.0.098/linux/mpi/mic/intel64/lib:
/home/someone/foam/someone-4.0/lib/linux64IccDPOpt:
/home/someone/foam/site/4.0/lib/linux64IccDPOpt:
/home/someone/foam/foam-extend-4.0/lib/linux64IccDPOpt:
/opt/intel/compilers_and_libraries_2017.0.098/linux/compiler/lib/mic:
/opt/crtdc/micgnu/lib:/opt/intel/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64:
/opt/intel/compilers_and_libraries_2017.0.098/linux/compiler/lib/intel64_lin:
/opt/intel/compilers_and_libraries_2017.0.098/linux/mpi/intel64/lib:
/opt/intel/compilers_and_libraries_2017.0.098/linux/mpi/mic/lib:
/opt/intel/compilers_and_libraries_2017.0.098/linux/ipp/lib/intel64:
/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:
/opt/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64:
/opt/intel/compilers_and_libraries_2017.0.098/linux/tbb/lib/intel64/gcc4.7:
/opt/intel/debugger_2017/iga/lib:
/opt/intel/debugger_2017/libipt/intel64/lib:
/opt/intel/compilers_and_libraries_2017.0.098/linux/daal/lib/intel64_lin:
/opt/intel/mic/myo/lib:/opt/intel/compilers_and_libraries_2017.0.098/linux/daal/../tbb/lib/intel64_lin/gcc4.4