[pvita]

Dear All!

I have two news from OpenFOAM-Cygwin-Windows battlefield for you---a good one and a bad one. The good news: OpenFOAM compiles on Windows. The bad news: OpenFOAM has some problems to run. I will start with that good one, so you can compile OpenFOAM yourself and I hope you can help a little bit to eliminate that bad news.

Compilation of OpenFOAM under Cygwin

Prerequisites
1. Cygwin 1.15.18-1 and up installed (provided preinstall script expect installation in C:\Cygwin)
2. Source distribution of OpenFOAM 1.2 placed in the root of your home directory (~/)
4. Attached patch file OpenFOAM-1.2.Cygwin.patch.gz in the root of your home directory
3. Attached script preInstall.sh in the root of your home directory
4. Coffee, aspirin, some paper for notices and some book to read

OpenFOAM-1.2.Cygwin.patch.gz
preInstall.sh

If all is on the place, pray to Cygwin and Windows Gods. You are ready to start now!

Preinstall
1. If you have a directory OpenFOAM in the root of your home directory rename it
2. If you have directories C:\Cygwin\managed-dir and/or /managed-dir, unmount them and rename them
3. Run script preInstall.sh

Launching of the script creates managed mount and mount it under mounting point /managed-dir. The OpenFOAM source distribution and patch file will be moved there, unpacked, extracted and OpenFOAM will be patched. Script will create a directory ~/OpenFOAM and link source distribution there, further will modify the source tree of OpenFOAM to make it compilable. Extraction takes some time, so drink coffee and read book, so far there should be no problem.

Note: If somebody goes wrong, inspect preInstall.sh and correct what needs to be corrected.

Installation
1. Read file ~/OpenFOAM/OpenFOAM-1.2/README to get idea what variables and scripts have to be modified.
2. Modify environment variables as described. The Cygwin patch file will do quite a lot itself but you have to at least select right compilation target (Opt, Debug or Prof). Do not be surprised that destination directory of libraries and executables points out of the managed mount, it is intended.
3. Start new shell or execute bashrc script to source and update variables.

After this step you should get directories ~/OpenFOAM/bin and ~/OpenFOAM/lib with their compilation targets inside. You should not have a time for book but drinking coffee helps so do it.

Compilation (Windows 2000 Professional)
1. Close all Windows application you really do not need. I mean all!
2. Use cd ~/OpenFOAM/OpenFOAM-1.2 to place yourself in the best shooting position.
3. Get task manager window up so you can observe what happens.
4. Type ./Allwmake and observe compilation output and task manager. You will get to point where OpenFOAM will try to link all its headers and source files into directory ~/OpenFOAM/OpenFOAM-1.2/src/OpenFOAM/lnInclude...
4a. Compilation got stucked! Nothing happens for a very long time. The task manager show process ln takes near 99% of CPU time, all in user space only. You are damned. Follow instructions in point 5.
4b. Compilation continues producing foamFlex++, libPstream.dll etc. There is any known problem now. Read book, drink coffee, you have around 3-4 hours time now. Cheer!
5. Make a note on the paper you are here.
5a. There is just one note on the paper. Immediately swallow the aspirin, get new coffee a be ready for hard fight. You should know, that the aspirin will need some time to work. Take your time, you do not need to collapse. Follow with point 6.
5b. If there is more then one notice, do not swallow the aspirin (the aspirin overdosis is dangerous) and start to panic, you are lost! I have no idea what you should do as next step. Google, make call to telephone guru, ask public...
6. Break compilation using Ctrl+C. Kill manually running process ln. If everything goes well, Windows are going to run quite mad now. Task manager shows that process Explorer runs amoc taking now 99% of CPU time. No hesitation, he desires it as well. Kill process Explorer. You loose taks bar, desktop and whole that pretty stuff but do not worry. Windows are back stabile again. A positive point is that compilation runs more quickly now. You should get through linking of headers and sources now. Follow instructions in point 2.
7. Restart your machine if your notice paper contains a note you have made in point 5.

You should have, or not, a compiled OpenFOAM on you machine. If yes, it produces around 90 library files and around 160 applications.

Running the OpenFOAM

Usage of the OpenFOAM remains same as any other Unix version. I suggest you copy toturial data files you want to try on non-managed mount so you encounter no problem with crypting filenames.

I tried to run blockMesh on an icoFoam example of cavity problem. Application binds all dll-files it needs and runs, till it comes to a mesh code. There is something wrong with a mesh. We tried to run icoFoam on a mesh generated in Linux and got to meshing problems as well... No idea where it is. I hope in your help with it.

Sincerely Yours,

PV

[pvita]

Looking at the problem to get OpenFOAM running I found there is something wron with lists. I traced where is the error in blockMesh comming from and created a minimalistic example that always abort with core dump on my machine. Have anybody of you idea what it could be? Compiler gcc-3.4.4?

#include "wordList.H"

using namespace Foam;

int main()
{
wordList wrdList(1);
wrdList[0] = "hello";
return 0;
}

Thanks,

PV

[hjasak]

This is a perfectly valid piece of code and it works fine (pointless showing you a result) :-)

If you cannot get this to run on Micros**t, there's no hope...

Here's some valgrind for you, just to set your mind at rest:

wooster*116-> valgrind /home/hjasak/OpenFOAM/OpenFOAM-1.2/applications/bin/linuxGcc4Debug/testWord
==17089== Memcheck, a memory error detector.
==17089== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==17089== Using LibVEX rev 1313, a library for dynamic binary translation.
==17089== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==17089== Using valgrind-3.0.1.SVN, a dynamic binary instrumentation framework.
==17089== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==17089== For more details, rerun with: -v
==17089==
/*---------------------------------------------------------------------------*\
| ========= | |
| \ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \ / O peration | Version: 1.2 |
| \ / A nd | Web: http://www.foamcfd.org |
| \/ M anipulation | |
\*---------------------------------------------------------------------------*/

==17089==
==17089== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 42 from 5)
==17089== malloc/free: in use at exit: 56907 bytes in 1598 blocks.
==17089== malloc/free: 11401 allocs, 9803 frees, 657190 bytes allocated.
==17089== For counts of detected errors, rerun with: -v
==17089== searching for pointers to 1598 not-freed blocks.
==17089== checked 526816 bytes.
==17089==
==17089== LEAK SUMMARY:
==17089== definitely lost: 0 bytes in 0 blocks.
==17089== possibly lost: 16379 bytes in 728 blocks.
==17089== still reachable: 40528 bytes in 870 blocks.
==17089== suppressed: 0 bytes in 0 blocks.
==17089== Reachable blocks (those to which a pointer was found) are not shown.
==17089== To see them, rerun with: --show-reachable=yes

[niklas]

There's a difference in how Winsux and linux treat IO streams (ive been told) and Im quite sure its related to this.

[pvita]

Yeap, I know this is a valid code and it works without problems on Linux, question is why not on Windows. What collapse is an assignment operation, I do not know why. I do not think it is stream related in this piece of code. Any idea where to look? Aside of it, have you, Niklas, compiled your Cygwin version already? Can you test it?

Thanks,

PV

[mattijs]

What line number in above piece of code? What happens if you compile with -DFULLDEBUG -g -O0 (add e.g. to EXE_INC variable in Make/options)

Or is it in the construction/destruction of static objects where Niklas had his problems (patch type object were constructed before the table was constructed they should be put on)

[pvita]

I am already working with full debug version with flags -m32 -DCygwin -Wall -Wno-unused-parameter -Wold-style-cast -O0 -fdefault-inline -ggdb3 -DFULLDEBUG -DNoRepository -ftemplate-depth-30.

The assignment collapse comes in line 8:

wrdList[0] = "hello";

I have tried to break it into even smaller parts but met no problem. I have tried to write out wrdList[0] with:

Info << wrdList[0] << endl;

I have tried to construct word object from string "hello" without problems as well.

Regarding the construction/destruction of static objects. Can you point me to that problem? How can I test/find that I am not dealing with this?

[mattijs]

It is probably not static construction/destruction since it goes wrong on assignment to a local object (wrdList). Do you have valgrind? Or a debugger?

(Niklas' problem with the patches not known is a typical static construction order problem; polyPatch gets initialized before the table that holds the constructors)

Does the Intel compiler do any better?

[pvita]

I feared you will ask. :-)

There is not a lot of development tools under Cygwin. I tried gdb but it runs immediatelly in problems. Using gdb I come into static construction/destruction troubles so this one was not option and it closes this problem in the example given above as well. I am looking for some debugger I can use now. I have downloaded Valgrind 3.1 as well and I will try to get it running.

I have no Intel compiler so I will have to look if I can get one somewhere. In every case I can report that gcc 4.0.2 doesn't solve this problem as I compiled it over night for Cygwin as well and it doesn't helps.

[pvita]

Some update. Valgrind 3.1 won't work on Cygwin. MinGW compiler is not usable as do not understand how to work with managed mounts and I was not so excited to compile it under Cygwin. Microsoft SFU tools are, hehe, done in a typical way of Microsoft---having broken concept of home directories and many other things and are unusable.

Near broken, on the end of ideas, playing with all possible compiler and linker switches I got a notion that something is not working as I expected. I minimalized an above provided minimalistic example of the always falling code to a version with an empty main. Application no output. One would expect it is ok, but I used a gdb over it to see what happens... What a shock that application got a SIGSEV even if absolutely empty... Ok, got it. Not every total failure is reported as long as application returns 0, application stackdump is not a core and my application does something already in static initialization that I didn't expect. The last think is really cool as I have to admit I will have to read manual of gdb to find out how to debug at initialization step of dlls in initialization of global variables. If add a line wordList wrdList(1); things are different again. How different? That I found mainly through strace who pointed me to the right direction.

Let's have a look in it. After my test application opens /dev/null it tries to read a file controlDict. It is not so simple as OpenFOAM tries more combinations of controlDict filename that are multiplied by Cygwin emulation layer:

2535 546582 [main] wordListTest 1912 stat64: entering
59 546641 [main] wordListTest 1912 normalize_posix_path: src /home/pvita/_OpenFOAM-1.2/controlDict
56 546697 [main] wordListTest 1912 normalize_posix_path: /home/pvita/_OpenFOAM-1.2/controlDict = normalize_posix_path (/home/pvita/_OpenFOAM-1.2/controlDict)
57 546754 [main] wordListTest 1912 mount_info::conv_to_win32_path: conv_to_win32_path (/home/pvita/_OpenFOAM-1.2/controlDict)
59 546813 [main] wordListTest 1912 set_flags: flags: binary (0x2)
56 546869 [main] wordListTest 1912 mount_info::conv_to_win32_path: src_path /home/pvita/_OpenFOAM-1.2/controlDict, dst C:\Cygwin\home\pvita\_OpenFOAM-1.2\controlDict, flags 0xA, rc 0
159 547028 [main] wordListTest 1912 symlink_info::check: GetFileAttributes (C:\Cygwin\home\pvita\_OpenFOAM-1.2\controlDict) failed
66 547094 [main] wordListTest 1912 geterrno_from_win_error: windows error 3 == errno 2
...
cut off other name variants as controlDict.exe, controlDict.exe.lnk, controlDict.exe.lnk, controlDict.gz, etc.
...

This goes for quite a long time till a right placement and filename combination of the file controlDict is found (in my case on non-managed mount) and filehandler is created. File attributes are checked and file is opened for reading. In order to read file attributes same steps are done as to open for reading that consequently fails. Note red part that looks strange in the following strace output:

122 898466 [main] wordListTest 1912 normalize_posix_path: src /non-managed-dir/OpenFOAM-1.2/_OpenFOAM-1.2/controlDict
57 898523 [main] wordListTest 1912 normalize_posix_path: /non-managed-dir/OpenFOAM-1.2/_OpenFOAM-1.2/controlDict = normalize_posix_path (/non-managed-dir/OpenFOAM-1.2/_OpenFOAM-1.2/controlDict)
58 898581 [main] wordListTest 1912 mount_info::conv_to_win32_path: conv_to_win32_path (/non-managed-dir/OpenFOAM-1.2/_OpenFOAM-1.2/controlDict)
58 898639 [main] wordListTest 1912 set_flags: flags: binary (0x2)
55 898694 [main] wordListTest 1912 mount_info::conv_to_win32_path: src_path /non-managed-dir/OpenFOAM-1.2/_OpenFOAM-1.2/controlDict, dst C:\Cygwin\non-managed-dir\OpenFOAM-1.2\_OpenFOAM-1.2\controlDict, flags 0xA, rc 0
141 898835 [main] wordListTest 1912 symlink_info::check: not a symlink
64 898899 [main] wordListTest 1912 symlink_info::check: 0 = symlink.check (C:\Cygwin\non-managed-dir\OpenFOAM-1.2\_OpenFOAM-1.2\controlDict, 0x22E080) (0xA)
59 898958 [main] wordListTest 1912 path_conv::check: this->path(C:\Cygwin\non-managed-dir\OpenFOAM-1.2\_OpenFOAM-1.2\controlDict), has_acls(0)
61 899019 [main] wordListTest 1912 build_fh_pc: fh 0x6115D054
64 899083 [main] wordListTest 1912 fhandler_base::open: (C:\Cygwin\non-managed-dir\OpenFOAM-1.2\_OpenFOAM-1.2\controlDict, 0x100000)
105 899188 [main] wordListTest 1912 fhandler_base::set_flags: flags 0x100000, supplied_bin 0x10000
64 899252 [main] wordListTest 1912 fhandler_base::set_flags: filemode set to binary
55 899307 [main] wordListTest 1912 fhandler_base::open: 0 = NtCreateFile (0x2E4, 80100000, C:\Cygwin\non-managed-dir\OpenFOAM-1.2\_OpenFOAM-1.2\controlDict, io, NULL, 0, 7, 1, 20, NULL, 0)
59 899366 [main] wordListTest 1912 fhandler_base::open: 1 = fhandler_base::open (C:\Cygwin\non-managed-dir\OpenFOAM-1.2\_OpenFOAM-1.2\controlDict, 0x100000)
58 899424 [main] wordListTest 1912 fhandler_base::open_fs: 1 = fhandler_disk_file::open (C:\Cygwin\non-managed-dir\OpenFOAM-1.2\_OpenFOAM-1.2\controlDict, 0x0)
56 899480 [main] wordListTest 1912 open: 4 = open (/cygdrive/c/cygwin/home/pvita/OpenFOAM/OpenFOAM-1.2/_OpenFOAM-1.2/controlDict, 0x0)
106 899586 [main] wordListTest 1912 _cygwin_istext_for_stdio: fd 4: opened as binary
1488 901074 [main] wordListTest 1912 readv: readv (4, 0x22E690, 1) blocking, sigcatchers 0
83 901157 [main] wordListTest 1912 readv: no need to call ready_for_read
85 901242 [main] wordListTest 1912 fhandler_base::read: returning 1023, binary mode
58 901300 [main] wordListTest 1912 readv: 1023 = readv (4, 0x22E690, 1), errno 2
17512 918812 [main] wordListTest 1912 readv: readv (4, 0x22E2E0, 1) blocking, sigcatchers 0
66 918878 [main] wordListTest 1912 readv: no need to call ready_for_read

Afterwards all opened streamed devices are closed and application ends.

I am not sure what happens but it looks that under Cygwin is this file instead as text file open as binary one. I am going to check this stuff now.

Have you experienced something like that already? Any other idea? Any comment?

PV

PS: Sorry for such long posting with extrem strace message...

[pvita]

Well friends, I know what is going on now. Question is what to do with it. There is a cure but on a medicine-man level that works only with a blessing of a God and it doesn't solve the problem...

What has really suprised me was the way gdb was reporting the problems. Symptoms and strace reports shown as programs run were very wierd and misleading. I have got and idea and became suspicious that I even do not enter the main program and all the stuff goes wrong already. Yes, I found that what is going wrong is an initialisation of non-local variables between translation units. I went through the OpenFOAM code little bit and was horrified seeing what I have seen.

To prove it and be sure I am dealing with this problem I simply exchanged the order of Pstream library object files to be sure some of imported objects (those residing in another library) are initialised before Pstream internal objects. Wow, gdb has immediately backtraced to a different place to a different initialisation point. To be pretty sure I even put content of my main program before an error line reported by gdb. After recompiling, gdb reported and error on a place two lines down as before having no problem with wordList object.

Gospel (Bjarne Stroustrup; The C++ Programming Language, Special Edition), Chapter 9, Section 9.4, 9.4.1 Initialisation of Nonlocal Variables:

In principle, a variable defined outside any function (that is, global, namespace, and class static variables) is initialized before main is invoked. Such nonlocal variables in a translation unit are initialized in their definition order.
.
.
.
There is no guaranteed order of initialization of global variables in different translation units. consequently, it is unwise to create order dependencies between initializers of global variables in different compilation units. In addition, it is not possible to catch an exception thrown by the initializer of a global variable. It is generally best to minimize the use of the global variables and in particular to limit the use of globals variables requiring complicated initialization.

Several techniques exist for enforcing an order of initialization of global variables in different translation units. However, none are both portable and efficient. In particular, dynamically linked libraries do not coexist happily with global variables that have complicated dependencies.

Apostolic letter No. 47 (Scott Meyers; Effective C++, 2nd Edition), Item 47: Ensure that non-local static objects are initialized before they're used

You do not want the behavior of your software to be dependent on the initialization order of non-local static objects in different translation units, because you have no control over that order. Let me repeat that. You have absolutely no control over the order in which non-local static objects in different translation units are initialized.

I have to admit I was told by my mentors who learned me to program that global variables are bad. They are source of errors. I can say global variables placed in different compilation units are absolute evil.

One should note that this is even not a problem of Windows. I assume that it is just an incredible piece of luck that OpenFOAM works on Linux whose compiler somehow managed to initialize all at start. I expect this is done in the moment he tries to find out run-time binding between different so-files. I do not want to speculate a lot but this could be the root of all evil with porting of OpenFOAM to other platforms as well.

I wonder what we can do with it. This is question moreless of core developers without their support, help and discussion nothing can be done here.

I have found there are two mainly used techniques to get rid of it:

• Singleton Design Pattern applied and presented in Apostolic letter No. 47
• technique first poineered by Jerry Schwarz in the iostream library

And the cure? To find out the right order of object files entering linkage. It is madness I am not going to do...

With kind regards,

PV

PS: I am atheist...
PPS: OpenFOAM header rises two times if libOpenFOAM.dll and libPstream.dll are linked. During the initialisation of non-local variables both of them use code that wrotes into cout.

[hjasak]

The issue of the order of initialisation has been taken care of very carefully in OpenFOAM and luck is not involved. It is true that you cannot guarantee the order of global construction, but the standard guarantees that the memory of newly allocated statics will be zeroed-out. This is used as the basis of the initialization and it seems that Micros**t does not do it properly.

Have a more detailed look at the standard document and you should be able to find out more about the zeroing out; this can also be easily tested. For examples of how the code uses zero memory (pointers) to do its job, have a look at the initialization of run-time selection tables.

Hrv

[pvita]

Hrv, I know that an order of initialisation in a translation unit is guaranteed in their definition order. I know that default initializer for built-in types and enumerations is 0, if initializer is not provided. This works well as long as you speak about single translation unit only.

Things are unfortunately more complicated in the case of dll-files. In the initialization of libPstream.dll is used code that resides in library libOpenFOAM.dll that is not initialised yet and all goes quickly to SIGSEGV signal. That is what I am observing here. Order of initialization between translation units is not guaranteed.

Note please, that this is not problem of Microsoft, I use only gcc under Cygwin to generate the code. As long as one dll-needs code of another one to get working and depends on order of initialization there is a fully pre-programmed problem.

PV

[hjasak]

Yes: a translation unit is a .C file and not a whole library and OpenFOAM handles those cases within a single library. Looks to me like libPstream gets loaded first and then the dynamic link loader goes stupid (if a symbol from another library is required, the library should be loaded and it does not seem to be so).

There's still things we can try (if you wish to experiment). Basically, get rid of all the libraries from the dynamic loading process and open them one by one using dllOpen in the top-level code. If you choose something simple, like your wordList example or icoFoam, this should be pretty painless. When opening, make sure you dllOpen libOpenFOAM first and then add other libraries.

Hrv

[pvita]

Yes, Hrv, you are right. I did not say something different. I didn't write that dll-file is a translation unit.

Being absolutly exact. In order to get libPstream.dll linked---in my case its dummy version---I need to compile three translation units: IPread.C, OPwrite.C and Pstream.C.

They depend on existance of initialized Foam::Vector<double>::typeName, that is in a static member of class placed in translation unit vector.C that reside in libOpenFOAM.dll. Any static class member is a nonlocal static object, see Apostolic Brief No. 47.

What should do a dynamic linker with it now? If we want to initialize libPstream.dll who depends on initialized code in libOpenFOAM.dll we get SIGSEGV. If we want to initialize libOpenFOAM.dll we need initialized libPstream.dll. Order of initialization is broken here and improved with a cyclus. This is that complicated dependecy between dll-files that God Stroustrup mentioned.

If you wish I try your experiment, I will do it, just I am very pesimistic about output of this test.

PV

[pvita]

Well Hrv, I have done experiment you requested as I owed to you and me. The test was surprising and there are two outputs of it. It seems to work well but in gdb replicates the error in initialisation of static variables. I didn't gave up and tried to find out what the hell happens... At the end I have written minimalistic example and opened a thread in Cygwin mailing list.

The first output is, that Cygwin has confused me.

This is not a real segmentation fault. It is just a side effect of how Cygwin does argument checking, it's completely benign. Just type "continue" in gdb and be on your way.

Seeing a SIGSEGV with backtrace pointing to my code should not be a "real" SIGSEGV. It seems quite strange to me, I won't get such an idea if somebody won't tell me, but it should be so.

This changes the situation as my interpretation of initialisation of static variables was based on a missleading gdb output, I have to take hard words on static initialisation back, shame me and continue with debugging.

I have debuged our wordList example, equiped with a new knowledge, wrote continue after SIGSEGV and got few steps ahead minimalising the example further. The minimalistic example got reduces to this version for now excluding the lists from the problem:

#include "word.H"

using namespace Foam;

int main()
{
word* wordArray = new word[1];
wordArray[0] = "hello";
return 0;
}

I am now going to find out what happens, just for your information, I have a small program that consist of reduced classes word, string and the main above without any dll-files or OpenFOAM libraries and it works without problems. I am trying to debug initialisation of libOpenFOAM.dll as it seems something is done there what breakes class word.

Sincerely Yours,

PV

PS: Sorry for my long monologues in this list, I am used to it from Bugzilla and it helps to think about the problem...

[mattijs]

Does adding the Pstream/dummy files to the OpenFOAM library (add them to OpenFOAM/Make/files) help? (so to remove that initial dependency)

[pvita]

No, elimination of libPstream.dll in this way doesn't help, so it is somewhere in the code. We have together with Bernhard constructors of word class in sight now, as we have yet another test example that crashes:

#include "word.H"

using namespace Foam;

int main()
{
word w1;
word w2("hello");
w1 = w2;
return 0;
}

I am compiling this example together with whole OpenFOAM to get one monstrous executable for debug without dll now.

PV

[pvita]

Well, after days of trying, debuging, analysing and searching I have found what was going wrong in examples above and got them running. Now making side tests and preparing big OpenFOAM compilation to see if we get OpenFOAM alone to work or we will meet new problem. Believe me, this was a really dirty bug. Detail report on Monday, patch with scripts and install instructions will follow. Stayed tuned!

PV

PS: The OpenFOAM code is not guilty.

Excellent! I'm looking forward to it.