[pbachant]

Having only started using OpenFOAM in 2013, I don't really know much of the history of why it became so fragmented. So, why is there not "One OpenFOAM?"

It seems to me that the ideal situation would be a single core codebase with additional solvers, utilities, models, wrappers, GUIs, available as extensions on top of that. Commercial proprietary stuff, or even all the cool things in foam-extend could be packaged as apps or libraries to plug into the "One OpenFOAM," instead of carrying around the entire (massive) codebase. If a certain player or company needed, e.g., an access function added to a certain class, they would propose that change to the core version. Of course, the maintainer of the core codebase would need to be flexible and willing to let the community have a say in the direction of the project.

Each player in the OpenFOAM community maintaining an entire fork is analogous to every Linux distribution maintaining its own fork of the kernel. It's especially unfortunate that most OpenFOAM forks don't share the same Git history, so automatic merging or cherry-picking between them isn't possible--new code or bugfixes have to be copy/pasted, which is inefficient, not fun, and could even introduce new bugs.

What do you think? Is it even a worthwhile goal to have one OpenFOAM, and if so, how could we get there?

[alexeym]

Hi,

1. You should add forgotten (TM) to every OpenFOAM mention So there could not be 'One OpenFOAM', cause you have to change name. Let's call it OurPrecious.

2. Behind OurPrecious there should be a person. Let's call it Project Leader. Project leaders of OpenFOAM(TM), foam-extend, etc are the reason for all these forks. So who will be consolidating person?

3. Who will pay for further development of OurPrecious? Donations? Current business-scheme is either 1. Introduce new things, 2. Train people, 3. PROFIT!!!, or 1. Break old API, 2. Present it as new things, 3. Train people, 4. PROFIT!!!

4. Who will define development directions? Community is rather vague thing (and usually has a problem with explication of its desires).

And these questions are just a tip of the iceberg.

[wyldckat]

Greetings to all!

@Pete: I've updated the title of the thread to the question you made in the post, to make it easier to understand the context of the thread and to avoid the collision with a possible future thread where this has already been sorted out...

I have so much I would like to write here to answer to your questions, but I'll try to keep it short and to the point.

Originally Posted by pbachant

Having only started using OpenFOAM in 2013, I don't really know much of the history of why it became so fragmented. So, why is there not "One OpenFOAM?"

You can make your own assessment from this statement: https://sourceforge.net/p/foam-exten...rojectPreamble - this was one of the reasons, but as in any story, this is only one perspective. There are several forks and each one has its own reason for existing, including internal development forks.

The usage of the GNU General Public License enabled this to happen, for better or for worse, because even if forking can have its toll on the community, it still gives us the ability to choose from the existing forks to do the tasks we have at hand.
In addition, I have yet to find a public and official document from the GNU Foundation that states all of the steps that need to be taken to create a fork of an open source code project.

Originally Posted by pbachant

It seems to me that the ideal situation would be a single core codebase with additional solvers, utilities, models, wrappers, GUIs, available as extensions on top of that. Commercial proprietary stuff, or even all the cool things in foam-extend could be packaged as apps or libraries to plug into the "One OpenFOAM," instead of carrying around the entire (massive) codebase. If a certain player or company needed, e.g., an access function added to a certain class, they would propose that change to the core version. Of course, the maintainer of the core codebase would need to be flexible and willing to let the community have a say in the direction of the project.

That is what makes sense. The problem.... well, the problem is that there were several problems. To name a few:

• We are all humans, therefore "communication flaws" are always expectable, i.e. difficulties in seeing each others perspectives and difficulties in agreeing in what should be done.
• Each group involved in the development of each fork are usually very small teams (1 to 6 people each?), therefore everyone has "limited resources" and everyone has deadlines to keep, therefore it's extremely hard for all projects to stay in sync and in agreement of what can or should be changed now or later.

And there are a lot of development situations where it's easier to fork, modify and release and have something for those who paid for the effort.

And as Alexey points out, one of the issues has usually been related to funding. If we do take the Linux Kernel project as a gold standard of reference for open source projects for comparison, here is the Linux Kernel project in a nutshell:

1. The Linux Foundation pays Linus Torvalds' salary, therefore it is ensured.
2. The main maintainers received their salaries from the companies that employ them, specifically to maintain the Linux Kernel and do other kernel-related developments.
3. The Linux Foundation receives funding from the companies that need the Linux Kernel for their products and services.
4. Linus Torvalds is known for communicating in a rough manner to make his point across.
5. When they needed a new source code revision tracking software, they created it (i.e. created "git"), because they had the resources to do so.

It wasn't always like this for the Linux Project and the majority of the companies that use open source at their core also had issues when they started out too, but right now many of them are huge enough and others were absorbed/merged with others.


I would like to write about the inception of the "OpenFOAM project", how it started and the problems it faced, but I would need clearance from all of the intervening parties about what I could write and make public, even if the majority is already public information, so... I'll focus instead on the future and what is really necessary to get things going forward towards the goal that a lot of people in the community has been asking for.

Originally Posted by pbachant

Each player in the OpenFOAM community maintaining an entire fork is analogous to every Linux distribution maintaining its own fork of the kernel.

The Linux Kernel does have several forks, although usually are referred to as branches... but for ARM processors a few years ago, I vaguely remember that they had to fork/branch the git tree... although I can't find a good reference of that online.
Either way, each long term release Linux Distribution has to maintain its own branch of the kernel, which extends for so long that it can end up looking more like a fork than a branch.

Originally Posted by pbachant

It's especially unfortunate that most OpenFOAM forks don't share the same Git history, so automatic merging or cherry-picking between them isn't possible--new code or bugfixes have to be copy/pasted, which is inefficient, not fun, and could even introduce new bugs.

I'll go through a bit of the past history, regarding the git repositories, to try and give the notion of how the structures are built. The OpenFOAM repository structures are as follows:

1. The git repositories were only started around the time OpenFOAM 1.5 was in development, as made available on the OpenFOAM-history repository several months ago: https://github.com/OpenCFD/OpenFOAM-...fbc6fe272f0776 - date: 15 of April 2008.
2. The first commit for the public repository of OpenFOAM 1.5.x was dated as the 5th of August 2008: http://repo.or.cz/OpenFOAM-1.5.x.git...4840c8d0fd894a
3. Since these two are separate trees, they cannot be merged easily, because the internal development tree had the developments that were still not ready and needed at least some additional code cleaning up.
4. The OpenFOAM 1.6.x and 1.7.x are in the same tree and available here: https://github.com/OpenCFD/OpenFOAM-1.7.x/ - but this tree does not come from the 1.5.x tree.
5. As for 2.0.x, 2.1.x, 2.2.x and 2.3.x, they are all individual trees as well. AFAIK, this was because the trees were developed independently and too many changes were done between versions, that would require such a large commit that it would be more efficient to start a new repository.
6. 2.3.x and 2.4.x share the same tree, because 2.4.0 was originally planned to be 2.3.2, which you can thank Alexey for proposing the revised version numbers... even if it the version numbering was changed again with OpenFOAM 4.0.
7. OpenFOAM-dev, 3.0.x and 4.x all share the same tree.

Given the split nature of each repository and not having a common tree due to there being too many changes, I did a few years ago a bit of git-based hacking and stitched the trees together from 1.5.x to 2.3.x, but I have not updated since then. You can find it here: https://github.com/wyldckat/OpenFOAM-combo/
My objective with this was to figure out how the code evolved over time and what came from where and how it evolved.
Sometime later, I created a repository for covering the versions that were not released in Git form here: https://github.com/wyldckat/OpenFOAM-combo-pre15x/ - the code there is in a revised form, since the first official releases had third-party code mixed around inside the directory trees, which was revised over time in the following releases.

Either way, the current development trees at the OpenFOAM Foundation and at ESI-OpenCFD do share a backbone (even if not perfect), so it should be easier to go back and forth in the trees/branches between them.


Going to the foam-extend trees ("-dev" and "-ext" in this context refers to the forked versions):

1. The first git tree was 1.6-ext: https://sourceforge.net/p/openfoam-e...nFOAM-1.6-ext/ - it seems to contain the history for 1.5-dev as well.
2. Which in turn came from the SVN repositories: https://sourceforge.net/p/openfoam-e...ee/trunk/Core/ - there you can find the 1.4-dev and1.5-dev development line for the foam-extend project (current name).

Theoretically, it's possible to do a bit of git-based hacking and find a common point in the 1.6.x and 1.6-ext trees, although I have not yet looked for a practical common point.
The problem is that then evolving through the git history all the way up to 2.3.x or even 4.x, is not at all practical, because there have been a massive number of changes that would still require so many changes to the code, that it's more practical to work on individual chunks of code and adapt it.

Originally Posted by pbachant

What do you think? Is it even a worthwhile goal to have one OpenFOAM, and if so, how could we get there?

To me, the goal is achievable and worthwhile, but it will take a lot of time to achieve it. There should be no illusions about what the amount of effort and time is needed to achieve this, but I expect that a 5-10 year time line is achievable based on the current resources.

The way I see it, the first step is hopefully already ongoing, namely with the OpenFOAM user conferences there has been discussions regarding on how to proceed with bringing code from foam-extend to OpenFOAM-plus. I have no idea how things are going on right now on that side, but I do see the ESI-OpenCFD maintained community repository as a step towards that: http://openfoam.com/community/repository.php

If and/or once the code and features from foam-extend are brought to OpenFOAM-plus and/or community repositories, the experience gained there will help shape how the next step should be taken, namely how these repositories will interact with the developments released by the OpenFOAM Foundation. Most likely, the shape will be similar to what Pete wrote, e.g.:

Originally Posted by pbachant

If a certain player or company needed, e.g., an access function added to a certain class, they would propose that change to the core version.

But!... keep in mind the several situations that have happened in the past on the Linux Kernel project's mailing list: Be certain that the proposed change has been properly studied+analysed and that it has a low chance of it being shot down by Linus Torvalds as a nonsensical change.


I believe that I've lost a few trains of thought in the paragraphs above, but for now I'll wrap it up at this point.

Best regards,
Bruno

[pbachant]

I would not argue that the Linux kernel is the gold standard of modern collaborative software development! I guess it works for such a huge project, but for something as tiny as OpenFOAM, the barriers to entry need to be as low as possible. Being mean and picky like Linus is a good way to ensure beginners stay away. "Beggars can't be choosers."

It does seem that ESI is poised to unify OpenFOAM, given they are now based on the Foundation version, but developing on a registration-required GitLab instance is a slight hindrance.

It's interesting that foam-extend seems to have broken off over a copyright issue, whereas I assumed it was over a disagreement over software design. Is it really true that the GPL can't be enforced by an individual or informal group of individuals--an organization is required? Is it possible that with multiple copyright holders some might choose to not enforce the GPL?

[wyldckat]

Hi Pete,

Sorry for the delay:

Originally Posted by pbachant

but for something as tiny as OpenFOAM, the barriers to entry need to be as low as possible.

The level of quality control that is required for contributed OpenFOAM-based source code and the limited number of contributors who can review other contributions, makes it's currently very hard to have a low barrier to entry, unless it's basic stuff that can easily be provided on the bug tracker. That's why this page was written: https://openfoam.org/dev/how-to-contribute/

Originally Posted by pbachant

It does seem that ESI is poised to unify OpenFOAM, given they are now based on the Foundation version, but developing on a registration-required GitLab instance is a slight hindrance.

There are pros and cons with the registration requirement and as part of a larger company, it makes some sense that they chose to tighten the control a bit more.
When the divide occurred back in late 2014, I was actually expecting that OpenCFD-ESI would have made their enterprise builds only open source to clients, but fortunately they are keeping the source code available to the public and for free, along with getting closer to the community... so in contrast to what it could have been, the registration requirement doesn't look like a problem to me.

But I haven't yet seem a bug report on they issue tracker requesting for it to be made public! So if no one asks politely, then it's likely it will stay as-is.

Originally Posted by pbachant

It's interesting that foam-extend seems to have broken off over a copyright issue, whereas I assumed it was over a disagreement over software design.

As I wrote, that was only one perspective to the situation. There is another perspective, from an infrastructure standpoint: when OpenCFD started, they didn't have the resources they have now.
For example, somewhere in this forum is a thread discussing back in 2004 or 2005 how the packages could be provided at SF.net... found it: Download Bandwidth
This should at least give you a glimpse to how the original resource/infrastructure was back then.

Another good thread on this overall topic, including on how people could contribute: Roadmap for new releases

Originally Posted by pbachant

Is it really true that the GPL can't be enforced by an individual or informal group of individuals--an organization is required? Is it possible that with multiple copyright holders some might choose to not enforce the GPL?

GPL can be enforced by a single individual or group of individuals, but AFAIK only respective to his/her/their own code. There is at least one Linux Kernel developer that is known for being emphatic in enforcing the GPL and going to court himself, but I can't state for certain at the moment who it is, since there seems to be more than one.

Some years ago I did some (re)search into this and here is a quick list of what I found as examples on how contributions should be legally done:

• The GNU "global" project is quick and simple and requires copyright transfer: http://www.gnu.org/software/global/contribute.html
• On the other hand, the Open-MPI project has an extensive list of requirements, but I can't remember if it requires transfer not: http://www.open-mpi.org/faq/?category=contributing
• An article written for the FSF on how copyright assignment makes it easier to defend: https://www.fsf.org/bulletin/2014/sp...ent-at-the-fsf - right in the first paragraph:
  

  Our ability to enforce the license on packages like GCC or GNU Emacs begins with a copyright assignment.

There was a project some years ago that had 20 or 50 or 100 authors go into court to defend the joint open source project. Unfortunately I can't find the articles that explained what happened exactly back then, nor which project it was.

There a pros and cons with each legal strategy.
Among some worst case scenarios is to imagine yourself in a situation where your own open source project is put into a tight spot because of a contribution done some 5 years ago by someone who currently is no where to be found, and is being contested by some weird company that came out of nowhere and claims it had the copyright to the code or some patent built into the code that is either older or newer than the code on your project itself... The first step would be to contact the FSF to learn what to do and get legal counselling, but try to imagine for yourself what you would need to do next?!

Either way, for small projects, we usually don't need to sweat the legal details. But for bigger projects, specially those related to some existing commercial enterprises, things can get tricky. Look up "Busybox" and the legal defences they've had to do in the past.

Best regards,
Bruno

[Cyp]

The next OF workshop is sponsored both by OpenCFD and Wikki. Sounds like a big step forward to unify OpenFOAM versions. I guess that we can expect that at least two of the three main forks will merge in a near future. Am I right?

[wyldckat]

Originally Posted by Cyp

The next OF workshop is sponsored both by OpenCFD and Wikki. Sounds like a big step forward to unify OpenFOAM versions. I guess that we can expect that at least two of the three main forks will merge in a near future. Am I right?

Quick answer: My estimate last year is that it could take somewhere between 3 to 5 years for foam-extend to be fully integrated into OpenFOAM+. It's not straight forward to do such a merger... well, I say/write this mostly because to me "near future" is within 12 months, +- 1 month

Nonetheless, I am hoping we can start seeing what have they been up to since last year, during this year's OFW12. And yes, OpenCFD being an official sponsor is already a very big step!

Unfortunately I will unlikely be attending this year's workshop, so I won't be able to give any news on this myself , so I'm counting on those attending to let us know on the forum!

[lourencosm]

I think it's improtant for users interested in this topic to read the following thread:

OpenFOAM.com versus OpenFOAM.org: Which version to use?