[gucong]

Originally Posted by alberto

My understanding is that a lot of the recent API changes were motivated by the need of differentiating access functions (ref/non-ref) to support some architecture. Others were due to cleanup and rationalization.

I don't like to change my code to keep up with API changes, but overall it isn't terrible. I actually found it quite simple to follow -dev directly, rather than freeze the development branch of my code on a release and wait until it is stabilized.

Definitely I do not agree with the RHEL approach applied to a CFD code: if you keep bugs in a CFD code to ensure API compatibility, you obtain incorrect results.

The sad thing is that following the -dev is kind of the only way to move on, because release versions are not maintained. There are no bugfixes, no new features, etc. I guess most bugfixes and features don't actually require API changes in the libraries (see foam-extend). I am just saying the time that the community are forced to spend on following API changes is not well spent. And these changes can often get in the way unexpectedly after a pull while we are working on some "real" problems. Codes shared in the community get rotten extremely fast if without active maintenance. If they maintain a API-stable line with bugfixes and new features, and a separate development line for restructuring core libraries, then we only need to do this once every several years, which can save a lot of trouble personally and community-wise.

[wyldckat]

Greetings to all!

Excellent, now we're having a discussion! Let me just answer Alberto's side-question first:

Originally Posted by alberto

Did you have to do anything in terms of "permissions" to do this, or did you just port the code? I was thinking of porting some of the extra BCs, but I don't know how that would work without a completely independent re-implementation in terms of copyright transfer.

There were three important parts for this:

1. The code changes in the OpenFOAM-history repository has already been transferred to the OpenFOAM Foundation, at least for the most part. Let me check... from here: http://www.openfoam.com/download/openfoam-history.php
   

   09/2015 — OpenCFD transfer copyright of its development repository to OpenFOAM Foundation

   Which means that the whole OpenFOAM-history repository can be used for code to be used by the OpenFOAM Foundation.
2. I have permission to contribute, since the person in charge at the company I work at has signed the contributors agreement.
3. I had to do adjustments where needed and test things.

The problem at the moment is that much of the new features that are in the OpenFOAM-history repository needs additional work, both for syncing up with the latest OpenFOAM-dev repository, as well as fixing and continuing work that was started in those features. Said work has continued in the OpenFOAM-plus development repository, which would require getting ESI-OpenCFD signing a transfer agreement for the new developments, which may or may not be possible, depending on who developed said fixes.


Now, regarding the ongoing discussions:

Originally Posted by alberto

Definitely I do not agree with the RHEL approach applied to a CFD code: if you keep bugs in a CFD code to ensure API compatibility, you obtain incorrect results.

I unfortunately used a comparison for the long term, based on the closest similar development profile. I'll address things better below, after addressing some of the mentioned details.

Originally Posted by gucong

I still consider such constant/rolling API changes in such scales for a library project with 10+ years of history is insane. I understand some API changes are essential for certain bug fixes and long term health of the project. But what about massive renaming with only marginal cognitive benefits? For example, they recently renamed very fundamental things like internalField() to primitiveField().

Well, OpenFOAM's originated from FOAM, which started in 1989 with a single author... so overall it's already beyond the 25 year mark. But to put this into perspective, we're talking about a project that has been developed as full time job by only 5 to 10 individuals, which means that the total number of hours in development is very likely much smaller than commercial CFD software, although commercial CFD software also has to account for documentation, GUIs and support. This to say that 20 years in OpenFOAM's development lifetime may equate to 5 years in Fluent of Star-CCM+ lifetime, which is a feat of its own.

In addition, AFAIK much of the time spent in development with OpenFOAM has been done to meet client's requests, which has lead to some situations where the optimal approach was not done right away. This is finally being sorted out, although it will take some more time until the core API changes stop occurring, at least from what I can figure out.

Originally Posted by gucong

Actually, it seems to me that the developers are totally not concerned with API stability and, in turn, the lifes of community application and enhancement developers.

Mmm... this is actually a two sided issue, not just one sided. Let's see if I'm able to explain this properly...
The development currently ongoing by CFD-Direct has been focusing on the architectural standing of OpenFOAM, but new developments have been made as well. The development seems one-sided and a recent couple of blog posts have made it look even more... er... unpopular with many people in the community. I'm referring to the following blog posts:

• http://cfd.direct/openfoam/free-software/issues-pt-1/
• http://cfd.direct/openfoam/free-software/issues-pt-2/

One of the emphasis made on those blog posts is regarding quick turnover of bug fixes and new features, including timely feedback from the bug reporters.

And this is where the other side comes in: not enough feedback/contributions. From my side, who has been trying to tackle 1-3 bugs per weekend on the OpenFOAM bug tracker, bugs big and small, which feels rather crazy to me that I'm currently a main contributor, when I actually want to find even more time to contribute new features, not just bug fixes. There are a few more contributors who have been contributing as much as they can, but we are still very far from anything similar to the bigger projects, such as the Linux Kernel or LibreOffice.

Originally Posted by gucong

It is almost like they are intentionally breaking things others built around OpenFOAM.

I know the feeling. MATLAB actually used to do that on purpose several years ago to enforce people to buy new licenses, so that their own code would work with newer versions; I'm not sure if they still do that.

Sorry, just a quick side-note before I address the bigger issue:

Originally Posted by gucong

There are no bugfixes, no new features, etc. I guess most bugfixes and features don't actually require API changes in the libraries (see foam-extend).

Are you sure regarding foam-extend? There have been several changes in foam-extend's API, some of which are incoming with the release of foam-extend 4.0. They might not be as frequent as when compared with OpenFOAM 3 and 4, but still... I can think of a few API changes that broke older code over the past few years:

1. A bug fix regarding symmetric tensor product, required reworking expressions that relied on that operator.
2. The full integration of the block-coupled matrix solvers, required a few changes in the code that relies on those solvers.
3. There have been considerable changes for adding compatibility with OpenFOAM 3.0 case formats, coming in foam-extend 4.0 (OK, this one is OpenFOAM's fault ).

But also keep in mind that there haven't been enough resources available to bring in several other features from OpenFOAM... although it is discussable if that's a necessary effort or not.

Originally Posted by gucong

When you are talking about bug persisting for 3-5 years in RHEL, remember they do it for a reason. Namely, API stability is considered more important in those cases. At least the bugs are known and documented, while a slight API change could cause unknown bugs in other parts of the system. That's partly why RHEL is preferred in any serious or critical situations. In my opinion, API stability shows the seriousness of the library developers and respect to users of the library. At least don't do that in a rolling release model, keep it stable within major versions that stay maintained and developed spanning several years. In contrast, versions now are just tags in OpenFOAM-dev, which is the only actively developed line, if I understand it correctly.

I'll address this after the following quote:

Originally Posted by gucong

The sad thing is that following the -dev is kind of the only way to move on, because release versions are not maintained. There are no bugfixes, no new features, etc. I guess most bugfixes and features don't actually require API changes in the libraries (see foam-extend). I am just saying the time that the community are forced to spend on following API changes is not well spent. And these changes can often get in the way unexpectedly after a pull while we are working on some "real" problems. Codes shared in the community get rotten extremely fast if without active maintenance. If they maintain a API-stable line with bugfixes and new features, and a separate development line for restructuring core libraries, then we only need to do this once every several years, which can save a lot of trouble personally and community-wise.

Let's see if I'm able to put this into a perspective on which we can work with and move forward, while solving at least some of these problems.

First a review of a particular detail: version numbers. There was a switch made sometime ago to a semantic version numbering system, after a discussion Alexey (alexeym) and I had here on the forum: http://www.cfd-online.com/Forums/ope...numbering.html
He essentially was complaining about the very same main issue: API changes and case set-up changes between versions are not documented well enough and the version numbers were not even reflective of those changes.
On that thread you can read about the two sides of discussion and the conclusion of what was the right change that needed to be made (Alexey was right, although the numbering change), which resulted in the following bug report and acceptance: http://bugs.openfoam.org/view.php?id=1674
If it weren't for that discussion and bug report, instead of OpenFOAM 4.x, we would be dealing with OpenFOAM 2.6 or 2.7 or something like that... with still these major API changes.

That said, although the numbering was being done with the semantic number convention for following versions 2.4.0 and 3.0.0 and 3.0.1, it was changed again with OpenFOAM 4.0. I was already somewhat expecting this (see post #8), when I referred to how GCC changed their numbering convention. I'll address the reason for this later below.

This was a somewhat roundabout way to say that:

1. The API changes should have been expected with the change in version between 3.x to 4.x.
2. If people contribute, things can change for the better.

Now, here's where it gets trickier to explain: RHEL 3.x to 7.x are fully not compatible API-wise between them. Neither is GNOME 2.x and 3.x. This to say that changes will be made between major versions and should be expected.
Sorry, you already know this, but I wanted to emphasize that the version numbers are still reflective of the compatibility that is expected from OpenFOAM's releases.

But this leads to the problem for my initial comparison: OpenFOAM's major version changes is currently all too frequent, with no long term support available for any version. Technically this is the critical issue that I'm seeing in everyone's complaints, specially regarding my comparison with RHEL.

My emphasis on the RHEL comparison is regarding maturity: the developments currently ongoing within OpenFOAM's source code are headed towards maturity. This means that the switch to a RHEL model cannot be achieved overnight, therefore the current developments are focused on reaching that stage, although at the sacrifice of backwards compatibility. Postponing those changes are currently be prohibitive on OpenFOAM's side, although some of them seem rather strange...


What can be done in the meantime? Here's what I see that needs to be done within or along with the OpenFOAM project:

1. Better documentation of API changes between major versions.
   1. This is already being done in the Git commit messages, but there is no single document that has the list of changes.
   2. Compiling said list takes time, which can be done by anyone in the community and can be contributed either at openfoamwiki.net or directly on the bug tracker for OpenFOAM.
2. Long term support of a major version. This is needed either way, regardless of how much is re-worked or not in the OpenFOAM-dev repository.
   1. This requires funding. A considerable amount of funding. It can be a one-person job, but it'll be pretty much a full time job. And this is essentially one of the reasons why there isn't a long term major version: funding. This is why there is no long term support and why the current emphasis is on a single bleeding edge development line on the Foundation's side. But the same goes for OpenFOAM-plus and foam-extend.
   2. For example, we could use OpenFOAM 3.0.x as a basis for the long term version at the very least as a proof of concept, where a good chunk of new features could be backported from 4.x to 3.0.x and released as 3.1.x and so on.
   3. Eventually we'll start to see the limitations on this and the time spent in the backports being somewhat counter-productive to some people as well.
   4. Nonetheless, this would effectively bring a more reliable way for people to still use older code that is compatible with 3.x.
   5. That said, this can be done by anyone with enough experience with OpenFOAM and software development and it can be done by any company or freelancer. The GPL license allows that and the contribution/cooperation would be very much welcome!
   6. Side note: Caelus-CML started with a fork from OpenFOAM 2.1.1 (if I'm not mistaken) and has had minimal changes to the API, because they have mostly started fresh with a core clean structure with which I can develop on top.
3. As for old contributions not maintained at the moment, this should be possible to do on OpenCFD-ESI's development platform in the future, if all goes well. The session done earlier this week at the user conference will hopefully being forth a platform under which we can all contribute to the maintenance of the older contributions.

Originally Posted by gucong

I am just saying the time that the community are forced to spend on following API changes is not well spent.

For this I'm not able to give a good solution for the short term. The best that I can think of is that we should have a way for newcomers to gain experience by contributing to the older contributions that need maintenance. This is a great way to learn the basis and how things work and it's from where I've gain a fair part of my experience.
I learned at the OFW11 this year that a platform/server had been prepared sometime ago for this, but it was not made public because some critical information was missing for making the project official on foam-extend's side.

The major changes within OpenFOAM have been a constant factor since 1.0 and it's something that will continue to occur unless someone or many people sponsor a more sustainable development with 3rd party developers (long term support). Regardless of funding, as I mentioned before, this is a two-sided issue:

1. People are doing their best to improve things on OpenFOAM's side;
2. And things can only get better if the people who use OpenFOAM will contribute more to OpenFOAM itself.

Pulling back the frequency of changes on OpenFOAM's API is not something that is sustainable at the moment, but will reduce it's frequency sooner than later, since the core code is gradually being reworked for maturity.
But the communication problems can be solved, if people get more involved and communicate more. Otherwise, it'll continue to look like it's a one-sided development line.

Please keep in mind that what I wrote here is along the efforts that I've been doing as well. Take a look at my Github repository list and you'll see that I've also been trying to help with what I can regarding old contributions: https://github.com/wyldckat?tab=repositories - And they are indexed at openfoamwiki.net

Best regards,
Bruno

[wyldckat]

Originally Posted by wyldckat

1. Better documentation of API changes between major versions.
   1. This is already being done in the Git commit messages, but there is no single document that has the list of changes.
   2. Compiling said list takes time, which can be done by anyone in the community and can be contributed either at openfoamwiki.net or directly on the bug tracker for OpenFOAM.

FYI: With the release of OpenFOAM+ v1612+, on OpenCFD's side, this is being worked on at the following pages:

• Upgrade guide for users (case set-up): http://openfoam.com/documentation/us...rade-guide.php
• Upgrade guide for source code changes: http://openfoam.com/documentation/de...rade-guide.php

These are mostly applicable for OpenFOAM-dev and OpenFOAM 4.* as well.