[aerosayan]

My code is getting very complicated, and due to recent changes some bugs were introduced that went unnoticed for some time. That made me extremely sad. I want a method to ensure that new addition to the codebase, doesn't introduce bugs.

How can implement such a regression testing system for CFD codes?

Specifically, I'm facing difficulty as CFD codes work on arrays containing thousands of values, and I don't know how I can implement a system to validate the results. Preferably something already exists, and I don't have to reinvent the wheel.

[ssh123]

That's the reason why I'm thinking of Java and migrating my C++ CFD code to Java. When implementing your code in Java, you can simply employ several test frameworks eg. Junit for regression testing.

For your case I think you can create wrappers to Java for methods which you want to test and employ Junit to call them.

[aerosayan]

Originally Posted by ssh123

That's the reason why I'm thinking of Java and migrating my C++ CFD code to Java. When implementing your code in Java, you can simply employ several test frameworks eg. Junit for regression testing.

Java for CFD? Probably going to end badly.
You're giving up performance for your code, in hopes of a better experience in testing?
That seems counter intuitive.

[ssh123]

Let me try with a simple 2D Euler equations solver. I'll update my finding in this thread.

[sbaffini]

Originally Posted by aerosayan

My code is getting very complicated, and due to recent changes some bugs were introduced that went unnoticed for some time. That made me extremely sad. I want a method to ensure that new addition to the codebase, doesn't introduce bugs.

How can implement such a regression testing system for CFD codes?

Specifically, I'm facing difficulty as CFD codes work on arrays containing thousands of values, and I don't know how I can implement a system to validate the results. Preferably something already exists, and I don't have to reinvent the wheel.

It really depends from what you already have, both in terms of testing and versioning control/branching model. I mean, hopefully, you already have:

1) unit tests: another piece of code where you basically test all the modules/libraries that couldn't, otherwise, be tested (say, a gauss-jordan or tridiagonal routine, a triangle-box intersection routine, etc.)

2) verification cases: where you test your whole code against grid and time step refinements of as much analytical solutions you can (I typically include the same case under different reference systems here)

3) validation cases: where you test your whole code against known experimental data

4) a versioning control method: where you have git/svn/whatever to reliably track changes in the code

5) a branching model: where you have very rigid rules that specify how a code goes into production, how bugs get fixed and how new developments get into production code. Say, this

6) Some mechanism of continuous integration/development: where all the above is automated and properly orchestrated as a function of your branching model. At the very minimum, all the tests (unit, verification and validation) should be performed when some code is ready to become production.

I guess you are lost somewhere in between 1 and 6?

[Alex C.]

Originally Posted by aerosayan

My code is getting very complicated, and due to recent changes some bugs were introduced that went unnoticed for some time. That made me extremely sad. I want a method to ensure that new addition to the codebase, doesn't introduce bugs.

How can implement such a regression testing system for CFD codes?

Specifically, I'm facing difficulty as CFD codes work on arrays containing thousands of values, and I don't know how I can implement a system to validate the results. Preferably something already exists, and I don't have to reinvent the wheel.

I'm not currently developing CFD, but still do C++ production code.

I'm using Google Test for unit testing and FitNesse for system testing.

I highly recommend Google Test framework.

FitNesse 'works' fine, meaning I can successfully test my software, but the setup is trickier to make than it is hard to break... which makes it a bit annoying. I work in a large organization, so I live with it. I would probably explore other solutions in a different context.

[aerosayan]

Originally Posted by sbaffini

I guess you are lost somewhere in between 1 and 6?

I have git based version control, a basic branching pipeline, and some basic unit tests.


Integration testing seems challenging, as we have to compare the current solution with the previously validated solution data. I'm currently guessing that we have to manually validate the solution once, then store the result in a file, which we can later use for our integration tests.


Of course that becomes challenging when the mesh sizes are very large.

[sbaffini]

Originally Posted by aerosayan

I have git based version control, a basic branching pipeline, and some basic unit tests.


Integration testing seems challenging, as we have to compare the current solution with the previously validated solution data. I'm currently guessing that we have to manually validate the solution once, then store the result in a file, which we can later use for our integration tests.


Of course that becomes challenging when the mesh sizes are very large.

Then, I guess, your code just needs a monitor feature, so that different point/surface/volume values/norms can be extracted. If written to file, combined with residuals, you should have a fairly large set of posssible comparisons for different metrics. This could be used both in validation and verification. Of course, you still need a reference toward which make comparisons, but this could now be just a line in a text file (yet, I would still sotre the whole file).

But, let me tell you that, frameworks and automations can't solve all the problems. My point of view on these things is that, together with everything we already mentioned, they are just needed pieces of a formal process.

Like having a perfect score in a driving licence exam, they won't, by themselves, save you from accidents (nor will any autopilot feature, for that matter, if you know what I mean).

In my case for example, I like the fact that the process is completely formalized and all (I even review each one of my own commits), but when something new gets in, you can't completely outsource things to some automation. If this was the case, there should be examples of bugfree softwares, but there aren't.