[aerosayan]

Hello everyone,

Looking at NASA's MPI codes like CFL3D, I'm physically repulsed by how much code is conditionally not compiled when the code is compiled without MPI. They did this to be able to compile their code to run with single core (in serial mode) or to run on multiple cores with MPI.

Here's an example : https://github.com/nasa/CFL3D/blob/9...lk.F#L907-L915

Here's another example : https://github.com/nasa/CFL3D/blob/9...lk.F#L737-L748


I think this is a horrible coding practice, to only compile certain sections of code when we're using MPI, and not compile them when we're only trying to use the code for serial execution.

This introduces the possibility of horrible bugs, and increases the complexity of someone trying to understand the code, and maintain it in future.

I would like to write my code in a way that would allow me to compile with MPI, but if necessary, only use a single core for executing the whole code. This is to help in debugging, as debugging serial code is far easier than debugging distributed code.

Is there a safe and easy way to do this? It's absolutely necessary.

Stackoverflow recommends wrapping code in if(rank == 0), to only execute them if we're on core-1, but it doesn't work.

I've thought of one solution, but don't know if it will work correctly. Kindly let me know if it will work.

Potential Solution : Run code in pseudo-serial mode. Run MPI code with 2 cores. Use core-1 to only manage MPI calls, and push all work to core-2. So in core-2, it looks like everything is executing serially, and it'll be easier to debug it.

Will this work?

Thanks
~sayan

[sbaffini]

When you find codes written that way the reason is not for running single core, but to allow compiling without MPI. There might be a number of reasons, none of which I fully understand, so I don't have that in my codes, also considering the relevance of the relative non parallel executable.

MPI has a standard, and it requires all the function calls to do the obvious thing in case they are called by a single process.

In the vast majority of cases, things can be written to fit both the serial and the parallel case. For few things it really makes more sense to do otherwise. For example, if you are running serially, you don't really want to call a mesh partitioner at all, but might still do it if you prefer ultra clean code over microoptimizations.

Of course, it also depends from the code, the algorithm, etc., but I tend to believe that there is always a way to rearrange things to work for any number of processes

[sbaffini]

On a second tought, I simply have no idea why some codes have conditional MPI compilation and I know nothing about CFL3D. What I know is that it is not required at all, MPI just works consistently with 1 process as well

[arjun]

Originally Posted by sbaffini

On a second tought, I simply have no idea why some codes have conditional MPI compilation and I know nothing about CFL3D. What I know is that it is not required at all, MPI just works consistently with 1 process as well

Sometimes MPI is not available to run, in that case one shall be able to run without mpirun . This is only possible explanation.

[sbaffini]

Originally Posted by arjun

Sometimes MPI is not available to run, in that case one shall be able to run without mpirun . This is only possible explanation.

I used to think so as well, and that's what I wrote in my first answer. But, I was wondering, is it really relevant for a CFD code today to exist without MPI? I mean, the flexibility of doing this comes, in my opinion, with very little advantage (if any at all) and a relative unreasonable cost in code complexity.

In the case of CFL3D, we are certainly speaking of old code, so it is more reasonable to think that back than MPI availability might have been an issue. But I see this also in modern code, and I have no reasonable explanation for that.

[andy_]

I have not looked at the NASA code you cite but have written and maintained serial and parallel CFD codes.

In order to maximise efficiency a serial code and a parallel code will often use both different data structures and different algorithms. Likewise in parallel for shared memory and distributed memory and even in some cases for small numbers of processors and large numbers of processors with distributed memory.

If you are writing a research code where the code is being changed a lot and serial code efficiency isn't vital then there can be a case for uniform data structures and algorithms and using the distributed data structures and algorithms on a single processor. I have done this in the past by providing a dummy MPI library to link against with routines and functions throwing an error or warning if called.

[aerosayan]

Originally Posted by sbaffini

I used to think so as well, and that's what I wrote in my first answer. But, I was wondering, is it really relevant for a CFD code today to exist without MPI? I mean, the flexibility of doing this comes, in my opinion, with very little advantage (if any at all) and a relative unreasonable cost in code complexity.

Exactly what I think too. Serial codes don't have any realistic use for industrial 3D CFD solvers. Only benefit of serial code, is they help us develop and debug our code easily. Even then, it's debatable ... we can definitely debug MPI codes too, it might be better to just bite the bullet, and debug the whole parallelized code, instead of increasing complexity by including a single threaded execution mode.

[sbaffini]

Originally Posted by andy_

I have not looked at the NASA code you cite but have written and maintained serial and parallel CFD codes.

In order to maximise efficiency a serial code and a parallel code will often use both different data structures and different algorithms. Likewise in parallel for shared memory and distributed memory and even in some cases for small numbers of processors and large numbers of processors with distributed memory.

If you are writing a research code where the code is being changed a lot and serial code efficiency isn't vital then there can be a case for uniform data structures and algorithms and using the distributed data structures and algorithms on a single processor. I have done this in the past by providing a dummy MPI library to link against with routines and functions throwing an error or warning if called.

I see what you mean but, have you ever been in a situation where you tought "wow, thank god I have this serial algorithm, orherwise I couldn't solve this"?

Also, what kind of systems don't have MPI but still need to run a CFD code?

[LuckyTran]

Fluent, OpenFOAM, Starccm all use this approach of launching a master process and submitting a slave process via mpi and it requires absolutely no change in your already parallel code.

We don't even make hardware with single cores anymore, even the dumbest smart phone has a quad-core or hex-core. It would be asinine in this day and age to truly write serial code that can only compile in a single-core environment. Let the job scheduler do the job scheduling and you just write your parallel code and stop optimizing for an environment that doesn't need optimizing. If you are running CFD (or any code for that matter) in a serial environment, speed is obviously not the issue. "If you judge a fish by its ability to climb a tree..."

[andy_]

Originally Posted by sbaffini

I see what you mean but, have you ever been in a situation where you tought "wow, thank god I have this serial algorithm, orherwise I couldn't solve this"?

Also, what kind of systems don't have MPI but still need to run a CFD code?

Not depending on particular parallel libraries was good practise in the 80s and 90s when there were various libraries around and some hardware worked best using their own proprietary library/ies (and even language in the case of the transputer and Occam). They all worked pretty much the same at the lowest level. I think one of my research CFD codes supported 5 different distributed parallel libraries at one time. The NASA code may date from this time.

It depends what you classify as serial code. CFD codes designed to run on local workstations rather than remote clusters are likely to be using few enough cores that a serial/shared memory model may be preferable. Also industry tends to perform quite a lot of modest sized parametric studies often in 2D which are normally most efficiently runs as 1 task per processor with no inter-process communication.

But if one opts to support only one set of data structures and algorithms for serial, shared memory or distributed memory for a 3D CFD code then I agree it would obviously be distributed memory. The overhead when running on local workstations or running sets of independent jobs 1 per processor is unlikely to be large. The loss of shared memory unlikely to matter much in most circumstances though it can be significantly quicker to write and develop a complicated code for shared memory on a workstation.

[sbaffini]

Originally Posted by andy_

Not depending on particular parallel libraries was good practise in the 80s and 90s when there were various libraries around and some hardware worked best using their own proprietary library/ies (and even language in the case of the transputer and Occam). They all worked pretty much the same at the lowest level. I think one of my research CFD codes supported 5 different distributed parallel libraries at one time. The NASA code may date from this time.

It depends what you classify as serial code. CFD codes designed to run on local workstations rather than remote clusters are likely to be using few enough cores that a serial/shared memory model may be preferable. Also industry tends to perform quite a lot of modest sized parametric studies often in 2D which are normally most efficiently runs as 1 task per processor with no inter-process communication.

But if one opts to support only one set of data structures and algorithms for serial, shared memory or distributed memory for a 3D CFD code then I agree it would obviously be distributed memory. The overhead when running on local workstations or running sets of independent jobs 1 per processor is unlikely to be large. The loss of shared memory unlikely to matter much in most circumstances though it can be significantly quicker to write and develop a complicated code for shared memory on a workstation.

Ok, I see, it makes sense. I was indeed implicitly reasoning under the very important assumption that one would use MPI for shared memory as well. Which is what I do because I am bad at openMP and I don't have time for both programming paradigms, but of course that is not the only way.

[aerosayan]

Originally Posted by sbaffini

Ok, I see, it makes sense. I was indeed implicitly reasoning under the very important assumption that one would use MPI for shared memory as well. Which is what I do because I am bad at openMP and I don't have time for both programming paradigms, but of course that is not the only way.

How to do shared memory operations with MPI? I need it, and don't want to mix OpenMP with my MPI code.

Resources appreciated. Thanks!

[sbaffini]

Originally Posted by aerosayan

How to do shared memory operations with MPI? I need it, and don't want to mix OpenMP with my MPI code.

Resources appreciated. Thanks!

You don't, it is MPI that just works in the shared memory case as well. You may or not have a dedicated implementation in the MPI library that will handle the case more efficiently for you, but that's not required. Honestly, considering how difficult a well done OpenMP is, for me it was a no brainer.

What MPI has, but you should use them with care, are one-sided operations, where one process accesses the memory of some other (provided it was previously designated for the task) without the latter being actively involved.

[aerosayan]

Originally Posted by sbaffini

What MPI has, but you should use them with care, are one-sided operations, where one process accesses the memory of some other (provided it was previously designated for the task) without the latter being actively involved.

Awesome! Could you please state the name of these operations and functions? I can't find them without a "googleable" name.

[sbaffini]

"MPI one-sided communication" or "MPI remote memory access" are the keywords you want. But let me suggest you the two books "Using MPI" and "Using Advanced MPI" both by MIT press, you want to read them before actively working with MPI.

[aerosayan]

Thanks! It's going to be fun with MPI!!!!