[Gerry Kan]

Folks:

I am not sure if I heard this right ... maybe another person in the know could confirm this.

In an MPI code I am working on, I made a few calls to RANDOM() to simulate stochasticity in my solution (yes, it is also seeded). Someone looked at it and told me that because of this "the results will depend on the chosen domain decomposition / number of cores."

To be honest I have never heard of any explicit dependence of RANDOM() with domain decomposition, aside from the remote possibility that it might affect the random seed because it depends on (say) the timestamp of the particular machine it is running on.

Have anyone heard of or experienced something like this?

Gerry.

[sbaffini]

If you properly/deterministically seed it on each process in the same way (which, however, unless otherwise needed, I would only do for tests), there should be no concern whatsoever. One case where it is important to do it correctly is for LES/DNS inlet or initializations. You need to be sure that each process produces the same sequence of random numbers.

However, admittedly, this is not something I used to test whenever I had access to a large machine with multiple nodes, only on single node ones.

Still, I think there are other sources of differences when running on different number of processes.

At the bit level, each process runs a loop over its portion of cells. Whenever this changes, the loop changes, and so I expect differences at machine precision level. This happens, say, for an explicit code. But a larger difference arises in linear solvers, because the order you use to update the variables will necessarily depend from the partitioning. This is usually under control as well (yet, certainly not at the bit level), but my experience is that more important differences may arise for non well conditioned matrices (say very high Re) when you restart a simulation with different number of processes. Nothing that few iterations can'cure, but it is there.

[sbaffini]

Basically, if you need the same sequence on each process but, at least in production, you don't want a fixed seed, you can let master seed randomly, get its seed and broadcast to the other processes. There is also a dedicated routine from Fortran 2018.

None of this is battle tested for heterogeneous systems

[Gerry Kan]

Paolo:

Thanks for the answer. I thought he might be referring to the seed issue, but I still don't see how the seed could make the code domain dependent. Even if there were any domain dependency from that (for this code you can't even restart with different decomposition), a few iterations would have cleared it.

As an aside, there are enough spots that are really grid and decomposition dependent ... these people should be spending time looking into those instead.

Gerry.

[sbaffini]

Well if you seed each process differently (or you don't seed at all) you are certainly introducing differences among the processes. But, the question is: is this desired behavior or not?

In the LES inlet example that I made, where I was thinking to Smirnov like methods, the random numbers are used to build a global spatial distribution of a field variable, so in order for this field variable to have a coherent spatial structure across the whole inlet (or domain for the initialization case) all the processes must have the same random number sequence.

But I can imagine a lot of scenarios where the opposite is actually required and you want different processes to work on different random realizations.

So, in the end, I would still always leave an open door to force a deterministic behavior in tests, but otherwise this seems application dependent.