[knuckles]

I'm implementing an algorithm which requires that I trade large amounts of data between neighbouring processes at each time step - something on the order of thousands of doubles. My understanding is that the ideal way of doing this is with one large transfer, as this should be much faster than many small ones. In MPI, I would build up a buffer on the sending process and then transfer the data all at once. In OpenFOAM, most things seem to be done using Pstream, but I haven't found documentation explaining the extent to which Pstream is buffered.

For my interest, I'm wondering:

1. Is a blocking Pstream buffered? Would "myBlockingPstream << 1 << 2;" result in two separate transfers of one int, rather than one transfer of two ints?
2. In normal operation, will a non-blocking Pstream send as soon as it can, or wait until it has to (full buffer/"wait for all transfers" command encountered)?

More importantly: for the task at hand, I'm wondering:

• Is there a way of loading a large amount of data into a Pstream's buffer and guaranteeing that none of it will be transferred until I say so?
• Is there a different or better way of transferring large chunks of data in a single operation within OpenFOAM?

[mkraposhin]

Originally Posted by knuckles

• Is there a different or better way of transferring large chunks of data in a single operation within OpenFOAM?

Hi, i'm not sure that this way is best, but it guarantee simultaneous transfer of data as one transaction.
You must use functions IPstream::read and IPstream::write.
For example, to read data on master from slave

scalarList arrayToRead (10);
label nPoints = arrayToRead.size();
IPstream::read(Pstream::scheduled, jSlave, reinterpret_cast<char*>(&arrayToRead[0]), nPoints*sizeof(scalar),
			    UPstream::msgType(), UPstream::worldComm)

And to write data from slave to master

scalarList arrayToWrite (10);
label nPoints = arrayToWrite.size();
OPstream::write(Pstream::scheduled, Pstream::masterNo(), reinterpret_cast<char*>(&(arrayToWrite[0])), nPoints*sizeof(scalar),
			UPstream::msgType(), UPstream::worldComm);

[knuckles]

Thanks for the prompt reply, mkraposhin!

This approach looks good to me. It seems to me that it's actually better than what I was originally suggesting, because you've eliminated the unnecessary step of copying the data from my external buffer into the buffer of a Pstream object for transferring.

Two questions about your solution:

• Am I correct in thinking that there's a typo in the first part - the buffer name changed from arrayToRead (at declaration) to arrayToSend (at use)?
• I notice that you asked for "scheduled" communication rather than "blocking." What is the difference, and why did you choose scheduled?

[mkraposhin]

Originally Posted by knuckles

• Am I correct in thinking that there's a typo in the first part - the buffer name changed from arrayToRead (at declaration) to arrayToSend (at use)?
  

Yes, you are right. I'll try to correct my mistake

Originally Posted by knuckles

• I notice that you asked for "scheduled" communication rather than "blocking." What is the difference, and why did you choose scheduled?
  

Actually, i'm not a professional in MPI, and this communication settings were suggested by my colleague from IT department.
I'll ask him for explanation tomorrow.

If you will find answer before me, can you be so kind to enlighten me?

[knuckles]

Thanks!

Do you happen to know whether there is a readWrite() function, for if I wanted to fully trade the data (master sends to slave and slave sends to master in one call)?