[lolno]

Hi Folks,

Recently i've faced some difficulty in trying to get sum of C_UDMI value of the neighborhood cell for a given cell c at thread t due to different partition mesh existing in my cell zone.



I've tried to parallelize my code and use PRF_CSEND/PRF_CRECV so that each computing node will send the requested C_UDMI value to node-0 for calculation purposes but I encountered a weird error from MPI saying that allocated memory buffer size is smaller than the received data. (will share the code once i've access to my laptop).

Below is draft code structure of my udf without PRF_CSEND/PRF_CRECV in which I first noticed the issue of getting C_UDMI value at partition boundary mesh.

c_face_loop(c,t,n)
{
f = C_FACE(c,t,n);
tf = C_FACE_THREAD(c,t,n);

if(!BOUNDARY_FACE_THREAD(tf))
{
/* Get neighborhood cell thread
c1 = F_C1(f,tf);
t1 = THREAD_T1(tf);
c_udmi_calc += C_UDMI(c1,t1,0);
}
else
{
c_udmi_calc +=0;
}
}

The code will return the correct sum of C_UDMI if the cell zone isn't separated by any partition mesh, and to make things harder I believe you cant use global reduction with UDM (correct me if I'm wrong)

This thread and this post in ANSYS Fluent forum also faced similar problem as mine where the c_udmi value at partition boundary mesh is unavailable/wrong and to this day no solution was found to fix this issue.

Appreciate if you guys can give me some thoughts/opinion to overcome this problem! Thanks!

[lolno]

just got access to my laptop. Below is the snapshot of message passing macro from other compute node to node-0 for averaging purposes. I've encountered an error using PRF_CSEND/CRECV due to buffer size is smaller than the transfer data size. I believe i have allocated all the memory correctly in the UDF, and not sure where the error is coming from.

float neighbor_avg(cell_t c, Thread *thread, int k)
{
/* Different variables are needed on different nodes */
#if !RP_HOST /* no host calculation */
        Domain *domain = Get_Domain(1);
        Thread *tf, *t1;
        face_t f;
        cell_t c1;

        int size; /* data passing variables */
        real *c_udmi_val;
        real avg_global = 0;
        int pe = 0, n, i;

        /* Each Node loads up its data passing c_udmi_val */
        size = C_NFACES(c, thread);
        c_udmi_val = (real *)malloc(size * sizeof(real));

        c_face_loop(c, thread, n)
        {
                f = C_FACE(c, thread, n);
                tf = C_FACE_THREAD(c, thread, n);
                if (!BOUNDARY_FACE_THREAD_P(tf))
                {
                        c1 = F_C1(f, tf);
                        t1 = THREAD_T1(tf);
                        c_udmi_val[n] = C_UDMI(c, thread, k);
                }
                else
                {
                        c_udmi_val[n] = 0;
                }
        }

        /* Set node_zero to destination node */
        if (!I_AM_NODE_ZERO_P)
        {
                PRF_CSEND_INT(node_zero, &size, 1, myid);
                PRF_CSEND_REAL(node_zero, c_udmi_val, size, myid);
        }

        if (I_AM_NODE_ZERO_P)
        {
                compute_node_loop_not_zero(i) /* see definition in para.h */
                {
                        PRF_CRECV_INT(i, &size, 1, i);
                        /* Reallocate memory for arrays for node-i */
                        c_udmi_val = (real *)realloc(c_udmi_val, size * sizeof(real));

                        PRF_CRECV_REAL(i, c_udmi_val, size, i);

                        /*PRF_CSEND_INT(node_host, &size, 1, myid);
                        PRF_CSEND_REAL(node_host, c_udmi_val, size, myid);

                        free((char *)c_udmi_val);*/
                }

                for (i = 0; i < size; i++)
                {
                        avg_global += c_udmi_val[i];
                }

                avg_global = avg_global / size;
                return avg_global;
        }

        free(c_udmi_val); /* free malloc on nodes after data sent */
#endif                    /* ! RP_HOST */
}

This is the MPI recv error that I've encountered.

Fatal error in MPI_Recv: Message truncated, error stack:
MPI_Recv(224).......................: MPI_Recv(buf=000001DA1193D06D, count=6, MPI_BYTE, src=6, tag=6, comm=0x84000004, status=00000093B01FF178) failed
MPIDI_CH3U_Request_unpack_uebuf(618): Message truncated; 48 bytes received but buffer size is 6
[mpiexec@hostname] ..\\hydra\\utils\\sock\\sock.c (420): write error (Unknown error)
send of 24 bytes failed.
[mpiexec@hostname] ..\\hydra\\utils\\launch\\launch.c (121): shutdown failed, sock 844, error 10093
 The fl process could not be started.

===============Message from the Cortex Process================================

Fatal error in one of the compute processes.

==============================================================================

[AlexanderZ]

I dont have access now to my workstation so cant check by myself
regarding your problem, I can see, that function has type float, however
return avg_global;
has real type, which is double in case you are running fluent in double precision

[lolno]

yup, i'm running double precision which means my data float/real should be 8bytes length