[jhz]

To my knowledge, when using a parallel solver in Fluent, the cells in the domain are partitioned (assigned) to the compute nodes. If there are 2 nodes, then the domain partition could look like:


| node 0 || node 1|

However, there is some overlap between cells assigned in terms of internal and external cells. For example, using the macro C_PART(c, t) and myid it is possible to check if a cell is internal or external to a node. The question is: When using DEFINE_SOURCE in this parallel example (running separately on both nodes), what happens in the solver when the same cell (duplicated between nodes) is assigned different sources on each node?

The same cell can be part of both DEFINE_SOURCE loops in node 0 and node 1 if, for example, it is near the partition boundary. This cell could be internal to node 0 but external to node 1. The following snippet is a sample code to demonstrate the situation I'm describing. By default, every cell will be injected an x-momentum source of 100, but if the cell is external to the compute node, it is assigned a source of 200 instead. This means that for a set of cells near the partition boundary, they are assigned source = 100 from one node's DEFINE_SOURCE but 200 from the other. A single cell in the Fluent solver cannot have two different values of source, so would the actual source for that cell be 200 + 100, or something else happens?

DEFINE_SOURCE(x_mom_source, c, t, dS, eqn){
  real source = 100;
  
  if(C_PART(c, t) != myid){ // if current cell is an external cell to node

    source = 200;

  }

  return source;
}

[AlexanderZ]

make the contour and check by your own

DEFINE_SOURCE(x_mom_source, c, t, dS, eqn){
  real source = 100;
  
  if(C_PART(c, t) != myid){ // if current cell is an external cell to node

    source = 200;

  }
  C_UDMI(c,t,0) = source;
  return source;
}

[jhz]

Unless I am misunderstanding how UDMs work, I have ran the test you suggested and it is inconclusive. It seems that each compute node will hold a separate instance of the UDM, so using C_UDMI to store the assigned source and post-processing it just confirms my code is working but does not explain how the solver is reconciling different sources for the same cell on node 0 and node 1.


I can elaborate further if it is not clear what I am asking. First, we can establish there is some fixed cell in the domain with a centroid location of (1.5, 1.5, 1.5) (for example) and let's say it just happens that when my domain is partitioned to 2 compute nodes (node 0 and node 1) that this cell can be accessed by both nodes' DEFINE_SOURCE loops. The reason could be that in n0's DEFINE_SOURCE, this cell is internal but in n1's DEFINE_SOURCE, this cell is accessed as external, but in these instances a different source is assigned according to my code. I have verified all of this so it is true. Then, using C_UDMI as you've suggested, I only know that for n0's version of UDM for this cell source=100 but in n1's UDM for this cell source=200. I already know this from my code.



The question is what is the actual imposed source value for cell (1.5, 1.5, 1.5), since it is inconceivable that a momentum source for one node's version of this cell can be different than the other (continuity problems)? What C_UDMI is storing is merely the value we ultimately return as the source but not the source as it is being handled by the solver for this cell across all nodes that include it.

[pakk]

In general, you should not define a source in a way that it depends on the domain partition. Only let it depend on physical things, then you don't have this issue.

Nothing wrong with wanting to know this anyway, but I put it here for less-experienced cfders, who otherwise might become very confused.

[jhz]

I'm not convinced that this issue can be circumvented by letting it depend on physical things. The code snippet I was using is only a demonstration but it describes a problem that would be encountered for any application where DEFINE_SOURCE needs to conditionally impose source terms based on cell location. See the following code snippet as an example.


The code here is straightforward for a serial solver: if the cell lies within some region of interest in the domain, then impose a source of 100, otherwise the source for unaffected cells is 0. I think this is a fairly common use-case for DEFINE_SOURCE. However, the problem is the same as with what's described earlier: if the domain is now partitioned to different nodes, this macro will loop over cells that are internal to one node but external to an adjacent node. It is not documented what the actual source would be for the entire solver. For such a cell, again say (1.5, 1.5, 1.5), in node 0's loop it will get assigned source = 100 and in node 1's loop it will also get assigned source = 100. The problem here is one cannot simply assume that it will just have a source = 100, because as I mentioned before there is nothing stopping you from giving it a different source like 200 in node 1. But, this seems to violate continuity. So, my question is asking what really happens for the source of this cell - whether it is actually summing 100 + 100 here or otherwise?

DEFINE_SOURCE(x_mom_source, c, t, dS, eqn){
    real source = 0;
    real centroid_array[ND_ND];
    C_CENTROID(centroid_array, c, t); // get current cell centroid

    if(centroid_array[0] < 1.0 && centroid_array[1] > 1.0){ // some arbitrary condition for cell location
        source = 100;
    }

    return source;

}

[pakk]

No, in this case it is documented what will happen.

A cell belongs to one partition and one partition only. But parallel computing requires info from other partitions, so Fluent uses the concept of internal cells and external cells.

Say that there are two partitions, and that cell A is in partition 1, but close to partition 2.

Cell A will get a value of 100 in partition 1, it is an internal cell, nothing unexpected here.
Cell B will also get a value of 100 in partition 2. But this value is only used indirectly: the only important thing is the effect on the cells in partition 2.

Then, at the end of the iteration, the nodes communicate. In partition 1, cell A is internal, so nothing changes. In partition 2, cell A is external, so its values are all updated from partition 1.

If you postprocess things, you look at internal cells, so cell A will show the value from partition 1.
The source is only used once, this is what we want.

If you let the source term depend on the partition, then results will depend on how your mesh was split, and convergence problems will occur because different partitions see different conditions, so the communication step will be problematic.

[jhz]

Thank you very much, this is the answer I was looking for!

Can you clarify which of the following is the correct interpretation of this behavior?

1. Cell A is internal to partition 1 but in partition 2 this cell is accessed (as an external cell), so whatever source is defined will just be disregarded? In this case, I think it does not matter whether the source term depends on the partition if like you said the cell belongs only to one partition - then whatever source assigned to it from another partition that sees it as external will be disregarded by the solver.

2. Or, if the solver does not necessarily disregard the assignment in one of the partitions, maybe a better wording is that: the user must not write UDFs in such a way that the source for a cell would be calculated to a different value if done in a different partition? So, in my latest example, it is fine because cell A would have source = 100 no matter if it is encountered in the DEFINE_SOURCE of partition 1 or 2.

Also, when you say post-processing, do you mean that there is a way to definitively check what the source imposed on a cell was after the fact (i.e., extract it from the flow equation)? Because as I mentioned before, if we use UDM to record the variable source it means nothing because that simply shows what we assigned separately in each partition.

[pakk]

Originally Posted by jhz

Thank you very much, this is the answer I was looking for!

Can you clarify which of the following is the correct interpretation of this behavior?

1. Cell A is internal to partition 1 but in partition 2 this cell is accessed (as an external cell), so whatever source is defined will just be disregarded? In this case, I think it does not matter whether the source term depends on the partition if like you said the cell belongs only to one partition - then whatever source assigned to it from another partition that sees it as external will be disregarded by the solver.

This interpretation is wrong. Let's say that you define cell A in partition 1 to have zero source, and in partition 2 to have a positive source.
The source is not present (zero) in partition 1, so it will have no effect there; but partition 2 thinks that there is a source, so some flux will 'leak through'. In your final solution, if you plot the sources, you will see nothing, but your results will appear to have a flux appearing from the interface.

2. Or, if the solver does not necessarily disregard the assignment in one of the partitions, maybe a better wording is that: the user must not write UDFs in such a way that the source for a cell would be calculated to a different value if done in a different partition? So, in my latest example, it is fine because cell A would have source = 100 no matter if it is encountered in the DEFINE_SOURCE of partition 1 or 2.

Yes, I think I agree with this interpretation.

Also, when you say post-processing, do you mean that there is a way to definitively check what the source imposed on a cell was after the fact (i.e., extract it from the flow equation)? Because as I mentioned before, if we use UDM to record the variable source it means nothing because that simply shows what we assigned separately in each partition.

Hmm, difficult question... The default ways of postprocessing (the normal ways to show results in Fluent, or export to Paraview, or make reports in Fluent) ignore the external cells, so you will only see the internal cells. With some programming, it is probably possible to show values from external cells, but I am not sure and I never tried this.

[jhz]

Thanks so much for your help!

[AlexanderZ]

Originally Posted by jhz

I'm not convinced that this issue can be circumvented by letting it depend on physical things. The code snippet I was using is only a demonstration but it describes a problem that would be encountered for any application where DEFINE_SOURCE needs to conditionally impose source terms based on cell location. See the following code snippet as an example.


The code here is straightforward for a serial solver: if the cell lies within some region of interest in the domain, then impose a source of 100, otherwise the source for unaffected cells is 0. I think this is a fairly common use-case for DEFINE_SOURCE. However, the problem is the same as with what's described earlier: if the domain is now partitioned to different nodes, this macro will loop over cells that are internal to one node but external to an adjacent node. It is not documented what the actual source would be for the entire solver. For such a cell, again say (1.5, 1.5, 1.5), in node 0's loop it will get assigned source = 100 and in node 1's loop it will also get assigned source = 100. The problem here is one cannot simply assume that it will just have a source = 100, because as I mentioned before there is nothing stopping you from giving it a different source like 200 in node 1. But, this seems to violate continuity. So, my question is asking what really happens for the source of this cell - whether it is actually summing 100 + 100 here or otherwise?

DEFINE_SOURCE(x_mom_source, c, t, dS, eqn){
    real source = 0;
    real centroid_array[ND_ND];
    C_CENTROID(centroid_array, c, t); // get current cell centroid

    if(centroid_array[0] < 1.0 && centroid_array[1] > 1.0){ // some arbitrary condition for cell location
        source = 100;
    }

    return source;

}

can you prove, that source will be applied several times? (any pictures?)
I've never checked this , however, in my vision the source term is been defining first for the whole domain (and 100 could be overwritten with 200 from your example) and computation starts later, so situation of source summation is not possible.
If you have data which proves the opposite, please show.

there are special loops

begin_c_loop_int(c, tc)
{
} end_c_loop_int(c, tc)

begin_c_loop_ext(c, tc)
{
} end_c_loop_ext(c,tc)

and few others to determine the cells, you want to have in your loop explicitly. (ofc not possible to use inside define_source)

[jhz]

Originally Posted by AlexanderZ

can you prove, that source will be applied several times? (any pictures?)
I've never checked this , however, in my vision the source term is been defining first for the whole domain (and 100 could be overwritten with 200 from your example) and computation starts later, so situation of source summation is not possible.
If you have data which proves the opposite, please show.

I'm not sure what you mean when you say "source will be applied several times". All I know for certain is that there exists many cells which will be accessed as internal to some node and external to an adjacent node in the respective DEFINE_SOURCE loops, and we know that a source must be returned for each cell to pass to the solver. I believe we all agree on this behavior, since it is known that DEFINE_SOURCE loops through all internal and external cells of a partition. The whole point of the question is to ask for the behavior so I am not claiming that summation is in fact happening. I think pakk's answer makes the most sense to me: it is the responsibility of the UDF programmer to ensure that for the same cell (i.e., fixed position in domain), the source assigned to it in any partition must be the same. Otherwise, continuity cannot be enforced.

I'm not sure about your description of the source where "100 will be overwritten with 200". I think you are referring to my first example where based on C_PART, external cells will have a source of 200? I don't think your logic makes sense because while it is true UDFs are executed before solver operations, there's no order of execution between the nodes - everything is done simultaneously unless there is global reduction (like GRFSUM1). Because of this, I don't think you can claim one source assignment will "overwrite" another since that implies order.

Again, I'm not sure what the exact behavior is. To my knowledge, there is no way to test/log what source the solver actually uses for a cell accessed in this "duplicate" way from multiple partitions. If we log using UDM or otherwise, we only get information to confirm that our code is correct (e.g., if we assigned it 100 in node 1 and 200 in node 2). We have to understand here that when we write in DEFINE_SOURCE

return source;

this is just an API to give a value back to the solver. However, the information is certainly passed twice, so for a specific cell A, node 1 will return 100 while node 2 will return 200. The value you return doesn't necessarily have to differ but I can confirm even if it does differ there's no error so it means it is allowed. This raises my question because clearly this cell can only have 1 source value but yet you are passing 2 potentially different values to the solver. What happens afterwards when the solver gets this information, I do not know.

[AlexanderZ]

my logic is very simple
as you don't have any case to show it's just a wasting of time