[billie]

I try to obtain the mass flow through a faceZone with swak4foam using the following function in my controlDict:

    flowIntFluid
    {
        type swakExpression;
        valueType faceZone;
        region fluid_water;
        zoneName int_fluid;
        expression "phi*flip()";
        accumulations (
            sum
        );
        verbose true;
    }

I have created a cellSet for the slave cells of the faceZone with the following action in topoSetDict:

    {
        name    int_fluidSlaveCells;
        type    cellSet;
        action  new;
        source  faceZoneToCell;
        sourceInfo
        {
            name int_fluid;
            option slave;
        }
    }

Everything works okay when I run the simulation on a single processor however when running in parallel the expression returns garbage, which is probably because of the set information missing in the individual processor subdirectories.

Is it possible to distribute this set information to the individual directories?

[billie]

I figured out that I do not need swak4foam for this task, using the following function in controlDict does the same and works when running in parallel.

    flowIntFluid
    {
        type            faceSource;
        functionObjectLibs ("libfieldFunctionObjects.so");
        enabled         true;
        outputControl   timeStep;
        region          fluid_water;
        log             true;
        valueOutput     false;
        source          faceZone;
        sourceName      int_fluid;
        operation       sum;

        fields
        (
            phi
        );
    }

[gschaider]

Originally Posted by billie

Everything works okay when I run the simulation on a single processor however when running in parallel the expression returns garbage, which is probably because of the set information missing in the individual processor subdirectories.

Is it possible to distribute this set information to the individual directories?

No. Sets are not decomposed. But what works is running the utilities that were used to create the sets on the parallel case

[gschaider]

Originally Posted by billie

I figured out that I do not need swak4foam for this task, using the following function in controlDict does the same and works when running in parallel.

    flowIntFluid
    {
        type            faceSource;
        functionObjectLibs ("libfieldFunctionObjects.so");
        enabled         true;
        outputControl   timeStep;
        region          fluid_water;
        log             true;
        valueOutput     false;
        source          faceZone;
        sourceName      int_fluid;
        operation       sum;

        fields
        (
            phi
        );
    }

Of course this result is only correct if all faces in the zone are oriented in the same way

[billie]

Originally Posted by gschaider

No. Sets are not decomposed. But what works is running the utilities that were used to create the sets on the parallel case

Running topoSet on the decomosed case creates the sets but I get garbage output as well.

[billie]

Originally Posted by gschaider

Of course this result is only correct if all faces in the zone are oriented in the same way

As I said running on a single core the swak4foam expression works but not when running on multiple cores, even when the set is created on the parallel case.

Running funkyDoCalc on the reconstructed case with the following dictionary works as well:

massFlow
{
    valueType faceZone;
    zoneName int_fluid;
    accumulations (
        sum
    );
    expression "phi";
}

So the problem is not related to the face orientation.

[billie]

Originally Posted by gschaider

Of course this result is only correct if all faces in the zone are oriented in the same way

I checked the face normals in paraview and they are all oriented in the same way. So the slave cell set is not required in this case. I still do however get the wrong output when running in parallel.

[gschaider]

Originally Posted by billie

Running topoSet on the decomosed case creates the sets but I get garbage output as well.

Define "garbage". Anyway: there may be a corner case where the sums are not correct: if a face in the set is on a processor-boundary. In that case it may be counted twice or not at all

[gschaider]

Originally Posted by billie

As I said running on a single core the swak4foam expression works but not when running on multiple cores, even when the set is created on the parallel case.

Running funkyDoCalc on the reconstructed case with the following dictionary works as well:

massFlow
{
    valueType faceZone;
    zoneName int_fluid;
    accumulations (
        sum
    );
    expression "phi";
}

So the problem is not related to the face orientation.

Then it worked OK in that case. But in general phi is determined by the order of the cells. Suppose you have a "channel" with four cells. Digits are are the cell numbers and letters are the faces in the face Set. Flow goes from left to right (the way it always does):

0 A 1
2 B 3

Here on A and B phi is positive. In another case

1 A 0
2 B 3

on A the phi is negative while on B it is still positive. Summing it up without multiplying it with the flip will give you a wrong result. That is what I mean with "orientation" and am not sure how you'd visualize this with paraview (as you mentioned in another posting)

[immortality]

what does flip do?

[billie]

Originally Posted by gschaider

That is what I mean with "orientation" and am not sure how you'd visualize this with paraview (as you mentioned in another posting)

Hm, I did look up the the face normals in paraview using the NormalGlyphs filter plus the cellNormals all have the same color.

Originally Posted by gschaider

Define "garbage". Anyway: there may be a corner case where the sums are not correct: if a face in the set is on a processor-boundary. In that case it may be counted twice or not at all

This may be one reason but even in this case the output should not change randomly or am I wrong. The output below btw is what I mean by garbage.

Output for single processing:

#         Time          sum
            1     -0.195176
            2    -0.195298
            3    -0.195382
            4     -0.19532
            5     -0.19533
            6    -0.195343
            7    -0.195337
            8    -0.195337
            9    -0.195339
           10    -0.195338
           11    -0.195338
           12    -0.195339
           13    -0.195339
           14    -0.195338
           15    -0.195338

Output for parallel processing:

#         Time          sum
            1      -172000
            2-6.34742e-310
            3 -0.000473733
            4         -nan
            5    0.0251599
            6     -4042.94
            7  -0.00175415
            8-8.93413e-310
            9  -0.00423105
           10         -nan
           11      5.71386
           12  3.23021e-51
           13       126807
           14     -15.0175
           15-1.42553e+248

One thing I do not understand. The faceZone consists of 204 faces. When runnig topoSet on the parallell case all 204 faces are on one processor and for all other processors but one there are zero faces. On one processor it looks like this:

1
(
-1
)

Could this be the problem or is this normal?

[gschaider]

Originally Posted by billie

Hm, I did look up the the face normals in paraview using the NormalGlyphs filter plus the cellNormals all have the same color.

That is not enough as you see from my example. Especially as paraview may have "corrected" the orientation before displaying it

Originally Posted by billie

This may be one reason but even in this case the output should not change randomly or am I wrong. The output below btw is what I mean by garbage.

Output for single processing:

Output for parallel processing:

Remark: Use CODE instead of QUOTE . That way your example shows up in your output

That really looks like garbage

Originally Posted by billie

One thing I do not understand. The faceZone consists of 204 faces. When runnig topoSet on the parallell case all 204 faces are on one processor and for all other processors but one there are zero faces. On one processor it looks like this:

Could this be the problem or is this normal?

It depends on the decomposition, but it is very well possible that all faces in the zone are on one processor. The "-1" on one processor may be that problem (although I'm a bit surprised that you didn't get a segmentation fault) as it will try to access data that isn't there (I don't remember checking for faces smaller than 0 anywhere)

[gschaider]

Originally Posted by immortality

what does flip do?

A function returning -1 or +1 that helps working around the situation I described above (the flipMap is usually set by some other utility)

[billie]

So probably this is meshing related. The mesh comes from Hypermesh and is exported directly to OpenFOAM format. It contains one internal surface which I want to use for mass flow calculation. From the export a faceZone is created for the internal surface as well as a boundary patch which I remove from the boundaries file as else the meshCheck fails. Afterwards I run renumberMesh to get rid of the unordered face warning

 ***Faces not in upper triangular order.
  <<Writing 186 unordered faces to set upperTriangularFace

Then I run topoSet to create the cell set for the slave cells of the faceZone.

Currently I do not know a better way of creating the faceZone and the required cell set, but maybe there is a more elegant way to do this.

[gschaider]

Originally Posted by billie

So probably this is meshing related. The mesh comes from Hypermesh and is exported directly to OpenFOAM format. It contains one internal surface which I want to use for mass flow calculation. From the export a faceZone is created for the internal surface as well as a boundary patch which I remove from the boundaries file as else the meshCheck fails. Afterwards I run renumberMesh to get rid of the unordered face warning

 ***Faces not in upper triangular order.
  <<Writing 186 unordered faces to set upperTriangularFace

Then I run topoSet to create the cell set for the slave cells of the faceZone.

Currently I do not know a better way of creating the faceZone and the required cell set, but maybe there is a more elegant way to do this.

If the serial mesh (which seems to work) is the only thing your mesher wrote, then I guess the problem is with some OpenFOAM-utility. If you didn't decompose the mesh with decomposePar but your mesher wrote the processor-directories (I understand that some meshers are capable of this) then it might be a mesher problem.

Have you tried leaving renumberMesh out (whether then things work)?

[billie]

Originally Posted by gschaider

If the serial mesh (which seems to work) is the only thing your mesher wrote, then I guess the problem is with some OpenFOAM-utility. If you didn't decompose the mesh with decomposePar but your mesher wrote the processor-directories (I understand that some meshers are capable of this) then it might be a mesher problem.

The serial mesh is created by Hypermesh and I use decomposePar for decomposing. So there is probably an OpenFOAM-utility at fault. Until now I tried openfoam-git in combination with swak4foam-hg as well as openfoam-2.2.0 in combination with swak4foam-0.2.3.

Originally Posted by gschaider

Have you tried leaving renumberMesh out (whether then things work)?

Leaving it out does not help.

My current workflow is as follows (it is a multi-region case btw.):

runApplication transformPoints -scale '(0.001 0.001 0.001)'
runApplication renumberMesh -overwrite
runApplication checkMesh -allGeometry -allTopology
runApplication splitMeshRegions -cellZonesOnly -overwrite

for i in solid_aluminium solid_steel
do
    rm -f 0*/${i}/{mut,nuTilda,k,epsilon,omega,p_rgh,U}
done

for i in fluid_water
do
    rm -f 0*/${i}/{dummy}
done

for i in fluid_water solid_aluminium solid_steel; do
    if [ -f log.changeDictionary.${i} ]; then
        echo "changeDictionary already run on ${PWD}: remove log to re-run"
    else
        changeDictionary -region ${i} 2>&1 >log.changeDictionary.${i}
    fi
done

runApplication decomposePar -allRegions

COUNT=0
NPROCS=$(getNumberOfProcessors)

while [ "${COUNT}" -lt "${NPROCS}" ]; do
    if [ -f log.topoSetDict.processor${COUNT} ]; then
        echo "topoSetDict already run on ${PWD}: remove log to re-run"
    else
        topoSet -case processor${COUNT} -dict ${PWD}/system/topoSetDict \
            -region fluid_water 2>&1 >log.topoSetDict.processor${COUNT}
    fi

    let 'COUNT += 1'
done

runParallel $(getApplication) ${NPROCS}
runApplication reconstructPar -allRegions -latestTime

[gschaider]

Originally Posted by billie

The serial mesh is created by Hypermesh and I use decomposePar for decomposing. So there is probably an OpenFOAM-utility at fault. Until now I tried openfoam-git in combination with swak4foam-hg as well as openfoam-2.2.0 in combination with swak4foam-0.2.3.

If the problem is the faceZone with "face" -1 than older versions of swak will show that behaviour too. The only way to know for sure is if you have a Debug-version of OF to run the case there. In the debug-version an access to index -1 will make the run fail and give a stack-trace

Originally Posted by billie

Leaving it out does not help.

Different decomposition? Just curious if this makes the -1 go away

Originally Posted by billie

My current workflow is as follows (it is a multi-region case btw.):

runApplication transformPoints -scale '(0.001 0.001 0.001)'
runApplication renumberMesh -overwrite
runApplication checkMesh -allGeometry -allTopology
runApplication splitMeshRegions -cellZonesOnly -overwrite

for i in solid_aluminium solid_steel
do
    rm -f 0*/${i}/{mut,nuTilda,k,epsilon,omega,p_rgh,U}
done

for i in fluid_water
do
    rm -f 0*/${i}/{dummy}
done

for i in fluid_water solid_aluminium solid_steel; do
    if [ -f log.changeDictionary.${i} ]; then
        echo "changeDictionary already run on ${PWD}: remove log to re-run"
    else
        changeDictionary -region ${i} 2>&1 >log.changeDictionary.${i}
    fi
done

runApplication decomposePar -allRegions

COUNT=0
NPROCS=$(getNumberOfProcessors)

while [ "${COUNT}" -lt "${NPROCS}" ]; do
    if [ -f log.topoSetDict.processor${COUNT} ]; then
        echo "topoSetDict already run on ${PWD}: remove log to re-run"
    else
        topoSet -case processor${COUNT} -dict ${PWD}/system/topoSetDict \
            -region fluid_water 2>&1 >log.topoSetDict.processor${COUNT}
    fi

    let 'COUNT += 1'
done

runParallel $(getApplication) ${NPROCS}
runApplication reconstructPar -allRegions -latestTime

I'm not sure, but one possibility is that the -1 face means that this face is on a processor boundary. One way to check that would be to look at the boundary and the faceZones-file of the processor that has the 208 processes and from the start-entries in boundary see if any faces in the faceZone are on a processor boundary.

[billie]

It looks like the problem was mesh related. The problems occurred with a small test model which consists of a simple channel with the interior surface in between. Today I tested it with a full model with one inlet which splits eight times and combines again afterwards. There I would like to know the mass flux distribution. For this model the flux is reported correct from the swakExpression as well as the faceSource. The sets created for the decomposed case do not include negative face numbers here.

Another thing I noticed the OpenFoam output from Hypermesh does not reliable create the faceZones, thus I exported first to Fluent format and converted to OpenFoam format using fluent3DMeshToFoam. This way I have all required boundaries, cellZones, faceZones etc. with no need to change anything manually. Plus it creates flipMaps for the faceZones. Could this probably why the reason the output from the faceZone is correct?

[billie]

Originally Posted by gschaider

If the problem is the faceZone with "face" -1 than older versions of swak will show that behaviour too. The only way to know for sure is if you have a Debug-version of OF to run the case there. In the debug-version an access to index -1 will make the run fail and give a stack-trace

Lets see if I can compile a debug version.

Originally Posted by gschaider

Different decomposition? Just curious if this makes the -1 go away

I already tried with simple, hierarchical and scotch as decomposition method.

Originally Posted by gschaider

I'm not sure, but one possibility is that the -1 face means that this face is on a processor boundary. One way to check that would be to look at the boundary and the faceZones-file of the processor that has the 208 processes and from the start-entries in boundary see if any faces in the faceZone are on a processor boundary.

I will have a look at this. I also have to re-check if using the the intermediate fluent fomat fixes this. I am not 100% sure I did check this already.

If the Hypermesh output is buggy I will contact the Altair support about this issue.

[billie]

Originally Posted by gschaider

I'm not sure, but one possibility is that the -1 face means that this face is on a processor boundary. One way to check that would be to look at the boundary and the faceZones-file of the processor that has the 208 processes and from the start-entries in boundary see if any faces in the faceZone are on a processor boundary.

I did check this and none of the faces of the faceZone seems to be on the processor boundary. The numbers of the faceZone range with gaps from 0 to 1992 and the lowest processor boundary starts at 4630.

While checking this I recognized that sometimes the startFaces of the inter processor boundaries are not the same. I think they should be same as they share the same faces. So I have procBoundary0to1 -> startFace 15775 and procBoundary1to0 -> startFace 15571.

What is strange that I get no other errors. Anyway I think there is a problem with the Hypermesh mesh export. I will take a look at this and report back here.