[rribeiro]

Hi everyone,

I'm studying some new HPC/parallel tools and using them on OpenFOAM but they require the combination of two OF features that i cant put them to work together -- mesh redistribution and mesh refinement.

When combining cell and fields redistribution and refinement, the execution stalls in a while(true) loop which seems like a face consistency-check plus action routine (2:1 conflicts). According to the debug information, the value of nChanged returned from faceConsistentRefinement is always greater than zero:

	while (true)
       {
		label nChanged = faceConsistentRefinement(maxSet, refineCell);
         
        reduce(nChanged, sumOp<label>());
           
        if (debug)
        	 {
    	      Pout<< "hexRef8::consistentRefinement : Changed " << nChanged
	          << " refinement levels due to 2:1 conflicts."
           	  << endl;
  	         }
         
        if (nChanged == 0)
         {
        	  break;
           }
       }

This loop is part of the file src/dynamicMesh/dynamicMesh/directTopoChange/directTopoChange/directActions/hexRef8.C.

The tools that i am using for each task are: dynamicRefineFvMesh for refinement, hexRef8 as mesh cutter, parMetis to develop a new distribution and the fvMeshDistribute to perform the actual distribution and field remapping.

The execution order from the start is Refine -> Re-distribute -> Work -> Stall in refinement.

The faceConsistentRefinement routine is the following:

// Updates refineCell (cells marked for refinement) so across all faces there will be 2:1 consistency after refinement.
Foam::label Foam::hexRef8::faceConsistentRefinement(const bool maxSet,PackedList<1>& refineCell) const
{
    label nChanged = 0;
    
    // Internal faces.
    for (label faceI = 0; faceI < mesh_.nInternalFaces(); faceI++)
    {
        label own = mesh_.faceOwner()[faceI];
        label ownLevel = cellLevel_[own] + refineCell.get(own);
        
        label nei = mesh_.faceNeighbour()[faceI];
        label neiLevel = cellLevel_[nei] + refineCell.get(nei);
        
        if (ownLevel > (neiLevel+1))
        {
            if (maxSet){ refineCell.set(nei, 1); }
            else
            {
                refineCell.set(own, 0);
            }
            nChanged++;
        }
        else if (neiLevel > (ownLevel+1))
        {
            if (maxSet) { refineCell.set(own, 1); }
            else
            {
                refineCell.set(nei, 0);
            }
            nChanged++;
        }
    }
	//Coupled faces. Swap owner level to get neighbouring cell level. (only boundary faces of neiLevel used)
    labelList neiLevel(mesh_.nFaces()-mesh_.nInternalFaces());
    
    forAll(neiLevel, i)
    {
        label own = mesh_.faceOwner()[i+mesh_.nInternalFaces()];
        
        neiLevel[i] = cellLevel_[own] + refineCell.get(own);
    }
    
    // Swap to neighbour
    syncTools::swapBoundaryFaceList(mesh_, neiLevel, false);
    
    // Now we have neighbour value see which cells need refinement
    forAll(neiLevel, i)
    {
        label own = mesh_.faceOwner()[i+mesh_.nInternalFaces()];
        label ownLevel = cellLevel_[own] + refineCell.get(own);
        
        if (ownLevel > (neiLevel[i]+1))
        {
            if (!maxSet)
            {
                refineCell.set(own, 0);
                nChanged++;
            }
        }
        else if (neiLevel[i] > (ownLevel+1))
        {
            if (maxSet)
            {
                refineCell.set(own, 1);
                nChanged++;
            }
        }
    }
    return nChanged;
}

The routine call trace is mesh.update() -> selectRefineCells -> meshCutter.consistentRefinement.

If we skip some re-distribution steps we may stumble into a what I think to be a related error, but this time in a unrefinement routine:

    [0] 
    [0] 
    [0] --> FOAM FATAL ERROR: 
    [0] problem
    [0] 
    [0]     From function hexRef8::consistentUnrefinementUnrefinement(..)
    [0]     in file directTopoChange/directTopoChange/directActions/hexRef8.C at line 5026.
    [0] 
    FOAM parallel run aborting
    [0]

To reproduce this, I have developed a simple GIT patch (attached) that introduces the required changes to combine both features.
We use the interDyMFoam solver (to which the patch will add code) and the damBreakWithObstacle tutorial.

The patch is applied to the master branch of the extend-3.1 repository

The changes required to the tutorial are:

• File decomposeParDict: metis $->$ parMetis
• Reproducible with only 2 procs, File decomposeParDict: numberOfSubdomains 2
• File controlDict: adjustTimeStep no
• File Allrun: runApplication decomposePar
• File Allrun: runParallel interDyMFoam 2

Any help would be appreciated,
Thanks