[internetscooter]

Hi,

I am what can best be described as an OpenFOAM hobbyist with access to a HPC facility.

I am tweaking the following to run my own Vespa model /usr/local/openfoam/2.2.2/tutorials/incompressible/pisoFoam/les/motorBike/

I am playing with scaling and am noticing that the Cd value changes based on number of cores - is this expected? I would have thought it would be somewhat deterministic - is it or should I be looking for an error in the way I am approaching?

Here are some results - the last Cd value in log.simpleFoam followed by the last in log.pisoFoam, both differ from run to run.

 grep Cd 20140212_*/*.o*
20140212_0843/nodes=1:ppn=12.o1132447:    Cd    = 0.650766
20140212_0843/nodes=1:ppn=12.o1132447:    Cd    = 0.628932

20140212_084738/nodes=1:ppn=10.o1132450:    Cd    = 0.650916
20140212_084738/nodes=1:ppn=10.o1132450:    Cd    = 0.628049

20140212_090106/nodes=2:ppn=10.o1132451:    Cd    = 0.651073
20140212_090106/nodes=2:ppn=10.o1132451:    Cd    = 0.628595

20140212_090150/nodes=3:ppn=10.o1132452:    Cd    = 0.650694
20140212_090150/nodes=3:ppn=10.o1132452:    Cd    = 0.626113

20140212_090216/nodes=4:ppn=10.o1132453:    Cd    = 0.650944
20140212_090216/nodes=4:ppn=10.o1132453:    Cd    = 0.638436

20140212_122959/nodes=5:ppn=10.o1132454:    Cd    = 0.650798
20140212_122959/nodes=5:ppn=10.o1132454:    Cd    = 0.640691

20140212_150541/nodes=6:ppn=10.o1132455:    Cd    = 0.650733
20140212_150541/nodes=6:ppn=10.o1132455:    Cd    = 0.631151

20140212_151006/nodes=7:ppn=10.o1132456:    Cd    = 0.651895
20140212_151006/nodes=7:ppn=10.o1132456:    Cd    = 0.625597

As it might help someone - here's how I do it...

In my PBS script...

echo Running with $PBS_NP processors across $PBS_NUM_NODES nodes. $PBS_NUM_PPN procs per node
rm ./vespa_env
echo "vespa_env_procs             $PBS_NP;" >> ./vespa_env
echo "vespa_env_num_nodes         $PBS_NUM_NODES;" >> ./vespa_env
echo "vespa_env_num_ppn           $PBS_NUM_PPN;" >> ./vespa_env
cat ./vespa_env

In bash scripts I use the PBS variables directly. in dictionaries I...

e.g. motorBike/system/decomposeParDict.hierarchical

#include "../../vespa_env"

numberOfSubdomains $vespa_env_procs;
method          hierarchical; 
// paulmc - we will split the problem across cores in X and nodes in y
hierarchicalCoeffs
{
    n               ( $vespa_env_num_ppn $vespa_env_num_nodes 1 );
    delta           0.001;
    order           xyz;
}

Cheers,

Paul