[musahossein]

Dear all:
I am running solshingTank2D in interDyMFoam. The tank has 40,000 cells. I did this to see how the cpu is utilized by OpenFOAM. My computer processor is a Intel i3 CPU M330@2.13Ghzx4 with 4 CPU's. I am running UBUNTU 12.10 and OpenFoam 2.2.0. When I check the processor performance, what I see is that one CPU is running at 99% while the others are below 10%. Is there any way to make OpenFOAM seek out and use all the CPU's? A screen shot of CPU usage while running OpenFOAM for this problem is attached.

Any advice would be greatly appreciated.

[akidess]

Have you checked the user manual? http://www.openfoam.org/docs/user/ru...s-parallel.php

[musahossein]

Originally Posted by akidess

Have you checked the user manual? http://www.openfoam.org/docs/user/ru...s-parallel.php

Thankyou very much for your response. I looked into decomposPar dict file and it seems that it is set up for parallel processing over several computers -- or it does not care and looks for the number of subdomains specified in the same computer before looking elsewhere?

[erichu]

Hello,

Are you running foamJob or mpirun? if not that explains why only one core is used.

[Mojtaba.a]

Originally Posted by musahossein

Thankyou very much for your response. I looked into decomposPar dict file and it seems that it is set up for parallel processing over several computers -- or it does not care and looks for the number of subdomains specified in the same computer before looking elsewhere?

You can easily use it in the same computer with multiple processors.
Just set decomposeParDict correctly with respect to number of processors you have and you are done.
the rest is in the user's manual.
Good luck

[musahossein]

I have 2 processors. Each processor has 2 CPUs. So I should set subdomain to the number of processors (2) or the number of CPUs (4). Also, I have to run blockMesh and setFields before running mpi, correct? Thanks.

[Mojtaba.a]

Originally Posted by musahossein

I have 2 processors. Each processor has 2 CPUs. So I should set subdomain to the number of processors (2) or the number of CPUs (4). Also, I have to run blockMesh and setFields before running mpi, correct? Thanks.

If you are using Gnome run

gnome-system-monitor

go to resources tab and see how many CPUs are listed there under CPU History. thats your number for defining in decomposeParDict.

Not sure about setFields, but about blockMesh, yes you must run it before mpi.

[erichu]

I am putting a file in my case folder named 'machines'. In this file I later write my config, I.e.
Workstation cpu=2

Where workstation is the host name. Sub domains are based on cores, so in your case 4, I think.

My normal procedure is
0) having machines file setup
1) Blockmesh/ideasUnvToFoam
2) decomposePar
3) foamJob -s -p solverName

[musahossein]

Gentlement:

Here is my attempt to run OpenFOAM over 4 CPU's that my computer has. I noted that in the first error message, it shows that it cannot find processor0. But I thought OpenFOAM would automatically detect the number of CPU's. Was that assumption incorrect? Thanks for your help / advice.
__________________________________________________ _____________________
musa@ubuntu:~/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D$ mpirun -np 4 interDyMFoam -parallel >log &
[1] 4814
musa@ubuntu:~/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D$ [0]
[0]
[0] --> FOAM FATAL ERROR:
[0] interDyMFoam: cannot open case directory "/home/musa/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D/processor0"
[0]
[0]
FOAM parallel run exiting
[0]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 4815 on
node ubuntu exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

[musahossein]

Originally Posted by Mojtaba.a

If you are using Gnome run

go to resources tab and see how many CPUs are listed there under CPU History. thats your number for defining in decomposeParDict.

Not sure about setFields, but about blockMesh, yes you must run it before mpi.

I am running Ubuntu 10.2. In the dash there is a way to check the system performance, and it shows 4 CPU's. But for some reason decomposePar is still finiding errors. Any comments/ suggestions will be appreciated. Thankyou

__________________________________________________ _____________________
Decomposing mesh region0

Create mesh

Calculating distribution of cells
Selecting decompositionMethod hierarchical


--> FOAM FATAL ERROR:
Wrong number of processor divisions in geomDecomp:
Number of domains : 4
Wanted decomposition : (2 2 2)

From function geomDecomp::geomDecomp(const dictionary& decompositionDict)
in file geomDecomp/geomDecomp.C at line 50.

FOAM exiting

[erichu]

Do you have a possibility to upload the decomposeDict file? It might be easier to find the source of the problem. However (2 2 2) is to me decomposition for 8 processors and not for 4.

[Mojtaba.a]

Originally Posted by musahossein

I am running Ubuntu 10.2. In the dash there is a way to check the system performance, and it shows 4 CPU's. But for some reason decomposePar is still finiding errors. Any comments/ suggestions will be appreciated. Thankyou

__________________________________________________ _____________________
Decomposing mesh region0

Create mesh

Calculating distribution of cells
Selecting decompositionMethod hierarchical


--> FOAM FATAL ERROR:
Wrong number of processor divisions in geomDecomp:
Number of domains : 4
Wanted decomposition : (2 2 2)

From function geomDecomp::geomDecomp(const dictionary& decompositionDict)
in file geomDecomp/geomDecomp.C at line 50.

FOAM exiting

As eric said upload your decomposeParDict to see how you have configured it.
also have a loook at this:
http://www.cfd-online.com/Forums/ope...tml#post189895

[musahossein]

Thanks for your help. The decomposePar works now. However, it stalls at the end of the run with the following message:

----- I have deleted preceeeding out put to keep this to the point -----------

Number of processor faces = 812
Max number of cells = 27540 (49.998% above average 18360.2)
Max number of processor patches = 2 (0% above average 2)
Max number of faces between processors = 408 (0.492611% above average 406)

Time = 0

--> FOAM FATAL IO ERROR:
Cannot find patchField entry for lowerWall

file: /home/musa/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D/0/alpha1.org-old.boundaryField from line 25 to line 33.

From function GeometricField<Type, PatchField, GeoMesh>::GeometricBoundaryField::readField(const DimensionedField<Type, GeoMesh>&, const dictionary&)
in file /home/opencfd/OpenFOAM/OpenFOAM-2.2.0/src/OpenFOAM/lnInclude/GeometricBoundaryField.C at line 154.

FOAM exiting
-----------------------------------------------------------------------------------------------------------------------
But, my blockmeshDict file has the patches as shown below:
vertices
(
(-0.05 -0.50 -0.35) // Vertex back lower left corner = 0
(-0.05 0.50 -0.35) // Vertex back lower right corner= 1
(-0.05 0.50 0.65) // Vertex back upper right corner= 2
(-0.05 -0.50 0.65) // Vertex back upper left corner = 3

(0.05 -0.50 -0.35) // Vertex front lower left corner = 4
(0.05 0.50 -0.35) // Vertex front lower right corner= 5
(0.05 0.50 0.65) // Vertex front upper right corner= 6
(0.05 -0.50 0.65) // Vertex front upper left corner = 7

);

blocks
(
// block0
hex (0 1 2 3 4 5 6 7)
(271 271 1)
simpleGrading (1 1 1)
);

//patches
boundary
(
lowerWall
{
type patch;
faces
(
(0 1 5 4)
);
}
rightWall
{
type patch;
faces
(
(1 2 6 5)
);
}
atmosphere
{
type patch;
faces
(
(2 3 7 6)
);
}
leftWall
{
type patch;
faces
(
(0 4 7 3)
);
}
frontAndBack
{
type Empty;
faces
(
(4 5 6 7)
(0 3 2 1)
);
}
);

Any comments / suggestions would be appreciated. Thankyou.

[erichu]

I wonder if you are missing the patch name in one of the U, p .... files? Upload the boundary files as well and we can check if you cannot find the source of your problem.

[musahossein]

Originally Posted by erichu

I wonder if you are missing the patch name in one of the U, p .... files? Upload the boundary files as well and we can check if you cannot find the source of your problem.

Actually I figured out what the problem was. OpenFOAM reads all the files in the "0" folder. So I had kept the original and revised versions of alpha1, p, and U files in the folder. It was trying to read all of them, eventhough they were named alpha1-old, p-old, U-old. Once I got rid of them, decomposePar ran w/o any problems. But when I do the mpi run, I get another error:


musa@ubuntu:~/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D$ mpirun -np 4 interDyMFoam -parallel >log &
[1] 2663
musa@ubuntu:~/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D$ [0]
[0]
[0] --> FOAM FATAL IO ERROR:
[0] cannot find file
[0]
[0] file: /home/musa/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D/processor0/0/alpha1 at line 0.
[0]
[0] From function regIOobject::readStream()
[0] in file db/regIOobject/regIOobjectRead.C at line 73.
[0]
FOAM parallel run exiting
[0]
[2]
[2]
[2] --> FOAM FATAL IO ERROR:
[2] cannot find file
[2]
[2] file: /home/musa/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D/processor2/0/alpha1 at line 0.
[2]
[3]
[3]
[3] --> FOAM FATAL IO ERROR:
[3] cannot find file
[3]
[3] file: /home/musa/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D/processor3/0/alpha1 at line 0.
[3]
[3] From function regIOobject::readStream()
[3] in file db/regIOobject/regIOobjectRead.C at line 73.
[3]
FOAM parallel run exiting
[3]
[2] From function regIOobject::readStream()
[2] in file db/regIOobject/regIOobjectRead.C at line 73.
[2]
FOAM parallel run exiting
[2]
[1]
[1]
[1] --> FOAM FATAL IO ERROR:
[1] cannot find file
[1]
[1] file: /home/musa/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D/processor1/0/alpha1 at line 0.
[1]
[1] From function regIOobject::readStream()
[1] in file db/regIOobject/regIOobjectRead.C at line 73.
[1]
FOAM parallel run exiting
[1]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 3 with PID 2667 on
node ubuntu exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[ubuntu:02663] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[ubuntu:02663] Set MCA parameter "orte_base_help_aggregate" to 0 to see all helpmusa@ubuntu:~/OpenFOAM/musa-2.2.0/run/tutorials/multiphase/interDyMFoam/ras/sloshingTank2D$

[musahossein]

If there are 4 CPU's then is the MPI error messages requiring that there be a alpha1.org folder for each processor?

[JR22]

Does your computer have:
1. Two separate processors with i3 two cores each?, or
2. One dual core i3 processor with hyperthreading (aka multithreading)?

If you have case #1, you should see 8 logical CPUs, if you have #2, you should see 4 logical CPUs. Only 50% of logical CPUs have the power to crunch data at fullest. For some jobs, however, the other 50% could give you an edge. What I am saying, is that your model might finish faster with the settings set to two CPUs than four. It also might explain your original observation of two CPUs working harder than the other two.

This is a good post that touches on the subject of hyperthreading:
http://www.cfd-online.com/Forums/ope...processor.html

[erichu]

Originally Posted by musahossein

If there are 4 CPU's then is the MPI error messages requiring that there be a alpha1.org folder for each processor?

I have never used the solver you are using but I cannot imagine that you need .org files. If the setup has hanged, you might also need to run decomposePar -force to update the folders.

In general, I would say that each decomposed processor needs a full set of boundary files.

I ran the tutorial of interDyMFoam sloshingTank2D (ras) using ./Allrun, then aborted when the solver started. Later decomposePar and finally
foamJob -s -p interDyMFaom

[hakonbar]

Have you done the damBreak tutorial described in the user guide? It's a pretty good step-by-step walkthrough of setting up a parallel run. It's even multiphase, so it uses the variable alpha.

By the way, I think you forgot to rename the file called "alpha1.org" to "alpha1" before decomposing. The ".org"-ending doesn't do anything, it's just to keep a backup of the alpha1 file that's not modified by the setFields application. This is also described in the damBreak tutorial.

[musahossein]

To all the forum members who put their time and effort in trying to help me out and responding to my posts -- a heartfelt thankyou. Paralle processing finally worked. The reasons it was not working are as follows:

1.decomposePar reads all the files in the "0" folder. So it not only read the alpha1, p and U folders that I had modified, but also read the original folders which I had saved with prefix such as "old" or "orignal". So the error was coming from decomposePar reading the original files after reading the files I had modified.

2. The ./Allclean kept deleting the alpha1, apha1.org files with the "rm" command. So I commented out that line.

3. I didnt realize that in the decomposePar dictionary, you should keep only the method you want to use and comment out or delete the other options.

After I took care of these items, the parallel processing works very well and the system monitor shows all the 4 CPU's running at 99%-100%.