[alberto]

Originally Posted by sandy

Hi Alberto, if I give the pdRefcell and pdRefvalue, is it still an ill posed problem?? I feel grey. Now I am actually in the Nuemann BC's CFD, namely zero Gradient in the academic informations. Thank you very much.

That's what you do for example in the cavity tutorial in OpenFOAM. Setting a pressure reference allows you to find a unique solution for the pressure.

Best,

[sandy]

Originally Posted by lakeat

Hi, Sandy, There are advisors and advisors ...

YES, lakeat, Alberto is I said he said ...

[alberto]

Originally Posted by sandy

YES, lakeat, Alberto is I said he said ...

I'm not an advisor, which is not easy if you want to do it right.
I was very very lucky during my studies and now in my post-doc to have good advisors though!

[sandy]

Originally Posted by lakeat

I am using pisoFoam simulating a cylinder flow...
...
The cylinder case I am solving is about 860000 grids, I found 32 processors or even 64 processors ...

Hi lakeat, why you need so many grids to simulate a cylinder flow. I ever read a paper to just use 50000 grids to simulate a hydraufoil with the Kunz's cavitation model. How could they get it? You think.

[alberto]

Hi Sandy,

I'm not sure how Lakeat is trying to simulate the flow, but if the Reynolds number is high enough (and it does not need to be very high), and he is doing LES, that number of cells does not seem huge to me. Keep in mind that in LES you must resolve scales until well inside the inertial subrange of the spectrum.

Best,

[sandy]

WOW~, in 2D, it still needs so huge grids? If that, I should choose a good rest but work so hard.

[lakeat]

Originally Posted by alberto

Hi Sandy,

I'm not sure how Lakeat is trying to simulate the flow, but if the Reynolds number is high enough (and it does not need to be very high), and he is doing LES, that number of cells does not seem huge to me. Keep in mind that in LES you must resolve scales until well inside the inertial subrange of the spectrum.

Best,

Yes, I am performing a LES simulation, this is what my thesis focuses.
I have never do an extensive survey, but I am not sure how good it will be to use RANS for a cylinder flow??????????

Will RANS give the Cd and Cl correct enough????

But Apparently, I will not expect good results from 2D simulation. The energy cascade is completely wrong in 2D simulation.

The Re (3900) is not high though, but already costs me a lot. This is LES.

I am expecting a much larger mesh for my next case Re=140000, the grid number would be 7 million.

You know what, someone told me that the "Bird Nest" in Beijing, certain man use just 80000 grids to do the simulation, and I was shocked, "how could he manage to do that?" "How could he get the content of TURBULENCE?"

[alberto]

Originally Posted by lakeat

But Apparently, I will not expect good results from 2D simulation. The energy cascade is completely wrong in 2D simulation.

In 2D you should not even do LES, since eddies are definetly 3D structures.

The Re (3900) is not high though, but already costs me a lot. This is LES.

Re = 3900 (referred to the mean velocity and the diameter I assume) is not high at all, and it should be possible to do that in 3D with a very high resolution. Is your domain a simple cylinder (pipe)? Can you describe it with periodic conditions?

You know what, someone told me that the "Bird Nest" in Beijing, certain man use just 80000 grids to do the simulation, and I was shocked, "how could he manage to do that?" "How could he get the content of TURBULENCE?"

Modelling it in some way, probably, as done in many cases for practical applications. Does he claim he is doing LES? You could check what kind of resolution he gets with the simple estimation formulas provided in turbulence books (Pope, for example).

Best,

[lakeat]

First, I want to say thank you, Alberto, and also thank you Sandy.

Okay, let me make myself clearer,

the cylinder case mesh is an O-type mesh, with Nx*Ny*Nz=165*165*32, Re=3900, with benchmark of LES simulation already published that can be easily followed.

To achieve Re=3900, I simply set inlet velocity 1m/s, and cylinder diameter 1m. Flow region is a circle with diameter of 15m, which means standing at the cylinder center, the distance to the inlet and out let are both 7.5m.

The timeStep is set to 0.0025 to keep Courant number no larger than unity.


Of course, this is full LES, a wall resolved LES, completely 3D simulation for a 2D geometry, as we can imagine from a wind tunnel section model. Periodic B.C. are applied in the front and back, I do not use convectiveOutlet for the moment.

Then I need to consider:
1. To achieve the highest efficiency, how many CPUs do I need for this case?
2. To achieve the highest efficiency and accuracy, which solver should I use for p and for U?

The 1st one, your experience is 8~16 processors are enough, right?
The 2nd one, you recommend GAMG for p and smoothsolver for U, right?

Here are some findings, some of which make me headache now,

No. 1
My teacher strongly doubt my simulation result, he said the simulation time is too long, and is unacceptable.

Time = 397.925

Courant Number mean: 0.0201989 max: 0.821767
smoothSolver:  Solving for Ux, Initial residual = 0.000263646, Final residual = 9.52965e-08, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.00125789, Final residual = 4.36729e-07, No Iterations 2
smoothSolver:  Solving for Uz, Initial residual = 0.00347622, Final residual = 1.45703e-06, No Iterations 2
GAMG:  Solving for p, Initial residual = 0.0120711, Final residual = 0.000402602, No Iterations 2
time step continuity errors : sum local = 2.96528e-09, global = 7.05016e-12, cumulative = -3.29373e-09
GAMG:  Solving for p, Initial residual = 0.000504671, Final residual = 4.92814e-05, No Iterations 3
time step continuity errors : sum local = 3.62961e-10, global = 3.00537e-12, cumulative = -3.29072e-09
ExecutionTime = 164607 s  ClockTime = 167103 s

Time = 397.927

Courant Number mean: 0.0202005 max: 0.821096
smoothSolver:  Solving for Ux, Initial residual = 0.000263663, Final residual = 9.53374e-08, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.00125653, Final residual = 4.36351e-07, No Iterations 2
smoothSolver:  Solving for Uz, Initial residual = 0.00347678, Final residual = 1.45956e-06, No Iterations 2
GAMG:  Solving for p, Initial residual = 0.0120541, Final residual = 0.000401538, No Iterations 2
time step continuity errors : sum local = 2.95737e-09, global = 6.11715e-12, cumulative = -3.2846e-09
GAMG:  Solving for p, Initial residual = 0.000502906, Final residual = 4.82624e-05, No Iterations 3
time step continuity errors : sum local = 3.5545e-10, global = 2.07349e-12, cumulative = -3.28253e-09
ExecutionTime = 164610 s  ClockTime = 167106 s

So what do you think, is there something wrong? Why my simulation is that long?

No. 2
Based on your experience, can you give a judge, are these figures correct? (I use pyFoam to get these figures)
I mean time spent.
Graph3-m.jpgGraph4-m.jpg
Note also the question in the picture. Will you meet the problem of unstable simulation time (ie. ExecutionTime) since the flow is unsteady?


Waiting for you,

[alberto]

Originally Posted by lakeat

First, I want to say thank you, Alberto, and also thank you Sandy.

Okay, let me make myself clearer,

the cylinder case mesh is an O-type mesh, with Nx*Ny*Nz=165*165*32, Re=3900, with benchmark of LES simulation already published that can be easily followed.

To achieve Re=3900, I simply set inlet velocity 1m/s, and cylinder diameter 1m. Flow region is a circle with diameter of 15m, which means standing at the cylinder center, the distance to the inlet and out let are both 7.5m.

If you want a constant mean velocity of 1 m/s in your pipe, this is not correct, since the dissipation will slow your flow down. Furthermore, how can you fix the velocity at the inlet if the domain is periodic? You should use something like channelFoam (OpenFOAM 1.6) or channelOodles (OpenFOAM 1.5), where the pressure gradient is adapted per each time step so that the mean flow rate is kept constant.

In addition, what is your initial condition? If you use a uniform, not perturbed, condition it is going to take a long time to actually develop turbulence structures. Eugene de Villiers published a code to initialize a perturbed flow in a cylinder: search the forum for it.

Moreover, the lenght of the system has to be enough not to feel the effect of periodic conditions (I would say this was checked in the original work).

Of course, this is full LES, a wall resolved LES, completely 3D simulation for a 2D geometry, as we can imagine from a wind tunnel section model. Periodic B.C. are applied in the front and back, I do not use convectiveOutlet for the moment.

I don't get it again. A cylinder has one wall BC and two sections as boundary conditions. If you specify periodic conditions in the flow direction, you cannot have other form of inlets/outlets.

The 1st one, your experience is 8~16 processors are enough, right?

Well, with ~800,000 cells, and a single-phase flow, yes, definetly.

My teacher strongly doubt my simulation result, he said the simulation time is too long, and is unacceptable.

Usually, if you start from a good initial condition, with fully developed turbulence, you need at least 20 characteristic times to extract statistics.
In other words, you initialize a perturbed flow as said above, you run until the flow is completely developed (and this takes time, a lot!), and then you reset the averages and start averaging for a sufficient number of characteristic times to get your statistics.

What does your teacher takes as reference to say the computational time is unacceptable?

P.S. What are you using as pressure residual?
Best,
A.

[sandy]

I never think that GAMG should be chosen in parallel because it is so complex. U equation does not spend your CPU time too much, whatever which solver is chose. The key is P equation forever.

[alberto]

Originally Posted by sandy

I never think that GAMG should be chosen in parallel because it is so complex.

Are you stating GAMG solvers are suitable only for serial calculations? I would not think so.

[Simon Lapointe]

Hi guys,

I feel there is some miscommunication between you. From what I understood of Lakeat's problem (correct me if I'm wrong), he is simulating the external flow around a cylinder, not an internal pipe flow. When he says he uses periodic conditions at front and back I think he means at both ends of the cylinder.

Concerning Lakeat's concerns,

1) Many factors can affect your parallel performance.
First, it depends on the computer you're using. A poor connection between the computing nodes can really slow down the calculation, causing a big difference in ExcutionTime and ClockTime.
Second, if the connection is good, I've found that the OpenFOAM compilation can greatly influence the computational time. Are you using a pre-compiled OpenFOAM ? If you do, I strongly suggest you compile OpenFOAM locally on the computer you're using for your parallel computations, preferably with the local compiler (such as an Intel compiler).
Concerning the decomposition, a "simple" decomposition can be used on simple geometries. If you aren't so sure how to decompose it, you should use MeTis.
From my personal experience of unsteady simulations of external flows, I've seen good speedup (linear or better) with as less as 10-15k cells per processor. But that depends on the computer and the case.

2) I've used the GAMG solver for the pressure equation in many parallel cases and I think it performs well, generally faster than PCG. I don't agree with sandy. How the does fact that it is more complex make it slower ? Did you compare it with other solvers in parallel cases ?

Hope this helps

[alberto]

Originally Posted by Simon Lapointe

Hi guys,

I feel there is some miscommunication between you. From what I understood of Lakeat's problem (correct me if I'm wrong), he is simulating the external flow around a cylinder, not an internal pipe flow. When he says he uses periodic conditions at front and back I think he means at both ends of the cylinder.

Oops. My fault. I should have thought to that when he described his system. Sorry about the confusion, please ignore my comment about the pipe flow simulation, the initialization of a perturbed flow and such.

About the domain decomposition, in the case of a flow around a cylinder, I agree with Simon when he suggests to use METIS, since it becomes more complicated to do by hand. I'm a bit skeptical about 10.000 cells/processor. In my experience a good trade-off is something between 40.000 and 80.000 cells per processor (meaning core), especially if you are not in a hurry to get the results and you share resources with others ;-)

2) I've used the GAMG solver for the pressure equation in many parallel cases and I think it performs well, generally faster than PCG. I don't agree with sandy. How the does fact that it is more complex make it slower ? Did you compare it with other solvers in parallel cases ?

I agree on GAMG. Also in my experience it is generally faster than PCG. The only side note I have is that sometime (I'm thinking to some multi-phase case), PCG are stabler, while with GAMG I had crashes in some runs. However this should not be the case for a simple single phase flow.

Best,

[lakeat]

Originally Posted by Simon Lapointe

Hi guys,

I feel there is some miscommunication between you. From what I understood of Lakeat's problem (correct me if I'm wrong), he is simulating the external flow around a cylinder, not an internal pipe flow. When he says he uses periodic conditions at front and back I think he means at both ends of the cylinder.

Yes, I'm very sorry for the miscommunication, my case is not a internal flow but an external flow past a circular cylinder. And thank you Simon for your patience to read so long a post.

Concerning Lakeat's concerns,

1) Many factors can affect your parallel performance.
First, it depends on the computer you're using. A poor connection between the computing nodes can really slow down the calculation, causing a big difference in ExcutionTime and ClockTime.

"big difference", no, I did not see big difference between the two.
You can see from my last post:

Time = 397.927
..................................
ExecutionTime = 164610 s ClockTime = 167106 s

ClockTime-ExecutionTime=167106-164610=2496, and 2496/164610~1.516%.

And 1.516% is not that big, right?

A poor connection between the computing nodes

Yes, but this is hardly the case of me, the super-computer I am using have 32 processors per node, even between different nodes, they are connected by what so called infiniband. So I guess this won't be a problem for me, right. But the question is raised here:

Will different mpi implementations like openmpi, mpich, mvapich, make great differences on the speed? Say, if I am decomposing the case to several nodes, and if I am using just one node.

And I remember wiki writes:

In computing, wall clock time is the actual time taken by a computer to complete a task. It is the sum of three terms: CPU time, I/O time, and the communication channel delay (e.g. if data are scattered on multiple machines). In contrast to CPU time, which measures only the time during which the processor is actively working on a certain task, wall time measures the total time for the process to complete. The difference between the two consists of time that passes due to programmed delays or waiting for resources to become available.

Second, if the connection is good, I've found that the OpenFOAM compilation can greatly influence the computational time. Are you using a pre-compiled OpenFOAM ? If you do, I strongly suggest you compile OpenFOAM locally on the computer you're using for your parallel computations, preferably with the local compiler (such as an Intel compiler).

Really? You know, in fact, I have compared the two, one is pre-compiled (I guess it was compiled on opensuse), the other is compiled one (The super-computer is installed with SUSE), and currently, I did not see compiled one superior to the pre-compiled one. Because I have also recompiled the openmpi, so I am not sure now, why that I did not see the "superior" to the pre-compiled OpenFOAM? Is it because my recompiled openmpi did not work very good...... I have no idea now.

Concerning the decomposition, a "simple" decomposition can be used on simple geometries. If you aren't so sure how to decompose it, you should use MeTis.

I hope someone will clear this up, you can see from the official website, new features of version-1.6, it seems scotch is obviously superior to metis, please correct me if I was wrong. Second, for a O-type grid, unless, you decompose it into 4 regions (2*2*1), I still cannot see why simple will work better than metis and scotch method. I one decomposed them into 8 and 16 and 32, and here are the output log files.

simple one:

Number of processor faces = 102046
Max number of processor patches = 6
Max number of faces between processors = 10888

metis one:

Number of processor faces = 68674
Max number of processor patches = 7
Max number of faces between processors = 5904

You see there is a lot of differences.

From my personal experience of unsteady simulations of external flows, I've seen good speedup (linear or better) with as less as 10-15k cells per processor. But that depends on the computer and the case.

Okay, this experience is very helpful, you know.
And for a first try, my cylinder case mesh, 860000/15000~57 processors??!!!! Is this what you meant??

I've used the GAMG solver for the pressure equation in many parallel cases and I think it performs well, generally faster than PCG. I don't agree with sandy. How the does fact that it is more complex make it slower ? Did you compare it with other solvers in parallel cases?

Hope this helps

Yes, your information is very very helpful.

PS: pls call me Daniel...

Thank you all.

[alberto]

Hi,

if you do not use a hardware-optimized implementation of the parallel libraries (HP, Intel, ...), you won't notice much difference switching between OpenMPI (which comes from LAM) and MPICH generic libraries. Performance mainly depends on how you compile them and your code.

Originally Posted by lakeat

I hope someone will clear this up, you can see from the official website, new features of version-1.6, it seems scotch is obviously superior to metis, please correct me if I was wrong. Second, for a O-type grid, unless, you decompose it into 4 regions (2*2*1), I still cannot see why simple will work better than metis and scotch method. I one decomposed them into 8 and 16 and 32, and here are the output log files.

A "simple" decomposition works OK if the geometry is not complicated and the mesh is quite uniform. In this way you can decompose the domain by hand in quite a precise way and control the decomposition easily. If the domain is more complicated and the mesh not simple either, an automatic decomposition saves you a lot of time and effort, and gives better results.

Okay, this experience is very helpful, you know.
And for a first try, my cylinder case mesh, 860000/15000~57 processors??!!!! Is this what you meant??

Hehe, your sysadmin and colleagues will be happy when you'll run bigger cases.

Best,

[Simon Lapointe]

Originally Posted by lakeat

Yes, but this is hardly the case of me, the super-computer I am using have 32 processors per node, even between different nodes, they are connected by what so called infiniband. So I guess this won't be a problem for me, right. But the question is raised here:

Will different mpi implementations like openmpi, mpich, mvapich, make great differences on the speed? Say, if I am decomposing the case to several nodes, and if I am using just one node.

I did not mean to say a poor interconnection was your problem, but I mentioned it because it is a common cause of slow parallel computations. Since it seems your system has a good interconnect this is probably not the case here.

Concerning the mpi implementations I'm not so sure, since with OpenFOAM I've only used OpenMPI. The best way to answer your question about the number of nodes is to try different combinations of nodes and processors per nodes. The best ratio will depend on the connection speed between the nodes, the connection speed between the processors and the usage of the processors on the different nodes you're using.

Originally Posted by lakeat

Really? You know, in fact, I have compared the two, one is pre-compiled (I guess it was compiled on opensuse), the other is compiled one (The super-computer is installed with SUSE), and currently, I did not see compiled one superior to the pre-compiled one. Because I have also recompiled the openmpi, so I am not sure now, why that I did not see the "superior" to the pre-compiled OpenFOAM? Is it because my recompiled openmpi did not work very good...... I have no idea now.

In my case, it made a huge difference (note I'm using 1.5-dev). With the pre-compiled version, there was a small speedup at 4 or 8 processors, but far from a linear speed. When using more than 8 processors, it was slower and the clocktime was much larger than the cpu time. After being compiled on the supercomputer with the Intel compiler, the speedup is much better.

Which compiler did you use ? Are there different compilers available ? You could try using a different compiler.

Originally Posted by lakeat

I hope someone will clear this up, you can see from the official website, new features of version-1.6, it seems scotch is obviously superior to metis, please correct me if I was wrong. Second, for a O-type grid, unless, you decompose it into 4 regions (2*2*1), I still cannot see why simple will work better than metis and scotch method.

I've never used Scotch so I can't comment on its performances. I didn't say simple would be better than metis or scotch, just that it works well on simple meshes. I suggest you try both MeTis and Scotch and compare your results. Others would be interested in such a comparison. Where did you see that Scotch was "obviously superior to metis" ?

Originally Posted by lakeat

Okay, this experience is very helpful, you know.
And for a first try, my cylinder case mesh, 860000/15000~57 processors??!!!! Is this what you meant??

Yes, this is what I meant. I've split meshes of 750k to 1M cells in 48 or 64 processors and the speedup was very good. But, this is very dependent of the computer used and the case, so I can't promise you anything...Also, as Alberto said, unless the usage of your system is low, it is much more reasonable to use a smaller number of processors (maybe 50k cells/processors). This way you'll still get decent speed and you could use a higher number of processors for cases that really need them.

[sandy]

Originally Posted by alberto

Are you stating GAMG solvers are suitable only for serial calculations? I would not think so.

Hi Alberto, I appreciate so much for your patiently and repeatly directing.

In addtion, maybe I will try GAMG for p equation. In fact, GAMG is solved by ICCG or Block-ICCG (Why not is BiCG?!!! In coarse grid, all matrices are symmetric??!!! Who can explain it??) in coarse grids, and maybe solved by Guass Seidel in the fine grid? So, it should not be difficult to uderstand this method maybe is faster than PCG.

WOW~, really?? GAMG just can be used in solving symmetrical matrix namely p equation?? I just find it!

If that, what kinds of Mutil-Grids methods can be used to solve the asymmetrical matrix in OpenFOAM?

@Sorry Daniel, I actually know nothing to parallel.

[braennstroem]

Hi Daniel,

another way to accelerate flow development is, to map fluctuations of a 'boxTurb' generated velocity field onto your initial RANS velocities.

Fabian

[alberto]

Originally Posted by sandy

Hi Alberto, I appreciate so much for your patiently and repeatly directing.

WOW~, really?? GAMG just can be used in solving symmetrical matrix namely p equation?? I just find it!

There is not such a restriction. Take a look at this document, where in one example (page 15), GAMG is used also for U: http://www.tfd.chalmers.se/~hani/kur...report-fin.pdf

Best,