I have a Polywell workstation with a dual 500MZ Pentium III motherboard but only one cpu was installed. The system has 512M of RAM and a 22GB IBM EIDE 7200 hard drive.

I am running a CFD code (NPARC3D)with a grid slightly larger than 2.5M on Windows NT; because of the size of the grid and the complexity of the geometry the grid is divided into 4 blocks; only one block can be in RAM and processed before it is output unto disk and then another block is read into RAM; I would guess that this results in a lot of io between RAM and disk. It takes about one hour for 4 iterations; and it takes about 20000 iterations, at least, to get a converged solution. At this rate, it will take 8 months to get a solution!

How can I speed up this system within the next few days as we need to get a set of solution before Feb. 2000?

1)One solution is to get more RAM because if I can get the whole grid within RAM, I can get a solution about 3x faster. I would need to get 256Mb or 512Mb RAM cards as the 4 slots are now filled with 128MB cards. How much are 256 or 512MB RAM cards?

2)Another solution may to get a faster hard drive: how much faster is a SCSI drive? How much more expensive it?

3)I can reduce the grid to get the minimum necessary or reduce the grid volume of calculation by boundary condition modification. Grid reduction will reduce RAM required.

Are there any other solutions? (besides buying a much more expensive system and besides getting other CFD sofware ... this will be our long term solution as we are considering a CFD code (WIND) that will use multiprocessors)

Here are a couple of ideas to improve your turn-around time with NPARC:

(1) If you are trying to obtain a steady-state solution, you may want to adjust the MBORD, IBORD values which control the order in which the blocks are solved each iteration. The default setting solves each block one time per iteration in ascending block order. For your case this would be MBORD=4, IBORD=1,2,3,4.

You could increase the number of times each block is solved before reading the next block. However, you will be using block interface (boundary) data that is "stale" - so don't get too carried away or the code will diverge. You can think of this as subiteration. MBORD=12, IBORD=1,1,1,2,2,2,3,3,3,4,4,4. Note that NMAX will now specify the number of times through IBORD, which no longer equals the number of iterations.

You don't have to list each block the same number of times in IBORD either. If one converges faster than the rest, you may only need to run it once during the IBORD cycle to update the boundaries.

(2) Again for steady-state, use DTBLK to set the time step for each block rather than use DTCAP and limit all of the blocks to the smallest timestep.

(3) Also for steady-state. Depending on your initial solution file and problem at hand, you may want to run inviscid for a short while to help initialize the flow. Then run laminar for a while, then turbulent. This may save you some cpu time from having to solve the turbulence equations during the initial transients. No need to fully converge each step, as you are only after a better "initial guess".

(4) During your first run, the code will spend a significant amount of time creating block interface interpolation files FT37, FT38. On subsequent runs, set LREST=1 to read these existing files rather than regenerate them. (If you modify your grid, you must remember to regenerate these files on the first run.)

Seriously consider upgrading to the WIND code. I can understand the need to compare with previous NPARC results and the time needed to become familiar with a new code, but WIND has some useful pre- and post-processing tools, grid-sequencing capability, easy-to-use parallel-processing features, and (IMHO) a faster turn-around time. You will need to e-mail NPARC Support (nparc-support@info.arnold.af.mil) or visit their homepage (http://www.arnold.af.mil/nparc/) regarding availability of the PC(WIN) version.

Just buy some computing time on a fast computer.

Dennis,

Thanks for taking the time to write your suggestions. I'll make a copy for future guidance.

1)Using IBORD and MBORD to keep a block within RAM for several iteration is a very good idea; I will do this when I am forced to use the hard disk to store intermediate calculations. But, when using PC's for CFD, keeping ALL blocks in memory is much preferred since disk speed is too slow to be useful. For example in my case, when all the blocks are in RAM (with MDISK=1) the number of iterations per minute is 6 times greater than with MDISK=0.

2)I use DTBLK usually for the reasons that you state.

3)I've tried an inviscid solution as a initial solution to a viscous, turbulent solution but NPARC3D seems to abort when this transition is tried. I suspect that the gradient at the wall is to great for NPARC3D. Maybe I should go from inviscid, to laminar, to turbulent as you suggested. Have you tried this?

4) I use LREST=1 once FT37 and FT38 are generated.

We do plan to get WIND, since we will be doing work where variable gamma is required.

Do you know if there is a Windows version of NPARC3D that can be run in parallel?

Have you simplified your problem enough? Can you make use of symmetry or periodicity? It is some times very difficult to give good suggestions without getting some details about the problem, but on the other hand most people do not want to disclose certain aspects of their problems because of company rules. Good luck.

Amadou, I have used the symmetry condition and also will be using boundary conditions to exclude from calculations parts of the flowfield that will not influence the region of interest

Steve,

You mention that the code aborts. If by this you mean it stops with a negative density or pressure, then it may be that the gradient at the wall is preventing you from using your inviscid solution as a starting point for your turbulent flow calculations. Easing into the solution via a laminar run may help. If the gradients are severe, you may need to reduce the time step during the early phases of the turbulent run. You might be able to gradually increase them as the solution progresses.

Also keep in mind that the two-equation turbulence models in NPARC are not self-starting. They require that you first run another model (say Spalart or Baldwin-Lomax), so that you have a turbulent viscosity profile it can use to back out non-uniform k and epsilon (or omega) values. Examine the turbulent viscosity (mu_t/mu) contained in the fort.31 output file. The maximum value in the flowfield should exceed 500 before you try starting the two-equation models. If there are no k and epsilon values in your initial restart file and the code expects them, it should stop. I don't know how meaningful an error message you get, but the code likely won't get past the first iteration.

NPARC3.0 and later had parallel processing capability, but I've never used it. Since it is distributed as raw source, you would need Fortran and C compilers on your PC. The makefile scripts were developed for Unix plaforms, so they may need some editing. The manual mentions PVM, perhaps MPI will work too, and uses a master-worker scheme. The manual is on-line at http://www.arnold.af.mil/nparc/ under Original_NPARC_Code then Documentation.

Dennis Yes, I think that the error message was about negative pressure/density. Next time I will use the laminar flow and small time step suggestions. I'm using the Spalart-Allmaras (sp?) model where I used about 1000 iterations with the Baldwin-Lomax model before starting Spalart-Allmaras (sp?).

I'll use your suggestions if I use a 2-equations model.

I'm not familiar with multi-processor procedures, software or terminology. Is MPI available for Windows NT? I have just installed a 2nd CPU on our PC (the ICEM-CFD grid mesher will automatically use multiprocessors);and, I think that it's worthwhile to pursue multi-processor NPARC computations on Windows NT and I would appreciate any help to implement it.

We are a very small company and probably can't afford the time needed.

Steve,

The only thing I know about PVM and MPI are their names.

If you only have two processors, you may not even see an improvement. The NPARC3.0 documentation page I cited earlier says that it uses a master-worker approach. With two processors, only one of them is a worker - so it acts the same as a serial run (probably slower, though, due to the added overhead). Perhaps there is a way to assign three tasks (master-worker-worker), but this is beyond my limited knowledge of the subject.

Here is the answer to a couple of questions and some things to think about. MPI is available for Windows. WMPI (non-commercial) and MPIPro (commercial) are two choices that come to mind. I am not sure about Windows based PVM. With MPI you can start a master-worker-worker problem on a single machine and this will utilize both processors (85-100%) the issue will be memory. Each process (worker) is going to need enough memory to process its blocks and communications. A problem with 3 processes (master-worker-worker) is larger than just problem size/3. If you can get your code to run on dual processors with MPI, then extending it to run on multiple machines is pretty easy.

Ken,

Thanks for the info on WMPI.

Do you have a source for the non-commercial WMPI?

I can get a version of the code to run in parallel mode. How do I install WMPI to do use this capabalilty?

Try here ... http://dsg.dei.uc.pt/wmpi All my work has been with the previous version, but I plan on upgrading to v1.3 in the near future. The documentation should tell you how to hitch it to your code. Hope this helps.

(1). I got the feeling that persons invented NPARC3D should be on the X-spacecraft or Mars probe. I mean they are living in the future. (2). Well, you can't blame them, because they have been inventing and super-computers for a long time, at NASA/Ames, NASA/Lewis, and Air Force/AEDC. (3). (There is a possibility to use a PC, but you need to know CFD and how to write your own code). (4). After having said that, it is not hopeless in your case at all. (5). You said the mesh size is 2.5M. I guess, it is 2,500,000 grid points (or cells). By any standard, it is a big number. So, I must say that there is something wrong in your project plan. Your are ahead of the places where the code was invented. And you could be out of your mind. (the person who made the suggestion to buy computer time on a fast computer was only half right, because in old days, the cost of the super-computer time was estimeted based on 1 US dollar/ 1 second. That was in early 80's money.) (6). So, if you are still living on earth, you really have to be more realistic than those "scientists" behind the super-computers. (7). If 40x40 results were acceptable in 70' and 80's, then a 50x50x50 grid should be a good starting point in 90's (your project was initiated in 90's ,even though it is 2000 now.) (8). A grid of 50x50x50 is 125,000 grid points (or cells). Using a commercial code, it can easily require a couple of hundred Mega Bytes of RAM. I think, this is practical in your hardware. (several hundred dollars for the RAM ) (9). So, my practical suggestion is: you have to sit in front of your computer (not in the launch vehicle), and start working with this magic grid of 50x50x50. (10). You also mentioned that you are using a commercial grid generation code. Then you should give the code a real workout and optimize the mesh to something like "Miss America" level. This is the time you decide whether you should keep the grid generation code or not. (11). So, I would be very happy to hear from you, if you can get a converged solution using this most powerful grid code and the space age software. (12). Assuming that the computing time is a linear function of the total number of mesh points, then from 2,500,000 mesh points to 125,000 mesh points, the computing time will be (1/20) of the original estimate time. It will take 12 days instead of the 8 months. (13). If you can't do that, I would strongly suggest that you either throw the codes into the garbage can , or move on to the government labs (there is no guarantee that a converged solution can be found there.) (14). The problem is: the whole thing is a joke. (as for my suggestion, no one will say that a converged 3D solution using a 50x50x50 mesh is no good, especially you have the commercial grid code and the government free software behind you.) (15). By the way, if you are designing an airplane, be sure to put yourself in the pilot seat first. After that, you can play your flight simulator. Have fun.

Hi John,

Some month ago 1 cpu hour on a NEC-SX4 including the license fee for StarCD costs about 75 DM (approx. $ 40). The speedup compared to 1 Processor on an Origin2000 (195 MHz) was about the factor of 6 , but included an unoptimized user-subroutine, which slowed down the NEC a little bit.

The guys there told me that a speedup of 10 or more is quite normal. So I think that this might be a realistic option if the project is setup carefully

(1). That's encouraging. (2). It depends on whether his project is allowed to use any available computer, anywhere. Thank you very much for the cost information.

do both one and two get faster disk (scsi) and about 2 to four times more (faster) RAM no matter the cost (<$1000us) it'll be worth it. three is up to you how accurate do you want your results to be. why not add the second processor. it probably won't hurt

Steve while i understand that you company is small but it seems to me that you need more investment in computer hardware. here at penn state in the aero dept we have a fifty processor pc cluster running linux (PII 500s) linked with 100MBit/s ethernet this machine cost $100,000 in 1998 money. if you had say 20 grand to spend you could set up at least a 10 processor machine. actually you could spend less because we used high end dells as the basis. if you used bargain basement type machines with the new 800 mhz amds your performance per cpu and number of cpu's could be higher. of course you'd have to obtain the parallel wind code and make sure it can work on these type of machines (optimising the code would also help improve performance as well). but if it's an MPI code it'll work just fine and the machines are easy to use and quite powerful. the people who write the WIND code(i assume NASA langley) can advise you on this. i assume that you've determined that the grid you're currently working on is appropriate and can't be coarsened if so your current machine is inadequate. there is a lot of info online about these pc clusters (indeed you could cluster your existing pcs) search for 'beowulf computers' info on the system described above and links can be found at: http://cocoa.ihpca.psu.edu/

if you want to get to Dallas from LA in three hours you can't drive buy a plane ticket, preferrably southwest (ie the cheapest one you can find)

Clifford,

I have bought 3 256MB RAM cards by shopping through www.computershooper.com and saved about $1200 when compared to the CompUSA prices. I have almost maxed out my machine (1 GB RAM); and, I have added the second CPU. I can now run a CFD code and gridgenerator simultaneously and get decent interaction with the grid generator. When I try running two CFD jobs, one job complains that the C disk has not enough temp space; and with Windows NT, the C disk (or partition) cannot have more than 2 GB so I have to move some files out of the C partition and unto the D partition. Hopefully the application files that I moved from the C partition will free sufficient space for the temp files.

Clifford,

I have bought 3 256MB RAM cards by shopping through www.computershooper.com and saved about $1200 when compared to the CompUSA prices. I have almost maxed out my machine (1 GB RAM); and, I have added the second CPU. I can now run a CFD code and gridgenerator simultaneously and get decent interaction with the grid generator. When I try running two CFD jobs, one job complains that the C disk has not enough temp space; and with Windows NT, the C disk (or partition) cannot have more than 2 GB so I have to move some files out of the C partition and unto the D partition. Hopefully the application files that I moved from the C partition will free sufficient space for the temp files.