Hi,

I am developing a 3D code utilizing the FEM. I am also using MPI.

Because the problem is 3D, my biggest problem is memory consumption and cpu time. The latter problem is partially taken care of with MPI.

My question is how to deal with the memory consumption. Currently, I have 28000 elements (755920 unknowns) and the code takes about 5GB of memory. Although this may seems to be a large number of elements, I need to further increase the number of elements. This amount of memory is used by the master CPU, the other CPUs are using about 150MB of memory.

Up to now, I am breaking the domain into pieces, and each CPU is solving the eqns on a portion of the domain. Then, I add all these matrices up without worrying about duplicated entries. Because of that, my code consumes a lot of memory.

I tried to fix the aforementioned problem by searching for duplicated entries in the global matrix. Although this has reduced the amount of required memory by a factor of 3, it takes for ever to construct the global matrix.

Does anybody know any other method to deal with this problem?

Thank you, Vasilis

It sounds like you need to get rid of the master CPU (if I've understood correctly that the master is using a lot of memory while the others are not). Basically you should avoid having a master cpu in MPI in general - for optimal performance all cpu's should be doing exactly the same amount of work.

Are you are only storing the nonzero coefficients of your stiffness matrix? It would appear that you are not. You should be taking advantage of the sparsity of the stiffness matrix.

I use the master CPU to store the global matrix. This matrix requires a lot of memory.

Yes, I am storing only the non-zero coefficients. The global matrix requires a lot of memory because I have duplicated entries. Now, I am developing a code that deals with these duplicated entries, but it is extremely slow. It is running for about 50min, and it's not finished.

He didn't say the matrix used 5 TB.

The matrix should also be distributed. It's really a lot of work to do a good job of this. I recommend using PETSc to manage the parallel data structures and solvers. This way, you distribute the mesh and call a function to insert values into the matrix. At the end of the assembly, you call a function which performs necessary communication. This scales well (10^10 dofs, 10^5 processors).

The matrix is distributed, but the master cpu assembles the global matrix. I am solving the eqns using MUMPS, which is a parallel solver.

PETSc sounds like a promising program, I wander how difficult is to use it, though

The matrix should never reside on one process. You should assemble for the owned elements on each process. PETSc can use MUMPS, but it can also use lots of other preconditioners (including BoomerAMG and ML). It is much easier to use PETSc than to not use PETSc. Read through the relevant sections of the manual and look at some examples.

I was assembling the local matrices to a global matrix (which demanded a huge amount of memory), when in fact MUMPS could solve the system with the local matrices. Surprisingly enough, now that I am using the local matrices its faster!!

Thank you all for your help!

Use UMFPACK.

[cra]

Hi Vasilis
I use a 3D – FEM model (written in FORTRAN 90). The problem is it can take awful long time to run for large spatial-temporal cases. Some instances the solver can take up to 90-95% of the run time. I’m keen to use a parallel solver to overcome this issue. I had a look at the MUMPS and think that can help. But I’m not a programmer and bit hard to get my head around how to go about this. As you’ve used the MUMPS with your FEM model (and MPI) just wonder whether you can advise me on how to do this. Is there any chance of giving me your source code so that I can see how should I modify my code?
Thank you in advance