[kostaspan]

I have installed PETSc libraries on Linux platform for solution of large sparse linear systems which derived by an implicit scheme at finite element code.
The installation integrated successful but i can use it.

I put as includes in C language #include"petscksp.h" but when i try to compile the program, i take errors. If someone know to tell me what flags must i write for compilation and what else muct be specified for right reading of all petsc libraries.

[jed]

If the errors are about the missing header, you probably need -I$PETSC_DIR/include -I$PETSC_DIR/$PETSC_ARCH/include. One way to do this is to use PETSc makefiles (which can also handle linking). If it's something else, please send mail to petsc-maint@mcs.anl.gov (you will get a prompt reply).

[kostaspan]

I use c of gcc. At ./configuration i put and --with-c-support. Its ok this for c of gcc?

[jed]

You can use --with-cc=gcc, but it will usually pick up this compiler anyway. You don't need --with-c-support, that is only if you are compiling PETSc with a C++ compiler, but want to be able to call PETSc from C.

[kostaspan]

Yes i put --with-c-support. Do you thing that i have problem with commands in code or compiling?
Now i reinstall --with-cc=gcc.


I want to ask you about functions for matrix construction. When i use MatSetValues function with very large matrix it take me so long time for finish. And is supposed that these constructions must be work quickly that to use in finite element code with very large systems. Have you any ideas for this? What i do?

[jed]

MatSetValues is slow because you didn't preallocate correctly. http://www.mcs.anl.gov/petsc/petsc-a...cient-assembly

[kostaspan]

Ok. I do some things which i read. But with 5000x5000 elements in matrix its take again much time. Below i give sou the construct of this matrix and please tell me what i do wrong. I have already lost a lot of days for this problem .
Below A: matrix , col3 : pointer to int for number of nonzeros at each row.


MatCreateSeqAIJ(PETSC_COMM_SELF,4000,4000,4000,col 3,&A);

for(n=0;n<4000;n++){
for(m=0;m<4000;m++){
acol1=n;
acol2=m;

if(run.STFMline[n*4000+m]!=0.0){
MatSetValue(A,acol1,acol2,run.STFMline[n*4000+m],INSERT_VALUES);
}
}
}

MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);



More time take first function.

[jed]

Looks like you have a dense matrix containing lots of explicit zeros, is possible that they are just very close to zero (but not exactly zero so they still get inserted)? The dimensions 4000x4000 are not at all large. How many nonzeros do you expect (what is in col3)? Run with -info | grep malloc, the allocation is almost certainly insufficient. To generate an error when inserting a value that has not been allocated, call

MatSetOption(A,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_TRUE);

after creating the matrix, but before inserting values.

[kostaspan]

Is there any function which convert a allocated pointer to Mat type matrix? Why to allocate memory for a matrix which i have already made in pointer (ex. double * or float *) ?

I did all above which you tell me but it takes same time...

[jed]

Were there any mallocs during insertion? How long does it take for a particular size? If you have a dense matrix, you can use MatCreateSeqDense() and pass in your array. But it doesn't make sense to hold a sparse matrix with dense storage, usually people create the matrix using MatSetValues rather than assembling into their own structure and copying over.

[kostaspan]

How many elements in matrix have you solved with petsc? I will try again other functions and i send you if i have problem.

Thanks for help

[jed]

Matrix dimensions of a few hundred million, others have done 500 billion, these matrices usually have from a few 10s to 100s of nonzeros per row. Usually memory limits single-process matrix dimensions to a couple million for 3D problems, a bit more in 2D (many more nonzeros per row in 3D). It would be helpful if you describe what you are trying to do (e.g. where the matrix comes from and what you want to do with it).

[kostaspan]

I have made code for spectral finite element analysis (DG method) in 3D for diffusion-convection equation. Until now i used only LU Dec direct method for steady or unsteady (explicit ) schemes because the DG method has an advantage of iterative procedure (i have many small systems with same left side at each element).
Before five days i make the implicit for above equation which requires the creation of stiffness matrix which include all DOFs of all elements. This matrix(by which we discuss) remain const for all time steps but change RHS where you know.
Therefore, at each time step i can solute with GMRES the system for to take the solution at next time. This matrix for 40000 elements and second order approximation (p2) become ~4000000 X4000000. But i here there is the problem. It take much time for allocate memory the petsc lib. I dont have such problems with malloc of c. It work quickly.
And documentation of Petsc dosnt give understandable things... If you can tell me which functions and which way you use...


Kostas

[proximityinfotech6]

I put as includes in C language #include"petscksp.h" but when i try to compile the program, i take errors. If someone know to tell me what flags must i write for compilation and what else muct be specified for right reading of all petsc libraries. E20 fuel, which blends 20 percent ethanol with gasoline, reduces the tail pipe emissions of hydrocarbons and carbon monoxide, compared with traditional gasoline or E10 blends, according to a new study in the Journal of Automobile Engineering. In addition, the research found no measurable impact to vehicle drivability or maintenance in conventional internal combustion engines.
_______________________________________________
Used Auto Parts | Used Car Parts

[jed]

You still haven't answered my question about number of mallocs during insertion, nor told me how slow is "slow". Your matrix dimension of 4M doesn't make sense, 40k Q_2 hex elements produces 27*40k=1.08M, 40k P_2 tet elements produces 10*40k=400k, both of which are significantly smaller than 4M. Additionally, your quoted code is clearly extracting from a dense matrix, which you certainly can't have for large sizes (and doesn't make sense for this problem). Of course creating a dense matrix bigger than available memory will cause lots of swapping and be tremendously slow. You should MatCreateSeqAIJ (or more generically, see the user's manual) with correct precallocation (the number of nonzeros per row depends on which elliptic flux you use) or at least an upper bound (e.g. 27*7 for structured hexes with a local flux), then loop over elements and faces, computing local matrices and inserting them.

[kostaspan]

I did relatively many mallocs because i havent completed the optimization. I usually do this in order to have easy procedures and structures as far as possible. Now i correct this and the only values which saved for procedure are these in Sparse Mat . Of course, only non-zeros it must be saved. Finally, i found the petsc functions for efficient creation Mat. I give these lines below:
Therefore all values as blocks of size (sizeb) saved in A which have preallocation memory for only non-zeros blocks.

MatCreateSeqBAIJ(PETSC_COMM_SELF,sizeb,rows,cols,8 ,0,&A);

MatSetValuesBlocked(A,1,&cola,1,&colb,ImpEline,INS ERT_VALUES);

for(f=0;f<run.el[i].tfcnume;f++){
MatSetValuesBlocked(A,1,&cola,1,&colb,ImpFline[f],INSERT_VALUES);

}
}

MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);



With this i run 100.000 x100000 matrices in a few sec...