SU2 7.0.7 Built on CentOS 7, parallel computation pyscript mpi exit error?

EternalSeekerX · October 9, 2020, 05:13

Hello everyone,

So I was testing SU2 parallel on my CentOS 7 machine. For this test I was running the Unsteady NACA 0012 compressible case. I ran it using 6 processor cores.

I notice that it satisfies convergence criteria at restart_flow_00532. And that is when the script launches the SU2_SOL and this is where the error shows up.

Code:

----------------------------- Solver Exit -------------------------------
All windowed time-averaged convergence criteria are fullfilled.
-------------------------------------------------------------------------
+-----------------------------------------------------------------------+
|        File Writing Summary       |              Filename             |
+-----------------------------------------------------------------------+
|SU2 restart                        |restart_flow_00532.dat             |
|Paraview                           |flow_00532.vtu                     |
|Paraview surface                   |surface_flow_00532.vtu             |
+-----------------------------------------------------------------------+

------------------------- Solver Postprocessing -------------------------
Deleted CNumerics container.
Deleted CIntegration container.
Deleted CSolver container.
Deleted CIteration container.
Deleted CInterpolator container.
Deleted CInterface container.
Deleted CGeometry container.
Deleted CFreeFormDefBox class.
Deleted CSurfaceMovement class.
Deleted CVolumetricMovement class.
Deleted CConfig container.
Deleted nInst container.
Deleted COutput class.
-------------------------------------------------------------------------

------------------------- Exit Success (SU2_CFD) ------------------------


-------------------------------------------------------------------------
|    ___ _   _ ___                                                      |
|   / __| | | |_  )   Release 7.0.6 "Blackbird"                         |
|   \__ \ |_| |/ /                                                      |
|   |___/\___//___|   Suite (Solution Exporting Code)                   |
|                                                                       |
-------------------------------------------------------------------------
| SU2 Project Website: https://su2code.github.io                        |
|                                                                       |
| The SU2 Project is maintained by the SU2 Foundation                   |
| (http://su2foundation.org)                                            |
-------------------------------------------------------------------------
| Copyright 2012-2020, SU2 Contributors                                 |
|                                                                       |
| SU2 is free software; you can redistribute it and/or                  |
| modify it under the terms of the GNU Lesser General Public            |
| License as published by the Free Software Foundation; either          |
| version 2.1 of the License, or (at your option) any later version.    |
|                                                                       |
| SU2 is distributed in the hope that it will be useful,                |
| but WITHOUT ANY WARRANTY; without even the implied warranty of        |
| MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU      |
| Lesser General Public License for more details.                       |
|                                                                       |
| You should have received a copy of the GNU Lesser General Public      |
| License along with SU2. If not, see <http://www.gnu.org/licenses/>.   |
-------------------------------------------------------------------------

----------------- Physical Case Definition ( Zone 0 ) -------------------
Input mesh file name: unsteady_naca0012_mesh.su2

-------------------- Output Information ( Zone 0 ) ----------------------
The tabular file format is Tecplot (.dat).
Flow variables file name: flow.

------------- Config File Boundary Information ( Zone 0 ) ---------------
+-----------------------------------------------------------------------+
|                        Marker Type|                        Marker Name|
+-----------------------------------------------------------------------+
|                          Far-field|                           farfield|
+-----------------------------------------------------------------------+
|                     Heat flux wall|                            airfoil|
+-----------------------------------------------------------------------+
Two dimensional problem.
14576 grid points before partitioning.
14336 volume elements before partitioning.
2 surface markers.
128 boundary elements in index 0 (Marker = airfoil).
352 boundary elements in index 1 (Marker = farfield).
Executing the partitioning functions.
Building the graph adjacency structure.
Calling ParMETIS... graph partitioning complete (465 edge cuts).
Distributing ParMETIS coloring.
Rebalancing vertices.
Rebalancing volume element connectivity.
Rebalancing markers and surface elements.
15510 vertices including ghost points. 
14797 interior elements including halo cells. 
14797 quadrilaterals.
Establishing MPI communication patterns.
Identify vertices.
Storing a mapping from global to local point index.

------------------------- Solution Postprocessing -----------------------
Volume output fields: COORDINATES, SOLUTION
  Info: Ignoring the following screen output fields:
  DRAG, LIFT, RMS_DENSITY, REL_RMS_DENSITY, CAUCHY_TAVG_DRAG, CAUCHY_TAVG_LIFT
Screen output fields: TIME_ITER, INNER_ITER
  Info: Ignoring the following history output groups:
  REL_RMS_RES, RMS_RES, AERO_COEFF, TAVG_AERO_COEFF, CAUCHY, D_TAVG_AERO_COEFF
History output group(s): ITER
Ignoring Convergence Field(s): REL_RMS_DENSITY 
Warning: No (valid) fields chosen for convergence monitoring. Convergence monitoring inactive.
Ignoring Time Convergence Field(s): TAVG_DRAG TAVG_LIFT 
Warning: No (valid) fields chosen for time convergence monitoring. Time convergence monitoring inactive.
Reading and storing the solution from restart_flow_00499.
Writing the volume solution for time step 499.
|Paraview                           |flow_00499.vtu                     |
|Paraview surface                   |surface_flow_00499.vtu             |
Reading and storing the solution from restart_flow_00500.
Writing the volume solution for time step 500.
|Paraview                           |flow_00500.vtu                     |
|Paraview surface                   |surface_flow_00500.vtu             |
Reading and storing the solution from restart_flow_00501.
Writing the volume solution for time step 501.
|Paraview                           |flow_00501.vtu                     |
|Paraview surface                   |surface_flow_00501.vtu             |
Reading and storing the solution from restart_flow_00502.
Writing the volume solution for time step 502.
|Paraview                           |flow_00502.vtu                     |
|Paraview surface                   |surface_flow_00502.vtu             |
Reading and storing the solution from restart_flow_00503.
Writing the volume solution for time step 503.
|Paraview                           |flow_00503.vtu                     |
|Paraview surface                   |surface_flow_00503.vtu             |
Reading and storing the solution from restart_flow_00504.
Writing the volume solution for time step 504.
|Paraview                           |flow_00504.vtu                     |
|Paraview surface                   |surface_flow_00504.vtu             |
Reading and storing the solution from restart_flow_00505.
Writing the volume solution for time step 505.
|Paraview                           |flow_00505.vtu                     |
|Paraview surface                   |surface_flow_00505.vtu             |
Reading and storing the solution from restart_flow_00506.
Writing the volume solution for time step 506.
|Paraview                           |flow_00506.vtu                     |
|Paraview surface                   |surface_flow_00506.vtu             |
Reading and storing the solution from restart_flow_00507.
Writing the volume solution for time step 507.
|Paraview                           |flow_00507.vtu                     |
|Paraview surface                   |surface_flow_00507.vtu             |
Reading and storing the solution from restart_flow_00508.
Writing the volume solution for time step 508.
|Paraview                           |flow_00508.vtu                     |
|Paraview surface                   |surface_flow_00508.vtu             |
Reading and storing the solution from restart_flow_00509.
Writing the volume solution for time step 509.
|Paraview                           |flow_00509.vtu                     |
|Paraview surface                   |surface_flow_00509.vtu             |
Reading and storing the solution from restart_flow_00510.
Writing the volume solution for time step 510.
|Paraview                           |flow_00510.vtu                     |
|Paraview surface                   |surface_flow_00510.vtu             |
Reading and storing the solution from restart_flow_00511.
Writing the volume solution for time step 511.
|Paraview                           |flow_00511.vtu                     |
|Paraview surface                   |surface_flow_00511.vtu             |
Reading and storing the solution from restart_flow_00512.
Writing the volume solution for time step 512.
|Paraview                           |flow_00512.vtu                     |
|Paraview surface                   |surface_flow_00512.vtu             |
Reading and storing the solution from restart_flow_00513.
Writing the volume solution for time step 513.
|Paraview                           |flow_00513.vtu                     |
|Paraview surface                   |surface_flow_00513.vtu             |
Reading and storing the solution from restart_flow_00514.
Writing the volume solution for time step 514.
|Paraview                           |flow_00514.vtu                     |
|Paraview surface                   |surface_flow_00514.vtu             |
Reading and storing the solution from restart_flow_00515.
Writing the volume solution for time step 515.
|Paraview                           |flow_00515.vtu                     |
|Paraview surface                   |surface_flow_00515.vtu             |
Reading and storing the solution from restart_flow_00516.
Writing the volume solution for time step 516.
|Paraview                           |flow_00516.vtu                     |
|Paraview surface                   |surface_flow_00516.vtu             |
Reading and storing the solution from restart_flow_00517.
Writing the volume solution for time step 517.
|Paraview                           |flow_00517.vtu                     |
|Paraview surface                   |surface_flow_00517.vtu             |
Reading and storing the solution from restart_flow_00518.
Writing the volume solution for time step 518.
|Paraview                           |flow_00518.vtu                     |
|Paraview surface                   |surface_flow_00518.vtu             |
Reading and storing the solution from restart_flow_00519.
Writing the volume solution for time step 519.
|Paraview                           |flow_00519.vtu                     |
|Paraview surface                   |surface_flow_00519.vtu             |
Reading and storing the solution from restart_flow_00520.
Writing the volume solution for time step 520.
|Paraview                           |flow_00520.vtu                     |
|Paraview surface                   |surface_flow_00520.vtu             |
Reading and storing the solution from restart_flow_00521.
Writing the volume solution for time step 521.
|Paraview                           |flow_00521.vtu                     |
|Paraview surface                   |surface_flow_00521.vtu             |
Reading and storing the solution from restart_flow_00522.
Writing the volume solution for time step 522.
|Paraview                           |flow_00522.vtu                     |
|Paraview surface                   |surface_flow_00522.vtu             |
Reading and storing the solution from restart_flow_00523.
Writing the volume solution for time step 523.
|Paraview                           |flow_00523.vtu                     |
|Paraview surface                   |surface_flow_00523.vtu             |
Reading and storing the solution from restart_flow_00524.
Writing the volume solution for time step 524.
|Paraview                           |flow_00524.vtu                     |
|Paraview surface                   |surface_flow_00524.vtu             |
Reading and storing the solution from restart_flow_00525.
Writing the volume solution for time step 525.
|Paraview                           |flow_00525.vtu                     |
|Paraview surface                   |surface_flow_00525.vtu             |
Reading and storing the solution from restart_flow_00526.
Writing the volume solution for time step 526.
|Paraview                           |flow_00526.vtu                     |
|Paraview surface                   |surface_flow_00526.vtu             |
Reading and storing the solution from restart_flow_00527.
Writing the volume solution for time step 527.
|Paraview                           |flow_00527.vtu                     |
|Paraview surface                   |surface_flow_00527.vtu             |
Reading and storing the solution from restart_flow_00528.
Writing the volume solution for time step 528.
|Paraview                           |flow_00528.vtu                     |
|Paraview surface                   |surface_flow_00528.vtu             |
Reading and storing the solution from restart_flow_00529.
Writing the volume solution for time step 529.
|Paraview                           |flow_00529.vtu                     |
|Paraview surface                   |surface_flow_00529.vtu             |
Reading and storing the solution from restart_flow_00530.
Writing the volume solution for time step 530.
|Paraview                           |flow_00530.vtu                     |
|Paraview surface                   |surface_flow_00530.vtu             |
Reading and storing the solution from restart_flow_00531.
Writing the volume solution for time step 531.
|Paraview                           |flow_00531.vtu                     |
|Paraview surface                   |surface_flow_00531.vtu             |
Reading and storing the solution from restart_flow_00532.
Writing the volume solution for time step 532.
|Paraview                           |flow_00532.vtu                     |
|Paraview surface                   |surface_flow_00532.vtu             |
Reading and storing the solution from restart_flow_00533.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD 
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------


Error in "void CSolver::Read_SU2_Restart_Binary(CGeometry*, CConfig*, std::string)": 
-------------------------------------------------------------------------
Unable to open SU2 restart file restart_flow_00533.dat
------------------------------ Error Exit -------------------------------


Traceback (most recent call last):
  File "/home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/parallel_computation.py", line 110, in <module>
    main()
  File "/home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/parallel_computation.py", line 57, in main
    options.compute      )
  File "/home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/parallel_computation.py", line 96, in parallel_computation
    info = SU2.run.merge(config)
  File "/home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/SU2/run/merge.py", line 81, in merge
    merge_solution(konfig)
  File "/home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/SU2/run/merge.py", line 107, in merge_solution
    SU2_SOL( config )
  File "/home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/SU2/run/interface.py", line 229, in SOL
    run_command( the_Command )
  File "/home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/SU2/run/interface.py", line 292, in run_command
    raise exception(message)
SU2.EvaluationFailure: Path = /home/CentOS-VM/Documents/SU2/Run/Compressible_Tutorial/Unsteady_NACA0012/,
Command = mpirun -n 6 /home/CentOS-VM/.root/opt/SU2/SU2-7.0.7/bin/SU2_SOL config_SOL.cfg
SU2 process returned error '1'
[localhost:04234] 5 more processes have sent help message help-mpi-api.txt / mpi-abort
[localhost:04234] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

It seems that when the script reaches the last data file, it throws this error, is this expected? Seems like mpi doesn't like being closed like this?

Thanks

pcg · October 9, 2020, 07:10

SU2_SOL is no longer necessary, files are written out by SU2_CFD using parallel I/O.

The parallel_computation script has lagged behind and still calls SU2_SOL (which used to be necessary in version 6).

Best advice I can give you is forget about all the "friendly" python scripts, just run mpirun/mpiexec -n ? SU2_CFD.

EternalSeekerX · October 9, 2020, 13:44

Quote:

Originally Posted by pcg

SU2_SOL is no longer necessary, files are written out by SU2_CFD using parallel I/O.

The parallel_computation script has lagged behind and still calls SU2_SOL (which used to be necessary in version 6).

Best advice I can give you is forget about all the "friendly" python scripts, just run mpirun/mpiexec -n ? SU2_CFD.

Ahh, I got it. I'll try using mpirun then. I did notice even with the script, It exits SU2_CFD fine and also even with the mpi error when running SU2_SOL all the result files do output and everything works fine, just mpi gets whirly about it. So I can safely assume my install was correct?

Edit: Just ran using mpirun and it completed the run and written the vtu files perfectly. So yeah I think the build went correctly then!?

pcg · October 9, 2020, 19:28

Yep all should be fine.

At the moment SU2_SOL is only useful if after running SU2_CFD you want to export to a different format.
The problem you described sounds like a bug in SU2_SOL for time domain problems, we'll have a look.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[OpenFOAM.org] Compile OF 2.3 on Mac OS X .... the patch	gschaider	OpenFOAM Installation	225	August 25, 2015 20:43
Undeclared Identifier Errof UDF	SteveGoat	Fluent UDF and Scheme Programming	7	October 15, 2014 08:11
Compiling dynamicTopoFvMesh for OpenFOAM 2.1.x	Saxwax	OpenFOAM Installation	25	November 29, 2013 06:34
CGNS lib and Fortran compiler	manaliac	Main CFD Forum	2	November 29, 2010 07:25
Installation OF1.5-dev	ttdtud	OpenFOAM Installation	46	May 5, 2009 03:32

October 9, 2020, 07:10		#2
pcg Senior Member Pedro Gomes Join Date: Dec 2017 Posts: 466 Rep Power: 13	SU2_SOL is no longer necessary, files are written out by SU2_CFD using parallel I/O. The parallel_computation script has lagged behind and still calls SU2_SOL (which used to be necessary in version 6). Best advice I can give you is forget about all the "friendly" python scripts, just run mpirun/mpiexec -n ? SU2_CFD.

October 9, 2020, 19:28		#4
pcg Senior Member Pedro Gomes Join Date: Dec 2017 Posts: 466 Rep Power: 13	Yep all should be fine. At the moment SU2_SOL is only useful if after running SU2_CFD you want to export to a different format. The problem you described sounds like a bug in SU2_SOL for time domain problems, we'll have a look.