[minzhang]

I have been running a lot of CFDEMcoupling cases successfully.
The only difference between this failed case and other successful cases is that I have more particles (the particle concentration is higher in this failed case).

To debug this failed case, my first try was to run this case again. Interestingly, for the identical two runs, I got different error messages. They crashed at different time points.

In addition, The first run stopped at the beginning of the LIGGGHTS run, but the second run stopped after the LIGGGHTS run finished.

The following is the detailed error messages for the two identical runs.
The first run: Case10:
Time = 0.026

Courant Number mean: 0.00334543 max: 0.0449133

Coupling...
Starting up LIGGGHTS
Executing command: 'run 100 '
srun: error: nid00401: tasks 0-25: Killed
srun: Terminating job step 2742389.0
srun: error: nid00404: tasks 78,80-102: Terminated
srun: error: nid00405: tasks 103-104,106-116,118-127: Terminated
srun: error: nid00403: tasks 52,55-77: Terminated
srun: error: nid00402: tasks 26-34,36-51: Terminated
srun: error: nid00404: task 79: Terminated
srun: error: nid00405: tasks 105,117: Terminated
srun: error: nid00402: task 35: Terminated
srun: error: nid00403: tasks 53-54: Terminated
srun: Force Terminated job step 2742389.0
TACC: MPI job exited with code: 143

TACC: Shutdown complete. Exiting.

The second run: Case10_again:
Time = 0.02678

Courant Number mean: 0.00334562 max: 0.0443878

Coupling...
Starting up LIGGGHTS
Executing command: 'run 100 '
run 100
Setting up run ...
Memory usage per processor = 27.8005 Mbytes
Step Atoms KinEng RotE ts[1] ts[2]
2677901 1460455 4.6400994e+08 391187.51 0.0010050491 0.00073688914
INFO: Particle insertion ins: inserted 3925 particle templates (mass 1.176354) at step 2677996
- a total of 1632546 particle templates (mass 489.286997) inserted so far.
CFD Coupling established at step 2678000
2678001 1464360 4.655741e+08 390592.78 0.0010050491 0.00073688914
Loop time of 2.35193 on 128 procs for 100 steps with 1464360 atoms

Pair time (%) = 0.528199 (22.4581)
Neigh time (%) = 0.0171635 (0.72976)
Comm time (%) = 0.104113 (4.42669)
Outpt time (%) = 0.0142617 (0.60638)
Other time (%) = 1.6882 (71.7791)

Nlocal: 11440.3 ave 21176 max 0 min
Histogram: 10 7 19 14 2 3 5 59 7 2
Nghost: 3240.59 ave 5988 max 0 min
Histogram: 9 6 8 24 3 12 30 17 8 11
Neighs: 74830.6 ave 226078 max 0 min
Histogram: 15 24 7 45 21 4 3 6 1 2

Total # of neighbors = 9578314
Ave neighs/atom = 6.54096
Neighbor list builds = 1
Dangerous builds = 0
LIGGGHTS finished
srun: error: nid00908: tasks 52-63,65-77: Killed
srun: Terminating job step 2789702.0
slurmstepd: error: *** STEP 2789702.0 ON nid00849 CANCELLED AT 2020-04-09T12:42:53 ***
srun: error: nid00908: task 64: Killed
srun: error: nid00849: task 24: Terminated
srun: error: nid01155: tasks 103-109,111-127: Terminated
srun: error: nid00872: tasks 26-31,33-51: Terminated
srun: error: nid00849: tasks 0,2-22,25: Terminated
srun: error: nid00849: task 1: Terminated
srun: error: nid00989: tasks 78-102: Terminated
srun: error: nid01155: task 110: Terminated
srun: error: nid00849: task 23: Terminated
srun: error: nid00872: task 32: Terminated
srun: Force Terminated job step 2789702.0
TACC: MPI job exited with code: 143

TACC: Shutdown complete. Exiting.