[calmdown911]

Hi,
I have a difficult problem which I hope someone would be so kind to help me out:
I need to use SLURM to run a STAR-CCM+ simulation. But I don't know how. I tried to write a script by myself. But the job could be submitted but will immediately failed with ExitCode 2.
Here is my script:
#!/bin/bash

#SBATCH --job-name 1218
#SBATCH --partition=DELL
#SBATCH --nodes=4
#SBATCH --ntasks=28
#SBATCH -n 84

#SBATCH -output=1218.out
#SBATCH -e error_flie.e

#SBATCH -t 12:30:00
#SBATCH --mail-type=all
#SBATCH --mail-user=abc@def.com

module load starccm/12.02
module load mpi

sim_file="1218.sim"
STARTMACRO="/home/MYCASE/run.java"

starccm+ 1999@abc.com -np 84 $1218.sim -batch $STARTMACRO -batch-report

[Paulh]

Take a look in the STEVE-Portal. There is an article with a couple of script examples.

[calmdown911]

Originally Posted by Paulh

Take a look in the STEVE-Portal. There is an article with a couple of script examples.

I submitted that question and the scripts seems not working.

[me3840]

Please post the complete error the scheduler is writing out.

[calmdown911]

SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2

[calmdown911]

Originally Posted by me3840

Please post the complete error the scheduler is writing out.

The complete error message is as below:

SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2

Thank you so much for your help!

[spartan42]

Originally Posted by calmdown911

The complete error message is as below:

SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2

Thank you so much for your help!

Is that the error given to you by the output file? [1218.out]

[spartan42]

Originally Posted by calmdown911

The complete error message is as below:

SLURM Job_id=56375 Name=1214 Failed, Run time 00:00:00, FAILED, ExitCode 2

Thank you so much for your help!

Maybe you can benefit out of my slurm file:

#!/bin/sh
#SBATCH --account= [your account]

#SBATCH --time=20:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=28
#SBATCH --mem-per-cpu=1990
#SBATCH --job-name= [job namr]

#SBATCH --error=Job_Results_ERROR-%J.err
#SBATCH --output=Job_Results.out

#SBATCH --export CDLMD_LICENSE_FILE = [1999@ your license]
export CDLMD_LICENSE_FILE=[1999@ your license]

## Simulation File: ====> Source
STARSOURCEFILE=[simulation name.sim]

## SLURM: ====> Job Node List (DO NOT MODIFY)
echo "Slurm nodes assigned :$SLURM_JOB_NODELIST"
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR
echo "SLURM_NTASKS="$SLURM_NTASKS


## Modules: ====> Loading desired mudule (DO NOT MODIFY)
module load starccm/starccm-10.02.012


## Working Directory: ====> Job Node List (DO NOT MODIFY)
cd $SLURM_SUBMIT_DIR
scontrol show hostname $SLURM_NODELIST | tr 'ec' 'ic'> machinefile_${SLURM_JOB_ID}

starccm+ -batch -np $SLURM_NTASKS -rsh ssh -machinefile machinefile_${SLURM_JOB_ID} -power $STARSOURCEFILE &>> job_$SLURM_JOB_ID.log

sleep 3

##rm $hostfile
echo "Run Finished" >> job_$SLURM_JOB_ID.log

[calmdown911]

Originally Posted by spartan42

Maybe you can benefit out of my slurm file:

#!/bin/sh
#SBATCH --account= [your account]

#SBATCH --time=20:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=28
#SBATCH --mem-per-cpu=1990
#SBATCH --job-name= [job namr]

#SBATCH --error=Job_Results_ERROR-%J.err
#SBATCH --output=Job_Results.out

#SBATCH --export CDLMD_LICENSE_FILE=[1999@ your license]

export CDLMD_LICENSE_FILE=[1999@ your license]

## Simulation File: ====> Source
STARSOURCEFILE=[simulation name.sim]

## SLURM: ====> Job Node List (DO NOT MODIFY)
echo "Slurm nodes assigned :$SLURM_JOB_NODELIST"
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR
echo "SLURM_NTASKS="$SLURM_NTASKS


## Modules: ====> Loading desired mudule (DO NOT MODIFY)
module load starccm/starccm-10.02.012


## Working Directory: ====> Job Node List (DO NOT MODIFY)
cd $SLURM_SUBMIT_DIR
scontrol show hostname $SLURM_NODELIST | tr 'ec' 'ic'> machinefile_${SLURM_JOB_ID}

starccm+ -batch -np $SLURM_NTASKS -rsh ssh -machinefile machinefile_${SLURM_JOB_ID} -power $STARSOURCEFILE &>> job_$SLURM_JOB_ID.log

sleep 3

##rm $hostfile
echo "Run Finished" >> job_$SLURM_JOB_ID.log

Thank you so much! I'll try your script. Hope it works!
Merry Christmas!

[calmdown911]

Originally Posted by spartan42

Is that the error given to you by the output file? [1218.out]

The error is send by email.

[spartan42]

tmas to you as well!

[spartan42]

Originally Posted by calmdown911

The error is send by email.

I hope it works. Let me know if you need some help.

Merry Christmas to you as well!

[calmdown911]

Originally Posted by spartan42

I hope it works. Let me know if you need some help.

Merry Christmas to you as well!

Hi, Spartan42,
I tried your scripts and got the error as below:
ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'starccm'
What shall I do for this kind of error?
Thank you so much for your help!

[spartan42]

Originally Posted by calmdown911

Hi, Spartan42,
I tried your scripts and got the error as below:
ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'starccm'
What shall I do for this kind of error?
Thank you so much for your help!

If you look at the script/ the section on the bottom is loading the module for starccm on the server. You need to find out the file directory.

I imagine your using some sort of command/ or putty. Correct?

In the command promp you can type [module avail] and it should populate a list with all the available modules on the server. From there you can search for the starccm module and replace the red part with yours


## Modules: ====> Loading desired mudule (DO NOT MODIFY)

module load starccm/starccm-10.02.012

[calmdown911]

Originally Posted by spartan42

If you look at the script/ the section on the bottom is loading the module for starccm on the server. You need to find out the file directory.

I imagine your using some sort of command/ or putty. Correct?

In the command promp you can type [module avail] and it should populate a list with all the available modules on the server. From there you can search for the starccm module and replace the red part with yours


## Modules: ====> Loading desired mudule (DO NOT MODIFY)

module load starccm/starccm-10.02.012

Hi, Spartan42,
It seems I finally got the simulation running on our cluster. Yet I came across another problem. I activated autosave when I built the case. Yet I cannot get the calculation result. The case file is exactly same before and after the calculation. The residual is recorded in the output file. Yet the result seems lost. What shall I do to get the calculation result from SLURM?

[spartan42]

Originally Posted by calmdown911

Hi, Spartan42,
It seems I finally got the simulation running on our cluster. Yet I came across another problem. I activated autosave when I built the case. Yet I cannot get the calculation result. The case file is exactly same before and after the calculation. The residual is recorded in the output file. Yet the result seems lost. What shall I do to get the calculation result from SLURM?

Can you elaborate a little more?

This is what I got:

You set auto save every x amount of iterations? But its not saving the new sim file correct?

You are getting the residuals from the output file because they are the only active reports in your simulation. If you create other reports and graphs you will see those results in the output file.

When it comes to simulation saving/ - you need to set up the autosave for every x iteration - might be not set up correctly and that is why your not seeing anything. If you getting residuals it means the simulation is running. Maybe take a snip of your autosave setting so i can look at it.

[calmdown911]

Originally Posted by spartan42

Can you elaborate a little more?

This is what I got:

You set auto save every x amount of iterations? But its not saving the new sim file correct?

You are getting the residuals from the output file because they are the only active reports in your simulation. If you create other reports and graphs you will see those results in the output file.

When it comes to simulation saving/ - you need to set up the autosave for every x iteration - might be not set up correctly and that is why your not seeing anything. If you getting residuals it means the simulation is running. Maybe take a snip of your autosave setting so i can look at it.

My autosaving is as attachment. Is it right?

[mazhar16823]

Originally Posted by calmdown911

Thank you so much! I'll try your script. Hope it works!
Merry Christmas!

Hi,


I have used this script but it gives me an error "invalid argument" for my account. Why is this happening?

[erik_lonroth]

Perhaps this could be of help for you aswell https://eriklonroth.com/2020/07/11/r...nd-infiniband/

[bluebase]

If you have a cluster with time limits for jobs, you might like my script https://wikis.ovgu.de/lss/doku.php?i...rccm#variant_1


It tells slurm to send a signal, when a job is about to reach its time limit. The above script traps that signal and will write an ABORT file into the work directory. That way, starccm can stop gracefully.
The advantage with this method is you will not lose the computation since your last (auto)save.
As long as you have an active stop file stopping criterion this methods will work with macro-based runs as well as command-style runs.

Anyhow, this script contains some lines which are specific to my university's cluster. This includes the compute node's resources, name of queues, work directories, availability and names of modules, custom prolog and epilog scripts, and mpi settings.
Of course, you need to adapt these information to your system.