[papamilad]

Hello all,

I have successfully setup a 3 node, 120 core HPC using an infiniband switch. It works great running one large case from the master and throwing the entire 120 cores at it but sometimes we have smaller cases we would like to run simultaneously with a smaller number of cores. Rather than putting the cases on each machine and running them from the nodes (or via SSH), is there an easy way to be able to assign IE 6 cases with 20 cores each and have them all ran from the master? I briefly looked into some cluster management software but couldn't find what I was looking for. I am very new to Linux and CFD in general so a nudge in the right direction would be much obliged.

Thank you!

Edit: I should mention we are running OpenFOAM 5 on Ubuntu 18.14

[naffrancois]

Hello,

How did you launch from the master your big simulation using all available nodes and cores?

Using mpi you can edit a machinefile containing the following info:

Node1:8
Node2:8

Assuming Node1 and Node2 properly defined through their respective ip addresses in etc/hosts

Then launch your exe as mpiexec -np 16 -f machinefile exe. It will launch a 16 core mpi job spread over Node1 and Node2.

Otherwise have a look to a job scheduler like slurm. It will allow you more control, and let you send jobs in queue until resource is available, which is quite great before holidays

[papamilad]

Originally Posted by naffrancois

Hello,

How did you launch from the master your big simulation using all available nodes and cores?

Using mpi you can edit a machinefile containing the following info:

Node1:8
Node2:8

Assuming Node1 and Node2 properly defined through their respective ip addresses in etc/hosts

Then launch your exe as mpiexec -np 16 -f machinefile exe. It will launch a 16 core mpi job spread over Node1 and Node2.

Otherwise have a look to a job scheduler like slurm. It will allow you more control, and let you send jobs in queue until resource is available, which is quite great before holidays

That's exactly how our cluster is setup and ran as is. The problem we're running into is if we have 2 cases that don't really scale as well with more cores (IE using 120 cores results in more than 1/2 the time as 60 cores) so running 2 at once is quicker than 2 back to back using the entire cores. The problem is if you run two mpirun -np 60 ... commands at once, it will try to use the same 60 cores for both rather than distribute things to empty cores. Can slurm replace mpirun to do what we want or does it work in conjunction?

[naffrancois]

I may not have understood exactly what you want to do. If you have two machinefiles:

File1:
Node1:40
Node2:20

File2:
Node2:20
Node3:40

Then you launch two distinct mpiexec instances:
mpiexec -np 60 -f File1 execname
mpiexec -np 60 -f File2 execname

Then I do not expect mpi to assign one or more cores to both jobs unless something wrong in your configuration or your cores are already busy with some other job.

Slurm works on top of mpi, it does not replace it. I am not an expert of it, I am using it to put lots of runs in queue and manage resource availability across different users.

[papamilad]

Originally Posted by naffrancois

I may not have understood exactly what you want to do. If you have two machinefiles:

File1:
Node1:40
Node2:20

File2:
Node2:20
Node3:40

Then you launch two distinct mpiexec instances:
mpiexec -np 60 -f File1 execname
mpiexec -np 60 -f File2 execname

Then I do not expect mpi to assign one or more cores to both jobs unless something wrong in your configuration or your cores are already busy with some other job.

Slurm works on top of mpi, it does not replace it. I am not an expert of it, I am using it to put lots of runs in queue and manage resource availability across different users.

That worked flawlessly! Thank you! I'm gonna give Slurm a shot from reading the wiki it seems like it removes the need to manually create a machines file for each case and keep track of what cores/nodes you assign.

[naffrancois]

i am glad you figured it out. As far as slurm is concerned, it is a bit of a pain to configure wrt to other tools but definitely worth the effort if you have to schedule many simulations or if there are various users