[Johan M]

Hi Everyone,

I will soon be attempting a single CFX simulation, that is part of a Workbench workflow, on a cluster for the first time. I have been testing out commands to automatically run my simulation from the CFX command line on my personal PC first. Currently I have the basic code below:

I first used the cd command to change the default directory to the directory containing my .def file:

C:\Users\User>cd C:\Scratch\Johan Sims\Test


I then added commands about using double precision and the number of partitions:
C:\Scratch\Johan Sims\Tesst>cfx5solve -def CFX.def -double -par-local -part 22


The above works fine. However, I would like to confirm a few remarks about adding more to the above lines and using clusters in general:

1. I explicitly stated the working directory to get the above running. I was wondering how one would go about this in the event the working directory on the cluster is not exactly known (I suspect this will be cluster case dependent scenario but I will ask in any case)

2. I noticed in my ''Running a Job guidelines'' doc it stated a line that needed a file path to a .err file. I was not exactly sure what file this is refering to, seen below:

-e /mnt/lustre3p/users/username/testCFX/test.err

3. There is another line of code in the Job template I'd like to clarify, it states:

cd /mnt/lustre3p/users/username/testCFX #change to CFX case directory

I recognize that the comment and code 'cd' refers to changing the directory. In that comment, it mentions CFX case - does that have any relevance to uploading the CFX case file or is it simply the generic folder name? I suspect the latter

4. I would like to save the backup files every 20 mins. Should I simply add the following to the existing lines of code:

-baket "20 [min]"

5. I would like to know what is the appropriate file type to put the code in to hand over to the cluster? eg such as a java script file

6. To confirm, only the .def file is required to be uploaded for a CFX cluster simulation?

7. I set sim to stop at a specific iteration. I then plan to extract the .res file from the cluster and import into the CFX solver manager to check residuals/monitor plots and CFX-Post on my personal PC to gather results. Is this workflow advisable or is there perhaps a better way

Thank you,
Johan

[ghorrocks]

1. I do not understand your question. The def file will be in a location, so a temp directory setup in that directory will be the working directory by default.

2. I do not understand this question either. What is the "Running a Job Guidelines" document? If that is the guidelines for your cluster then I cannot say what it means as I have no idea how your cluster is set up.

3. I do not understand this either. What is the "Job template"?

4. Your cluster may have controls to do this - if so then I cannot help as I have no idea about your cluster. But CFX can be set up to save backup and/or transient results files every few time steps/iterations.

5. I have no idea, it depends on your cluster. When I run CFX I usually do it from solver manager (for one-off stuff) or the command line (for batching lots of jobs together). We have previously used batch scheduling software (PBS pro) when we had a large group of people doing simulation work but as there is only a small group of survivors left now that is not required and we just use the command line.

6. The def file defines most CFX simulations. Some simulations require other files, such as initial conditions or user fortran.

7. Yes, that can work. But if you map/mount the cluster drive location on your personal PC then you do not need to move the files anywhere, you can view them easily directly from your PC. But your cluster may have rules about moving data off the drive or maximum user data size which means you will have to move data around.

[Johan M]

Originally Posted by ghorrocks

1. I do not understand your question. The def file will be in a location, so a temp directory setup in that directory will be the working directory by default.

2. I do not understand this question either. What is the "Running a Job Guidelines" document? If that is the guidelines for your cluster then I cannot say what it means as I have no idea how your cluster is set up.

3. I do not understand this either. What is the "Job template"?

4. Your cluster may have controls to do this - if so then I cannot help as I have no idea about your cluster. But CFX can be set up to save backup and/or transient results files every few time steps/iterations.

5. I have no idea, it depends on your cluster. When I run CFX I usually do it from solver manager (for one-off stuff) or the command line (for batching lots of jobs together). We have previously used batch scheduling software (PBS pro) when we had a large group of people doing simulation work but as there is only a small group of survivors left now that is not required and we just use the command line.

6. The def file defines most CFX simulations. Some simulations require other files, such as initial conditions or user fortran.

7. Yes, that can work. But if you map/mount the cluster drive location on your personal PC then you do not need to move the files anywhere, you can view them easily directly from your PC. But your cluster may have rules about moving data off the drive or maximum user data size which means you will have to move data around.

Hi Ghorrocks,

Thanks for the reply. Sorry, I should have mentioned that I was referring to guidelines within the cluster for certain points:

1. The default directory was not the working directory containing my .def for the sim on my local pc. I had to manually change the directrory in the Command line. My initial question was along the lines of how users generally specify the working directory on a cluster

2. Sorry I should have mentioned that initially. Yes, it is guideline from our cluster. It is a .qsub file but I uploaded it as a notepad file here

3. The Job template is a generic script they provided us for our cluster. I have attached it

4. Thanks, I will look into it

5., 6., 7. Thanks for the confirmation

Kind regards,
Johan

[ghorrocks]

1. The command "cd" is about as basic as they come. Note that clusters almost always run from the command line (ie no GUI), so you are going to have to learn some basic command line stuff to run on it.

2. I do not see any guidelines doc. Don't worry about uploading it, I do not have time to read your cluster documentation, that is for you to do.

3. The attached file makes it look like you are using PBS on a linux cluster. So you have a template to set up your jobs - so go for it.

[Johan M]

Originally Posted by ghorrocks

1. The command "cd" is about as basic as they come. Note that clusters almost always run from the command line (ie no GUI), so you are going to have to learn some basic command line stuff to run on it.

2. I do not see any guidelines doc. Don't worry about uploading it, I do not have time to read your cluster documentation, that is for you to do.

3. The attached file makes it look like you are using PBS on a linux cluster. So you have a template to set up your jobs - so go for it.

Hi Ghorrocks,

Thank you for the advice, I will get on it.

Kind regards,
Johan