help to run CFX simulation via HPC, please

happy · February 23, 2012, 04:57

Hi anyone,,,
I'm now trying to conduct my simulation faster via HPC at my Uni. the probles is that my script file give me the results in .out file not .res file. Actually, I took the csript file from website and I have no idea how I can make my outlet as .res file. Here you are my script file.
#!/bin/bash -l
#
##### Job Name #####
#PBS -N 3cm
##### Select resources ####
#PBS -l nodes=1

pn=4
#PBS -l walltime=01:00:00
##### Goto the directory from which you submitted the job
cd $PBS_O_WORKDIR
##### Request that regular output (stdout) and terminal output (stderr) go to the same file
PBS -o /home/eng/0000000/Gap_width=3cm.out
#PBS -wd
##### Mail Options #####
#PBS -m abe
#PBS -M 0000000@gmail.000.edu.au
##### Queue #####
#PBS -q standard
##### Add ansys_module
module load ansys
##### Specifies the version of ANSYS CFX
version=3d
# The following line would start the simulation from a
# previous results file, Gap_width=3cm_001.res
cfx5solve -double -def Gap_width=3cm.def -initial Gap_width=3cm_001.res -part 4 $nprocs -par-local
I appretiate any advise.

Lance · February 23, 2012, 05:45

Did you get the script from a random webpage or was it given to you by the IT-guys and girls at your uni? Anyway, they should be able to help you with your batch script. Are you sure you are supposed to use PBS?

Also, this is a bit odd:
##### Specifies the version of ANSYS CFX
version=3d

I doubt you need to specify 3D, as CFX always solve in 3D (unlike Fluent, which can be 2D)

ghorrocks · February 23, 2012, 07:18

CFX has a demo script file for running CFX under PBS in the documentation and the script file is at <CFX_ROOT>/extras/qcfx5solve.pl

happy · February 23, 2012, 20:12

Thanks guys
For Glenn, yes,, U R right.
Our ICT devition do not want to help me to write my script file because as I understand they feel it is not their job and they get mad quickly. that is why I'm email you to help me.
I got this error however It produce my out file but it also give the messgae:

Capability CFX_SOLVER_BASIC_MP_LIMIT does not exist in the ANSYS licensing pool.
None of the products enabling this capability are available in the specified license path:
ANSYSLI_SERVERS: 2325@139.86.160.180
FLEXlm Servers: 1055@139.86.160.180
=>> PBS: job killed: walltime 92 exceeded limit 50
A fatal error has occurred in cfx5solve:
cfx5solve was killed by the user.For general information, the ansys should be work in batch mode.

happy · February 23, 2012, 23:58

Quote:

Originally Posted by happy

Thanks guys
For Glenn, yes,, U R right.
Our ICT devition do not want to help me to write my script file because as I understand they feel it is not their job and they get mad quickly. that is why I'm email you to help me.
I got this error however It produce my out file but it also give the messgae:

Capability CFX_SOLVER_BASIC_MP_LIMIT does not exist in the ANSYS licensing pool.
None of the products enabling this capability are available in the specified license path:
ANSYSLI_SERVERS: 2325@139.86.160.180
FLEXlm Servers: 1055@139.86.160.180
=>> PBS: job killed: walltime 92 exceeded limit 50
A fatal error has occurred in cfx5solve:
cfx5solve was killed by the user.For general information, the ansys should be work in batch mode.

my out file is here and the solver stop due to error; hwever, my out file is standards file nor in .res form!!!!how I can load .out file to cfxpost. I tried but give me an error!!!By the way, in my home directory there is many files created by solver due to calculation but interstingly without extention. help me ASAP

Lance · February 24, 2012, 03:47

Quote:

Originally Posted by happy

None of the products enabling this capability are available in the specified license path:
ANSYSLI_SERVERS: 2325@139.86.160.180
FLEXlm Servers: 1055@139.86.160.180
=>> PBS: job killed: walltime 92 exceeded limit 50

Lower your time to less than 50 wall clock hours. Also, check that your license settings are correct.
I couldnt check your out file, post the error messages here instead.

ghorrocks · February 24, 2012, 07:15

Also, do you need to use PBS? If you are not familiar with batch scheduling software it is just making things hard for you. You can run the solver from the command line, and you can queue up a bunch of jobs if you like. If that is all you want then forget about PBS.

But if you need the power of a batch scheduling system or the HPC submits all jobs with PBS and you cannot go with the command line then you better stick with it.

happy · February 25, 2012, 02:35

Hi all
I would like to thank Lance for his/her input and would like to tell Glenn that my simulation now are running by PBS in perfect way

ghorrocks · February 25, 2012, 06:54

Good to hear.

I have wondered why CFX includes such a complex PBS script when much simpler scripts (like the one you have) work. PBS is a bit tricky to set up, especially if you want it to queue based on license availability rather than just CPU availability.

happy · February 25, 2012, 23:05

Hi Glenn,,
please could you please give me more explaination for 'based on license availability rather than just CPU availability'. I mean which sort of mode can work based on the CPU availability?
Actually, I did not have HPC license.
Best Regards

ghorrocks · February 26, 2012, 05:13

The normal mode of operation of PBS is to queue jobs based on the availability of CPUs. This is fine for unlicensed software, but for licensed software often the limitation is how many licenses you have. To make PBS also queue based on license availability we had to add a custom routine to PBS. From memory it checks the license availability on the license server every minute or so and picks up available licenses that way. We talked to PBS support to help develop this. But I remain a little surprise about this because I would have thought there would be lots of customers out there who are license limited so there should be strong demand for this.

happy · February 26, 2012, 20:59

Thanks Glenn for your input.
I read your thesis and you looks ANSYS-CFX exprtise

. First, could you please tell me the cause of this error when I conduct my running via HPC(PBS).
PBS: job killed: walltime 43238 exceeded limit 43200
However, when I reduce the no. of iteration from 1000 to 300, the .res file produced. I tried to set the walltime but without any benefit. does this error come from the limitation mem on my home directory???By the way, I did not set the required memory for my job into my script file.
I appreciate any input

.
Regards

ghorrocks · February 27, 2012, 00:05

Firstly, PBS is not part of CFX. PBS is a third party commercial software from Altair http://www.pbsworks.com/. If you want help with PBS you are best to ask your Altair support person.

But to my reading your error message is pretty obvious - you have set a maximum wall time limit and that was exceeded so it stopped the job. If you want to run longer you need to define a longer wall time limit, or no limit at all.

monkey1 · February 29, 2012, 08:47

Hi Happy!
Just as a hint: When you get an error and CFX "crashes" or stops prematurely without finishing the run correctly you will NEVER get a .res file, only an .out file.
The .res File can only be produced if CFX finishes successfully. If it doesn't you will be provided with the .out file to be able to retrace the Problem that caused the premature stop!

Zigainer · February 29, 2012, 09:54

And just if you can't change the max wall time then split your job. Let's say do 300 Iterations and get a .res file. Then start your second calculation with the old .res file and continue with another 300 iterations.

happy · March 2, 2012, 01:31

hi again
thanks for your advices. That what I did, the perfect way is spilt my tasks to many depending on the no. of iterations. once again, thanks

beyonder1 · February 27, 2016, 14:07

Quote:

Originally Posted by ghorrocks

The normal mode of operation of PBS is to queue jobs based on the availability of CPUs. This is fine for unlicensed software, but for licensed software often the limitation is how many licenses you have. To make PBS also queue based on license availability we had to add a custom routine to PBS. From memory it checks the license availability on the license server every minute or so and picks up available licenses that way. We talked to PBS support to help develop this. But I remain a little surprise about this because I would have thought there would be lots of customers out there who are license limited so there should be strong demand for this.

I am having the same problem of license unavailability for Ansys CFX. I want my job on queue till the license gets available. you have talked about this issue. Is there some part of code that I can add in my script to resolve this. Please help me out.

ghorrocks · February 28, 2016, 05:54

Are using PBS?

beyonder1 · February 28, 2016, 11:35

Yes, I am using PBS. I am not able to figure out this part.

ghorrocks · February 28, 2016, 18:18

Quoting my previous post:

Quote:

We talked to PBS support to help develop this. But I remain a little surprise about this because I would have thought there would be lots of customers out there who are license limited so there should be strong demand for this.

February 23, 2012, 04:57	help to run CFX simulation via HPC, please	#1
happy Senior Member Safia Join Date: Oct 2010 Location: Australia Posts: 161 Rep Power: 16	Hi anyone,,, I'm now trying to conduct my simulation faster via HPC at my Uni. the probles is that my script file give me the results in .out file not .res file. Actually, I took the csript file from website and I have no idea how I can make my outlet as .res file. Here you are my script file. #!/bin/bash -l # ##### Job Name ##### #PBS -N 3cm ##### Select resources #### #PBS -l nodes=1pn=4 #PBS -l walltime=01:00:00 ##### Goto the directory from which you submitted the job cd $PBS_O_WORKDIR ##### Request that regular output (stdout) and terminal output (stderr) go to the same file PBS -o /home/eng/0000000/Gap_width=3cm.out #PBS -wd ##### Mail Options ##### #PBS -m abe #PBS -M 0000000@gmail.000.edu.au ##### Queue ##### #PBS -q standard ##### Add ansys_module module load ansys ##### Specifies the version of ANSYS CFX version=3d # The following line would start the simulation from a # previous results file, Gap_width=3cm_001.res cfx5solve -double -def Gap_width=3cm.def -initial Gap_width=3cm_001.res -part 4 $nprocs -par-local I appretiate any advise.

February 23, 2012, 05:45		#2
Lance Senior Member Lance Join Date: Mar 2009 Posts: 669 Rep Power: 22	Did you get the script from a random webpage or was it given to you by the IT-guys and girls at your uni? Anyway, they should be able to help you with your batch script. Are you sure you are supposed to use PBS? Also, this is a bit odd: ##### Specifies the version of ANSYS CFX version=3d I doubt you need to specify 3D, as CFX always solve in 3D (unlike Fluent, which can be 2D) happy likes this.

February 23, 2012, 20:12	thanks for your input and further question& help me	#4
happy Senior Member Safia Join Date: Oct 2010 Location: Australia Posts: 161 Rep Power: 16	Thanks guys For Glenn, yes,, U R right. Our ICT devition do not want to help me to write my script file because as I understand they feel it is not their job and they get mad quickly. that is why I'm email you to help me. I got this error however It produce my out file but it also give the messgae: Capability CFX_SOLVER_BASIC_MP_LIMIT does not exist in the ANSYS licensing pool. None of the products enabling this capability are available in the specified license path: ANSYSLI_SERVERS: 2325@139.86.160.180 FLEXlm Servers: 1055@139.86.160.180 =>> PBS: job killed: walltime 92 exceeded limit 50 A fatal error has occurred in cfx5solve: cfx5solve was killed by the user.For general information, the ansys should be work in batch mode.

February 25, 2012, 02:35	my reply	#8
happy Senior Member Safia Join Date: Oct 2010 Location: Australia Posts: 161 Rep Power: 16	Hi all I would like to thank Lance for his/her input and would like to tell Glenn that my simulation now are running by PBS in perfect way

February 25, 2012, 23:05	need more information	#10
happy Senior Member Safia Join Date: Oct 2010 Location: Australia Posts: 161 Rep Power: 16	Hi Glenn,, please could you please give me more explaination for 'based on license availability rather than just CPU availability'. I mean which sort of mode can work based on the CPU availability? Actually, I did not have HPC license. Best Regards

February 23, 2012, 07:18		#3
ghorrocks Super Moderator Glenn Horrocks Join Date: Mar 2009 Location: Sydney, Australia Posts: 17,870 Rep Power: 144	CFX has a demo script file for running CFX under PBS in the documentation and the script file is at <CFX_ROOT>/extras/qcfx5solve.pl

February 24, 2012, 07:15		#7
ghorrocks Super Moderator Glenn Horrocks Join Date: Mar 2009 Location: Sydney, Australia Posts: 17,870 Rep Power: 144	Also, do you need to use PBS? If you are not familiar with batch scheduling software it is just making things hard for you. You can run the solver from the command line, and you can queue up a bunch of jobs if you like. If that is all you want then forget about PBS. But if you need the power of a batch scheduling system or the HPC submits all jobs with PBS and you cannot go with the command line then you better stick with it.

February 25, 2012, 06:54		#9
ghorrocks Super Moderator Glenn Horrocks Join Date: Mar 2009 Location: Sydney, Australia Posts: 17,870 Rep Power: 144	Good to hear. I have wondered why CFX includes such a complex PBS script when much simpler scripts (like the one you have) work. PBS is a bit tricky to set up, especially if you want it to queue based on license availability rather than just CPU availability.

February 26, 2012, 05:13		#11
ghorrocks Super Moderator Glenn Horrocks Join Date: Mar 2009 Location: Sydney, Australia Posts: 17,870 Rep Power: 144	The normal mode of operation of PBS is to queue jobs based on the availability of CPUs. This is fine for unlicensed software, but for licensed software often the limitation is how many licenses you have. To make PBS also queue based on license availability we had to add a custom routine to PBS. From memory it checks the license availability on the license server every minute or so and picks up available licenses that way. We talked to PBS support to help develop this. But I remain a little surprise about this because I would have thought there would be lots of customers out there who are license limited so there should be strong demand for this.

February 26, 2012, 20:59	error with the time	#12
happy Senior Member Safia Join Date: Oct 2010 Location: Australia Posts: 161 Rep Power: 16	Thanks Glenn for your input. I read your thesis and you looks ANSYS-CFX exprtise. First, could you please tell me the cause of this error when I conduct my running via HPC(PBS). PBS: job killed: walltime 43238 exceeded limit 43200 However, when I reduce the no. of iteration from 1000 to 300, the .res file produced. I tried to set the walltime but without any benefit. does this error come from the limitation mem on my home directory???By the way, I did not set the required memory for my job into my script file. I appreciate any input. Regards

February 27, 2012, 00:05		#13
ghorrocks Super Moderator Glenn Horrocks Join Date: Mar 2009 Location: Sydney, Australia Posts: 17,870 Rep Power: 144	Firstly, PBS is not part of CFX. PBS is a third party commercial software from Altair http://www.pbsworks.com/. If you want help with PBS you are best to ask your Altair support person. But to my reading your error message is pretty obvious - you have set a maximum wall time limit and that was exceeded so it stopped the job. If you want to run longer you need to define a longer wall time limit, or no limit at all.

February 29, 2012, 08:47		#14
monkey1 Senior Member Join Date: Jul 2011 Location: Berlin, Germany Posts: 173 Rep Power: 15	Hi Happy! Just as a hint: When you get an error and CFX "crashes" or stops prematurely without finishing the run correctly you will NEVER get a .res file, only an .out file. The .res File can only be produced if CFX finishes successfully. If it doesn't you will be provided with the .out file to be able to retrace the Problem that caused the premature stop!

February 29, 2012, 09:54		#15
Zigainer Senior Member Join Date: May 2011 Location: Germany Posts: 130 Rep Power: 15	And just if you can't change the max wall time then split your job. Let's say do 300 Iterations and get a .res file. Then start your second calculation with the old .res file and continue with another 300 iterations.

March 2, 2012, 01:31	thanks	#16
happy Senior Member Safia Join Date: Oct 2010 Location: Australia Posts: 161 Rep Power: 16	hi again thanks for your advices. That what I did, the perfect way is spilt my tasks to many depending on the no. of iterations. once again, thanks

February 28, 2016, 05:54		#18
ghorrocks Super Moderator Glenn Horrocks Join Date: Mar 2009 Location: Sydney, Australia Posts: 17,870 Rep Power: 144	Are using PBS?

February 28, 2016, 11:35		#19
beyonder1 Member beyonder Join Date: May 2015 Posts: 67 Rep Power: 11	Yes, I am using PBS. I am not able to figure out this part.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
ERROR #001100279 - CFX 13 - transient run	Zigainer	CFX	14	December 9, 2011 14:22
CFX thermo-fluid 2D simulation problems	maryliz	CFX	6	November 1, 2011 00:26
nucleate boiling simulation in CFX	Anil	CFX	3	August 25, 2010 15:18
mesh from Workbench Simulation to Cfx Pre?	Dai	CFX	6	November 11, 2006 13:19
Procedure to run unsteady simulation?	STN	Main CFD Forum	2	February 16, 2002 05:37