[-Maxim-]

Hello,

I hope this is the right section of the forum, if not, please move my post. Thanks.

I am trying to set up Ansys RSM with ARC (Ansys Remote Cluster) on my little linux cluster running CentOS and Ansys 2022 R2. The cluster consists of a master with 2 nodes (16 cores each), connected via infiniband.

So far I have copied over my .def files, started the calculation via putty from my Windows workstation and copied the results back. Now I would like to use the optimization features of Workbench and for that I need to set up RSM to send the jobs to the cluster from the Workbench.

What I've done so far is the following (all as root on master):
- ./rsmclusterconfig to create service scripts rsmlauncher, arcmaster and arcnode
- ./rsmconfig -launcher --> successfully installed and launched --> RSM Launcher Service is running
- ./arcmaster start --> succesful
- ./arcnode start --> successful

Then I launched the RSM config tool on my windows workstation, defined name of cluster, staging directory on network share and on cluster, stored user credentials, imported and refreshed the queues.

I have two ARC queues now: Default and local which would start the test job on the master. When I test either of them now, the test runs now successful. (*edited* - no idea why it works now...)

However, I am not able to set up my compute nodes in ./arcconfigui unfortunately. When I want to add the node01 (that's the name of my node), it states the following in putty:

/usr/ansys_inc/v222/RSM/ARC/tools/linx64/arcdeploy -f "/root/.ansys/v222/deploy.csv" -nomaster
You entered the following 3 command line arguments:
-f
/root/.ansys/v222/deploy.csv
-nomaster
Starting ARC service on node01
I am running the following commands
ssh -tt node01 "sudo -E /usr/ansys_inc/v222/RSM/ARC/tools/linx64/install_daemon -arcnode"
Installing Ansys RSM Cluster Node service
arcnode222 installed.
Please use /etc/init.d/arcnode222 {start|stop|restart|status|try-restart|condrestart} to monitor arcnode222 service
Setting environment variables from '/usr/ansys_inc/v222/RSM/ARC/tools/linx64/arc_env_profile'
Ansys RSM Cluster Node Service is not running
Setting environment variables from '/usr/ansys_inc/v222/RSM/ARC/tools/linx64/arc_env_profile'
Starting Ansys RSM Cluster Node Service ...
Starting /usr/ansys_inc/v222/Framework/bin/Linux64/runwb2 -cmd mono /usr/ansys_inc/v222/RSM/ARC/bin/ArcNodeService.exe -noshutdown...
Ansys RSM Cluster Node Service start failed.  Please check the Logfile for Errors
mkdir: cannot create directory ???/home/rsmadmin???: Permission denied
 This process is not being run as an Administrator.
An Unhandled Exception has occurred. Access to the path '/home/rsmadmin/.ansys/v222/ARC' is denied.

I am running that as root, therefore I should be Administrator. The user "rsmadmin" seems to be auto-created (/home/rsmadmin exists). There is no firewall active.


Any pointers would be helpful. Thank you for your help!
-Maxim

[evcelica]

Have you contacted ANSYS support? They have been very helpful in the past for issues like this.

[-Maxim-]

Thank you evcelica for your answer.

I have contacted my local Ansys guy about setting up RSM and he said that this will probably take a day (incl. setting up PBS queueing and potentially updating the OS) and gets billed separately and not within our normal support contract. That's why I wanted to see if I can set it up myself with ARC and test if that's enough for my purposes.
I will probably just contact him again with that specific question.

[-Maxim-]

quick update:
with the help of the Ansys support we figured out that ARC created the user rsmadmin on the master and the nodes separately so that those users had separate user IDs. Therefore, the rsmadmin user from the master didn't have rights to access the home dir of the rsmadmin on the node.
We fixed that by creating the rsmadmin user manually with a fixed user ID on both the master and the nodes.


BUT: I still can't deploy the ARC node service from the arcconfig to the cluster:

  Database Directory is: /home/rsmadmin/.ansys/v222/ARC
  The UpdateMasterLoad method failed to update execution node load table to master: node01.cluster:13222.
  Please check that the master service is started and that there is no firewall blocking access on ports 11222, 12222, or $

There is no firewall active on the nodes.



Ansys support is out of ideas. Their suggestion was now to reinstall the whole server to a newer CentOS, newer Infiniband drivers, etc.
Maybe you guys have another idea first? Thanks.

[-Maxim-]

quick update:
with the help of the Ansys support we figured out that ARC created the user rsmadmin on the master and the nodes separately so that those users had separate user IDs. Therefore, the rsmadmin user from the master didn't have rights to access the home dir of the rsmadmin on the node.
We fixed that by creating the rsmadmin user manually with a fixed user ID on both the master and the nodes.


BUT: I still can't deploy the ARC node service from the arcconfig to the nodes:

  Database Directory is: /home/rsmadmin/.ansys/v222/ARC
  The UpdateMasterLoad method failed to update execution node load table to master: node01.cluster:13222.
  Please check that the master service is started and that there is no firewall blocking access on ports 11222, 12222, or $

There is no firewall active on the cluster.



Ansys support is out of ideas. Their suggestion was now to reinstall the whole server to a newer CentOS, newer Infiniband drivers, etc.
Maybe you guys have another idea first? Thanks.

[ams_sharif]

Hi,
I know I am 6 months late, but I have just stumbled into the same error
The fix for me was quite easy though


/ansys_inc/v*/RSM/ARC/tools/linx64/install_daemon -arcmaster

[-Maxim-]

Originally Posted by ams_sharif

Hi,
I know I am 6 months late, but I have just stumbled into the same error
The fix for me was quite easy though

/ansys_inc/v*/RSM/ARC/tools/linx64/install_daemon -arcmaster

Hi ams_sharif,
thanks for taking the time to sign up and responding to my post. However, I am not quite sure if that will fix my problem too. The arcmaster daemon is already installed on my HPC head/master node and sending jobs to the master works well. The problem is on the calculation nodes where I should only install the arcnode daemon.


Do you also have a HPC cluster setup and can send jobs to each individual node or do you only send jobs to the head/master?

[ams_sharif]

Oh I see what you are trying to do. I'm not sure if this is possible. As far as I'm aware, every node should have one master only.
I submit all my jobs through Slurm to the master node and it takes care of the process. The PCs I use are a bit old so I allocate all of the processors to a single job and the remaining jobs would be queued.

Have you tried running arcconfigui on a 2nd node? Choose it to be a master, add the nodes and see if both configs keep the nodes under them. If yes then you are good to go. You would just need a job handler/scheduler to handle your submitted jobs among both configs.
I'm currently away from the cluster. I might try it out next week and brief you with my findings.