[mighelone]

Hello to everybody!

I'm testing the new release of openFoam (1.4.1) with the openmpi libraries.

Starting a job with mpirun, I've obtain this error:

michele@enercluster:~$ mpirun -hostfile machines -np 3 a.out
bash: orted: command not found
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1164
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90
[enercluster:06382] ERROR: A daemon on node 192.168.0.2 failed to start as expected.
[enercluster:06382] ERROR: There may be more information available from
[enercluster:06382] ERROR: the remote shell (see above).
[enercluster:06382] ERROR: The daemon exited unexpectedly with status 127.
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 188
[enercluster:06382] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1196
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons for this job. Returned value Timeout instead of ORTE_SUCCESS.

If I run the process only in the local machine it works!
I suppose is that a problem of PATH on the remote machines, infact executing this command:

michele@enercluster:~$ ssh node2 printenv |grep PATH
PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games

It means that when I log using ssh the openmpi are not correctly configured.

But if I log in the remote node using ssh:

# ssh node2
# echo $PATH
/opt/Fluent.Inc/bin:/home/michele/OpenFOAM/linux64/paraview-2.4.4/bin:/home/mich ele/OpenFOAM/OpenFOAM-1.4.1/src/openmpi-1.2.3/platforms/linux64GccDPOpt/bin:/hom e/michele/OpenFOAM/OpenFOAM-1.4.1/src/mico-2.3.12/platforms/linux64GccDPOpt/bin: /home/michele/OpenFOAM/linux64/j2sdk1.4.2_05/bin:/home/michele/OpenFOAM/linux64/ gcc-4.2.1/bin:/home/michele/OpenFOAM/michele-1.4.1/applications/bin/linux64GccDP Opt:/home/michele/OpenFOAM/OpenFOAM-1.4.1/applications/bin/linux64GccDPOpt:/home /michele/OpenFOAM/OpenFOAM-1.4.1/wmake:/home/michele/OpenFOAM/OpenFOAM-1.4.1/bin :/usr/local/bin:/usr/bin:/bin:/usr/games

Since it is a problem of the PATH environment variable with non-interactive ssh session.
The OpenFoam path is defined in .bashrc

Any ideas?

Thank you
Michele

[gtg627e]

Michele,

check out this post:

http://www.cfd-online.com/OpenFOAM_D...es/1/5473.html

I ran into a similar problem a while back.

Alessandro

[mighelone]

Hi Alessandro,

Thank you for your answer, but I've already read your post, but I suppose that my problem is different.

OpenFoam is correctly installed in every nodes (I try to run the case not in parallel in some nodes, without problem).

I guess the problem is related to the ssh non-interactive login, when I execute mpirun.
Infact during the non-interactive ssh login, my .bashrc settings file is not read, so my PATH doesn't include the PATH for mpi and OpenFoam.

I guess also that is a problem of openssh package distributed by debian, because working with other distro the .bashrc is correctly read ( I test using "ssh HOST printenv").


Michele

[olesen]

Hi Michele,

There are a few problems that you might be experiencing.

Open-MPI needs to find its own binary path in order to boot the orted. OpenFOAM sets the variable 'OPENMPI_ARCH_PATH' to point to the openmpi installation. You can use that to help openmpi find itself. Calling mpirun with either of these should get the orte to boot:

a) $OPENMPI_ARCH_PATH/bin/mpirun ...
b) mpriun -prefix $OPENMPI_ARCH_PATH

If you try 'mpirun -help' and get the message the it can't find anything, try adding the following:

$ export OPAL_PREFIX=$OPENMPI_ARCH_PATH


Now that openmpi can boot, the remaining problem is getting the OpenFOAM environment set on the remote nodes. The nicest solution (thanks Henry) is to use foamExec to wrap the call to your application.

Attached is a modified version of foamExec in which the version is optional.

[mighelone]

Hi Mark,

I'm trying to run a very simple parallel application (hello world!) with openmpi. If I run without -prefix option:

mpirun --hostfile machines -np 4 ./a.out

I obtain the previous error:
bash: orted: command not found
mpirun: killing job...

If I run with the option -prefix:

mpirun -prefix $OPENMPI_ARCH_PATH --hostfile machines -np 4 ./a.out

I don't receive any answer from my application.

furthermore I try to install the openmpi from my distro, installaed in canonical path (/usr/bin e /usr/lib), and the same program run without problem.

Now I will try the foamExec with native openmpi lib!

Thank you Michele

[olesen]

It might be that orte is working, but takes a long time to boot. Check if the orte is running on any of the remote nodes. I find the following alias quite useful:

alias psf='/bin/ps -e f -o user,pid,ppid,pgrp,command'


Increasing the mpirun verbosity and/or the debug-daemons might help figure out what is happening (see mpirun -help), but I haven't done this sort of thing for a long time.

Also try with an absolute path to your a.out, in case the working directory is somehow getting lost across the nodes.

[mighelone]

Nothing to due!

I f a log in the remote machine I don't have any process related to mpi.

Also giving the absolute path of a.out I don't receive any message!

I've found that the PATH during a non-interactive PATH, it is possible to set the environment variables in the file /etc/environment or in $HOME/.ssh/environment

The problem is that OpenFOAM has too many variable to set, and I don't found any way to execute the OpenFOAM script during the interactive login.

[olesen]

The problem is that OpenFOAM has too many variable to set, and I don't found any way to execute the OpenFOAM script during the interactive login.

The problem is setting *any* variables, not how many variables there are.

This is what foamExec helps you do. It provides a simple wrapper to source the requisite bashrc before executign whatever command.

In fact, we only set the FOAM environment as required and don't put it in my ~/.profile or ~/.bashrc.

Thus if I ssh to a remote host and check the OpenFOAM variables:

Eg,
$ ssh HOST printenv | grep WM_

shows nothing

but if I use foamExec:

$ ssh HOST /path/to/foamExec printenv | grep WM_

then I see about 18 OpenFOAM variables.
Does this not work for you?

[mighelone]

Hi Mark!

At the moment I don't try foamExec, but I solve the problem in this way:

# printenv > .ssh/environment

and I copied this file in all remote nodes.

In this way I define all my environment variables in the file .ssh/environment, that is read by ssh during non-interactive login.

Anyway the use of foamExec is very interesting in order to have a clean system without the several variables used by OpenFOAM.

I suppose that using foamExec I have anyway to set the variable related with openmpi in .ssh/environment, like:

OPENMPI_HOME=/home/michele/OpenFOAM/OpenFOAM-1.4.1/src/openmpi-1.2.3
OPENMPI_VERSION=1.2.3


since without these variables mpirun is not able to find the command orte in the remote machine.

Thank you
Michele

[mighelone]

Hi Mark!

I'm trying to use foamExec, after removing any references in .bashrc and .ssh/environment about OpenFOAM (if I well understood foamExec is able to set all the foam variables, before running any applications), but I obtain the following error:

Error : bashrc file could not be found for OpenFOAM-1.4.1

That I suppose is related to the fact that the script do not found the bashrc file of OpenFOAM.

How can I solve this inconvenient?

Michele

[olesen]

Error : bashrc file could not be found for OpenFOAM-1.4.1
That I suppose is related to the fact that the script do not found the bashrc file of OpenFOAM.

I assume
1) that you are using the foamExec that I posted
2) that you copied foamExec to $WM_PROJECT_DIR/bin/
3) that $WM_PROJECT_DIR/.OpenFOAM-$WM_PROJECT_VERSION/ exists and that $WM_PROJECT_DIR/.OpenFOAM-$WM_PROJECT_VERSION/bashrc exists
4) that you are using the fully qualified path to foamExec -OR- you are using the -v option.

Check what foamExec is doing:

$ /bin/sh -x /path/to/foamExec /bin/true
OR
$ /bin/sh foamExec -v 1.4.1 /bin/true

[mighelone]

Hi Mark!

1) I'm executing your script
2) the script is not in the $WM_PROJECT_DIR/bin/ but in another place
3) These files and directories exist
4) I'm using the fully qualified path, but in the wrong place

Now I'm not at work, so I can not try your advices, Monday I will try again!

Thank you again for your help

Michele

[cricke]

Hi Mark and Michelle!
I am following your thread but get following message when trying to execute the following command

Command:

foamExec -v OpenFOAM-1.4.1 $OPENMPI_ARCH_PATH/bin/mpirun --hostfile machines -np 4 simpleFOAM $HOME VAFAB_multi -paralell

Message:

foamExec: access denied

1) I'm executing your script
2) the script is in the $WM_PROJECT_DIR/bin/
3) These files and directories exist
4) I'm using the -v option

When using the '$OPENMPI_ARCH_PATH/bin/mpirun ...' as you suggested the mpirun and orte is started on the host machine but not on the slave.

What am I doing wrong?

Regards

Christofer Ivarsson

[olesen]

Message:
foamExec: access denied

What about the next point?
5) Is foamExec readable and executable by the users (eg, chmod 0755).

[cricke]

Hi and thanks for your instant reply!


The file is perfectly readable and saved as an application-file. This is what happends when trying to execute foamExec only

a403518@ENEPCST75:~$ foamExec
bash:home/a403518/OpenFOAM/OpenFOAM-1.4.1/bin/foamExec: Access denied

Obviously its all about environment variables, right? I get the same result even with no extra environment variables added in the .bashrc

Shall I add som kind of path in the .bashrc to point out the foamExec?

/Chris

[olesen]

Hi Christofer,

With "Access Denied", I still suspect some sort of file permissions problem. Check that the rest of your OpenFOAM installation is readable.
(eg, chmod -v -R a+rX ...). Note use large 'X' and not small 'x' in the chmod command.

Recheck what foamExec is doing:
$ /bin/sh -x /path/to/foamExec /bin/true
OR
$ /bin/sh foamExec -v 1.4.1 /bin/true


When I issue foamExec without any arguments, I get the message "no application specified" and then the usage as per the -help option.

[cricke]

ok, sorry for me being a nut head but do you want me to write those commands in the terminal because that is what I did and here is the results.

a403518@ENEPCST75:~$ /bin/sh -x /patch/to/foamExec /bin/true
/bin/sh: Can't open /patch/to/foamExec
a403518@ENEPCST75:~$ bin/sh foamExec -v 1.4.1 /bin/true
bash: bin/sh: File does not exist

I have no restricted access to my OpenFOAM directory and OpenFOAM runs smoothly. Do you still think its a permission problem? When I run the chmod command it suddenly changed the permission to my OpenFOAM-directory to locked so I had to change that back again.

/Chris

[olesen]

There was a typo there:

$ /bin/sh -x /the/path/to/foam/bin/on/your/machine/foamExec

I don't know the path to foam on your machine.

[cricke]

Thanks for you being patient with me. Following happends

Recheking what foamExec is doing when:

a403518@ENEPCST75:~$ /bin/sh foamExec -v 1.4.1 $HOME/OpenFOAM/OpenFOAM-1.4.1/bin/true
Executing: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/.bashrc
Executing: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/.OpenFOAM-1.4.1/apps/ensightFoam/bashrc
Executing: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/.OpenFOAM-1.4.1/apps/paraview/bashrc
exec: 141: /home/a403518/OpenFOAM/OpenFOAM-1.4.1/bin/true: not found

OR

403518@ENEPCST75:~$ /bin/sh -x $HOME/OpenFOAM/OpenFOAM-1.4.1/ bin/true
a403518@ENEPCST75:~$

The latter seem to work since it doesnt protest in the command window. However the first command seem to find all env-variables except 'true'. I must tell that I have already added the first three:
.../.bashrc
.../paraview/basrc
.../ensightFOAM/bashrc

in my .bashrc so that they set automatically when opening a terminal window.

So whats up next?
I get contact with my slave node and I enter the password. Then nothing else happends in the command window. The mpirun and orted start but are listed as sleeping processes. The solver never starts.

/Chris

[olesen]

Hi Christofer,

I really thought that absolutely every Unix machine had '/bin/true', but if not you can always test with a command such as 'hostname' which normally lives under '/bin' but if your system is really strange it might be under '/usr/bin' or I don't know where else.

You *do* however need to setup ssh to connect without passwords, otherwise you can forget about the rest. There must be something about this in the open-mpi FAQ. If not, check the mpich FAQ.

/mark