running atk with mpich2 over multiple machines with ssh

[esp]

hello,

I can run atk using mpich with multiple processes, but not over multiple machines ... the other machines can only be accessed via ssh with password

There is a line in the mpich doc that says "
    ssh othermachine date
If you cannot get this to work without entering a password, you will need to configure ssh or rsh so that this can be done. "

Any help on how to do this? I have no idea how to configure ssh

[Nordland]

The way I personally do it, is to generate ssh key without a password. Add this key to your accepted keys on your machines, and then there is no prompt for password anymore.

[esp]

Ok thank you I will try this (send to IT people who understand) ..


Ed

[esp]

Ok so the IT people said this would be too unsecure .. cant do this ...

but i found if you install mpich2 with the compilation option --with-pm=smpd
then you can use some extensions which allow you to set the user and password ...

I run like this:
$MPICH2_BIN_DIR/mpiexec -hosts 4 <host1> 2 <host2> 2 -pwdfile pwd.txt -phrase <password> -p 3396 $ATK_BIN_DIR/atkpython $SRC_DIR/$SCRIPT2 > $LOG_DIR/$SCRIPT2.log.txt

first it said i needed to provide a password so i added the -phrase above ... now it says:

op_read error on left context: Error = -1
unable to read the challenge string, Error = -1


Any ideas here?

[esp]

I posted this question on an mpich forum and those users said (as you said) that you have to setup ssh without a password ... the IT people at my university said this cannot be done ... so this means that either I cannot use parallelization, or I am stuck with running separate jobs on multiple machines .... this is not really a great solution ... any ideas?

[Nordland]

Your machine is that a Windows or Linux? And the machines you are trying login into?

[Nordland]

If you are under linux, you can do this evil trick to trick your evil admins:

Start up the terminal, go to .ssh and write
ssh-add your_password_key

Then it will remember that password for length of this terminal session. You can then fire of ATK and it should not prompt for password.


http://linux.die.net/man/1/ssh-add

[esp]

oooh that sounds good ... thank you i will try it

[Anders Blom]

Just as a general note, ATK does not support the "smpd" process manager. It may appear to work (it runs), but if you look carefully each node will actually act as master, so you will not get any parallelization + problems with I/O since all nodes will attempt to write the NetCDF files at the same time. It will be obvious that something is wrong from the log file too, since all statements will appear multiple times.

[esp]

thank you.  I only tried that because i saw there were options to supply a password, but it didnt work anyway ... thank you. 

[esp]

followed this tutorial and i am able to ssh without passwords .. might be help to others

http://www.mtu.net/~engstrom/ssh-agent.php

ed

[esp]

ok well i have tried and tried, it seems i cannot figure out how to get mpich working across multiple machines .. it works fine for 1 machine and multiple threads .. but over mult machines ... not so good ... if you guys ever provide a detailed guide that would be very helpful ..

[Anders Blom]

Is this Linux or Windows? It would help to know more in detail what goes wrong - then maybe we can work out a guide as a result of our discussions in this thread.

I take it you have gone through the basic stuff, installing MPICH2 and ATK on the 2 machines, and set up MPD?

[esp]

Yes I have 4 linux machines with Ubuuntu ... all have atk installed in exactly the same way, same paths, same options .. also they all have 2 installations of mpich, one with the smpd option and one without in different paths ...

all of them work fine runnning atk on their own, even parallelized but on one machine with a mpich -n type run

i just cannot get it to work over multiple machines starting from one job .... one reason is because all the machines have ssh passwords to access... i recently posted about ssh-agent, which i successfully figured out which saves your password for a session ... that solved one problem, but then when i try to run the job i get all sorts of wierd errors, ... not from atk but from mpich i think ... i will try again and post those errors ... my comment is just that in general this is not a simple thing to do and a tutorial would help (linux machines, ssh with passwords, running atk over mult machines)

[Anders Blom]

I agree it's not an entirely trivial thing to do. However, it may be that you have a working MPI environment already. You mention smpd - that's a no-no with ATK. But ok you have another one, that's the only one which will work. Yes, post your error messages, that will be the only way to proceed. Also, you can try to execute something simpler than ATK, you can do simple tests like

Code

mpiexec -n 3 echo $HOSTNAME