Transparent double SSH connections and issuing the same command to multiple machines at once

Goal: Connect directly to any final host you have an account on, using a jumphost and typing your passphrase only once at each reboot of your client machine.

On your client, generate your SSH keys one off. Please choose to encrypt your private key using a passphrase:

client$ ssh-keygen -t dsa -C "$(whoami)@$(hostname)-$(date -I)"
client$ ssh-copy-id username_on_jumphost@jumphost_fully_qualified_name

Now install keychain if you do not have it already and enable it for your local user by adding an alias to your .bashrc:

client$ echo "alias ssh='eval \$(/usr/bin/keychain --eval --agents ssh -Q --quiet ~/.ssh/id_dsa) && ssh'" >>~/.bashrc

Source .bashrc or reopen your terminal to make sure this alias is defined. Next step is to configure ssh for easy connection to our hosts. Create ~/.ssh/config if it does not exist else add to it:

ControlMaster auto
ControlPath /home/YOUR_LOCAL_USERNAME/.ssh/tmp/%h_%p_%r

Host jumphost
  ForwardAgent yes
  Hostname jumphost_fully_qualified_name
  User YOUR_USER_NAME_ON_JUMPHOST

Host ...
  ForwardAgent yes
  User YOUR_USER_NAME_ON_ALL_HOSTS
  ProxyCommand ssh -q jumphost nc -q0 %h 22

Where ... is a blank-separated list of all host names reachable from the jumphost. You can use wildcards, e.g. host*

Next, create a temporary directory to keep track of all connections, so that ssh can reuse a connection to the jumphost multiple times and logins are visibly faster:

client$ mkdir ~/.ssh/tmp

You can now login to every host from your client with one short ssh command, e.g.:

client$ ssh host1

client$ ssh host2

Goal: execute the same command(s) or edit the same file on multiple hosts, but type stuff only once.

You may also find useful to install a utility like Parallel SSH on either your client or the jumphost. Parallel SSH allows you to execute commands on multiple hosts in parallel, using only one command. See this tutorial and remember that on ubuntu pssh is called parallel-ssh. E.g. here is how to get a description of linux distribution installed on multiple hosts:

jumphost$ parallel-ssh -h host_list -i lsb_release -d

Where host_list is just a file containing IPs or host names, one per line. Note we use the option -i rather than -P else output looks messy.

Another interesting utility is clusterssh. You install it locally and it will open multiple terminals. You can edit the same file on multiple hosts: your input goes to all terminals if directed to a small empty control windows. If focus is on a specific terminal, it will only go to that terminal. Of course, it is up to you to make sure these two mechanisms are used correctly:

client$ cssh host1 host2