SSH bastion/jump host

Introduction

It is common sense on medium-to-large IT departments that you should have as few as possible network devices exposed to the web. But modern organizations need to offer numerous e-services to the public. They also need to be manageable by people working from home (at least during COVID19 times).

The first acronym that comes to everyone’s mind as a solution to the problem is VPN (Virtual Private Network). It is basicaly one service, it listens to one port and its job is to authenticate/authorize your device and make it a member of a virtual private network that has access to the real private organization’s network.

But you need to always have in mind that VPN is yet another “public” network. Maybe less public than the whole internet, but every device/person with a valid username/password or a key is a trusted member. As long as you sit in your private network, VPN is -and should be treated like- a public network.

Bastion host

Another solution is "Bastion host”, a computer that lives on the public side of Demilitarized Zone (outside of your firewall). It is also known as “Jump box” or “Jump host” and it has the following key characteristics by definition:

1. It is properly configured to resist attacks coming from the public network
2. It contains no private information so that even if compromised, the attacker will find no useful information to access the private network
3. It has access to strictly defined internal services (by organization’s firewall)
4. The private network administrator can use it but has not superuser privileges on it

The scope of this article is to describe how we can implement all of the above 4 elements on an SSH bastion host and use it to access our private servers via SSH. And we will do this without complicating the day-to-day experience of administrators. So lets add a number...

5. Users will access internal servers as effortless as from internal network (or VPN)

SSH Jump Box

We will use a CentOS Linux release 8.2.2004 running OpenSSH_8.0p1 as our bastion host. As we said, we need to make it more resistant than the default configuration. There are unlimited ways to do that. Lets use only two of them:

1. Completely disable password authentication and use only SSH keys
2. Configure openssh to listen to a random high port (eg. 22222) and relieve our server from 99.999% of the dump bots out there, scanning and probing known ports

In order to generate and use SSH key pair, take a look on Using SSH Effectively.

As for implementation, edit /etc/ssh/sshd_config modifying the following parameters:

  /etc/ssh/sshd_config

Port 22222
passwordAuthentication no
ChallengeResponseAuthentication no
UsePAM no

...and run systemctl reload sshd.service to reload the service.

If you have SELinux on enforcing mode (of course you have!! - this will be an internet facing server) you also need to tell SELinux about the port change:

[root@jumpbox ~]# semanage port -a -t ssh_port_t -p tcp 22222

SSH Private Servers

We now need to decide which of our private servers need to be managed from outside the organization and allow jump box to access them. Lets say that priv-server1.example.com is one of them. We will run some self-explanatory firewall-cmd commands:

[priv-server1]# firewall-cmd --new-zone=SSH_access —permanent
[priv-server1]# firewall-cmd —reload
[priv-server1]# firewall-cmd --zone=--add-service=SSH_access —permanent
[priv-server1]# firewall-cmd —zone=SSH-access --add-source=XXX.XXX.XXX.XXX/XX —permanent
[priv-server1]# firewall-cmd —reload

Now our bastion host, that has the private IP XXX.XXX.XXX.XXX/XX, is allowed to access the selected private servers.

SSH Agent Forwarding

On our local client machine (at home), we can execute SSH agent and load our private key:

[admin-pc]$ eval $(ssh-agent)
[admin-pc]$ ssh-add

Then we can access jump box, forwarding SSH agent:

[admin-pc]$ ssh -A jbuser@jumpbox.example.com

Everything works as expected, without the need of any password.

But, before continue, lets check our environment variables on jump box running:

[jumpbox]$ env | grep -i ssh_auth
SSH_AUTH_SOCK=/tmp/ssh-iOyEN3vXnp/agent.80756

The existence of this socket on the bastion host is problematic.

Lets check the manual:

[admin-pc] $ man ssh

…

…

So this contradicts to one of the key elements of a Bastion host:

It contains no private information so that even if compromised, the attacker will find no useful information to access the private network

To address that, add another configuration parameter to disable Agent forwarding on bastion SSH server:

  /etc/ssh/sshd_config

AllowAgentForwarding no

...and run systemctl reload sshd.service to reload the service.

Use ProxyJump parameter

Now lets try the command:

[admin-pc]$ ssh -A -o ‘ProxyJump=jbuser@jumpbox.example.com:22222' admin@priv-server1.example.com

You can check for yourself that SSH agent is forwarded to the end server "priv-server1.example.com" and there is no socket on the jumpbox (you can check its /tmp from another session).

Making our lives easier

On your client machine create the following file:

  ~/ssh/config

Host jumpbox.example.com
    HostName jumpbox.example.com
    User jbuser
    Port 22222
Host *.example.com !jumpbox.example.com
    HostName %h
    ProxyJump jumpbox.example.com

Comments on the client's config:

• We are using wildcard "*" to include all the servers of the specific organization instead of listing them one-by-one
• We need to exclude jumpbox from the other *.example.com servers (!jumpbox.example.com). If not, the client will try to use ProxyJump to access the jumpbox itself and will crash looping
• We need to protect ~/.ssh/config file for the owner only to have read permission (600)

Now every ssh command from your local machine simplifies like this:

[admin-pc]$ ssh -A admin@priv-server1.example.com

In other words, you work from home as if you were in your office with respect to all the security measures.