Securely administer multiple server via SSH

The basic question answered by baSSHtion.org is:

How can an operator/administrator securely manage backend server through the internet?

Often machines running in private nets need to be administered from the internet. The hypothetical example used here is a moderate complex application running on the internet. This application has a webserver facing to the internet, a database server, and an application server as backend systems. These machines are administered via SSH. For security reasons it is undesirable to expose the backend systems to the internet.

SSHv1 is completely out of scope, because is has severe security issues. Only SSHv2 options are considered. The SSH implementation used in the exapmles is OpenSSH, YMMV for other implementations.

Setup used for examples.

Managing such a setup relies on (at least) two roles: Application Operators (short: operator) that manage the applications on the backend systems (but do not manage the operating system), and System Administrators (short: administrator) that manage the operating system (including user management, and SSH). System administrators have access to the role of root, whereas operators don’t have this role. Distinguishing between these roles allows for a better reasoning in terms of access control, even if these roles are often carried out by the same person.

Several solutions to this problem statement are dicussed here.

Basic Requirements

These are the basic requirements that each of the solutions is evaluated against:

  • REQ 1: From the internet to the backend An operator can access the backend system from the internet (via ssh).
  • REQ 2: Secured to the end The whole connection from the operators system (internet) to the backend system must be secured via SSH.
  • REQ 3: no escape for operators Operators must not be able to break out of the system (whereas system administrators might be able to do so). What breaking out means depends on the solution but it basically means, that the operator has to follow the rules set by the system administrator.
  • REQ 4: Bastion Host Only one host shout be exposed to the internet (the bastion host). This is the core means to prevent any host on the internet to directly access the backend systems.

Solutions

There exist several possibilities to allow operators to access the backend systems. Some of the more popular solutions are:

  1. Port DNAT, where a host exposed to the internet DNATs some of its ports to the backed systems. E.g. public-ip:2224 is forwarded to db-server:22, and public-ip:2222 is forwarded to app-server:22. To access the DB server the operator points ssh to the public ip, e.g. ssh public-ip -p2224.
  2. A built in solutions is the ProxyCommand to provide the connection to the backend system (e.g. ssh public-ip -o ProxyCommand "nc app-server 22")
  3. Jump Host with interactive session, where a host exposed to the internet provides SSH access with an interactive terminal. Operators connect to this host, and then manually open a new SSH connection to the next host. Authentication on the backend hosts is (often) done via agent forwarding to prevent storing secret keys on the exposed machine. E.g. ssh -A public-ip and there ssh db-server.
  4. Jump Host without interactive session, as above but without an interactive session (shell) on the bastion host. When system administrators connect to this host a new SSH connection to the next host is automatically opened via a forced command. Authentication on the backend hosts is done via agent forwarding. E.g. ssh -A public-ip app-server.
  5. Other solutions

All of these solutions 1-4 fulfil the requirement of a single exposed host, though in practice additional requirements are identified.

Extended Requirements

When the system gets larger, and/or handles sensitive data, additional requiremens emerge:

  • REQ 5: Fine grained permissions Certain options should be configurable on the bastion host on a per user and system base (e.g. allow scp, sftp, or port forwarding for user X to system Y).
  • REQ 6: Robustness Configuration and management should be robust. It should be easy for the administrator to execute a given task (e.g. create new user, add a new backend system) but difficult to bring the system in an inconsitent state.
  • REQ 7: Four eyes principle for usermanagement It should be impossible for the administrator of a single system to grant a new operator/administrator access from the internet.
  • REQ 8: Port forwarding from the client The operator should be able to use SSH port forwarding from his workstation to the backend systems.
  • REQ 9: scp to the backend The operator should be able to use scp from his workstation to the backend systems.
  • REQ 10: Auditable All communication must be monitored in such a way that a forensic analysis is possible at a later time (audit)

Wording

  • Administrator : The role responsible for managing the access control system (and the operating system). Can add/delete users on all systems. Has root access to all machines.
  • Operator : A user responsible for managing the application. Has accounts on the backend systems, but not root. An operator might have accounts on the bastion host, but never with root level access.
  • SSH : The OpenSSH tools (client/server) or protocol. SSHv1 is not considered.
  • ssh : The OpenSSH SSH client.

Ressources