How to Setup SSH on Linux Server – Complete Guide

To set up SSH on a Linux server, install the OpenSSH server package, start and enable the sshd service, allow the SSH port in your firewall or security group, configure secure settings in /etc/ssh/sshd_config, set up SSH key authentication, and test a new session before disabling passwords or root login.

Secure Shell (SSH) is the standard for remotely managing servers. In this guide, you’ll learn how to setup SSH on Linux server properly, from installation to hardening, using clear steps and best practices we use daily on production systems.

Prerequisites and Quick Overview

Before you begin, make sure you have:

• Root or sudo access to the Linux server.
• Console access via your hosting panel in case SSH is misconfigured.
• Ability to open ports in the server firewall and cloud security groups (AWS, GCP, Azure, or your VPS panel).
• Package manager access (apt, dnf/yum, zypper).

High-level steps:

• Install OpenSSH server.
• Start and enable sshd.
• Allow SSH in firewall/security groups.
• Configure sshd_config securely.
• Generate and deploy SSH keys.
• Disable password and root login after you confirm key access works.

Step-by-Step: Install and Enable OpenSSH Server

Install OpenSSH by Linux distribution

OpenSSH is the most widely used SSH implementation. Install it using your distro’s package manager.

# Debian/Ubuntu
sudo apt update
sudo apt install -y openssh-server

# RHEL 8+/CentOS Stream/AlmaLinux/Rocky
sudo dnf install -y openssh-server

# Older CentOS/RHEL
sudo yum install -y openssh-server

# Fedora
sudo dnf install -y openssh-server

# openSUSE/SLES
sudo zypper install -y openssh

Start, enable, and verify the SSH service

sudo systemctl enable sshd
sudo systemctl start sshd
sudo systemctl status sshd
# Verify it listens (default port 22)
sudo ss -tnlp | grep sshd

On Debian/Ubuntu, the service may be named “ssh” but systemd aliases it to sshd. The above commands work across modern distros.

Open SSH in the firewall (and cloud security groups)

If you use UFW (Ubuntu/Debian):

sudo ufw allow 22/tcp
sudo ufw reload
sudo ufw status

If you use firewalld (RHEL/CentOS/Fedora):

sudo firewall-cmd --permanent --add-service=ssh
# Or if you plan to change the port: --add-port=2222/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

Also allow the SSH port in your cloud provider’s security group or your VPS panel. If a security group blocks SSH, the server-side firewall changes won’t help.

Configure SSH Securely (sshd_config)

The main configuration file is /etc/ssh/sshd_config. Always back it up first, edit carefully, and validate before reloading.

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
sudo nano /etc/ssh/sshd_config   # or use vim

Recommended baseline settings

Use these as a starting point. Do not disable passwords until keys are set up and tested.

# Change the default port if desired (also open it in your firewall/SG)
Port 22

Protocol 2
PermitRootLogin no
PasswordAuthentication yes    # switch to 'no' after key auth works
PubkeyAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes

# Limit authentication attempts and session behavior
MaxAuthTries 3
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2

# Restrict who can log in (replace with your usernames)
#AllowUsers adminuser devops

# Reduce surface area
X11Forwarding no
PermitEmptyPasswords no

# Log more detail for troubleshooting
LogLevel VERBOSE

Save, then validate and reload:

sudo sshd -t   # syntax check (no output means OK)
sudo systemctl reload sshd

Test safely before closing your current session

Open a new terminal and test SSH before you end your existing session, especially after changes like port updates or disabling passwords.

# If changed port, use -p
ssh -p 22 user@server-ip

# For first-time connections, verify the host key fingerprint.

Set Up SSH Key Authentication (Recommended)

Create a strong SSH key pair

Use ED25519 unless you need RSA for legacy systems.

# On your local machine (macOS/Linux/WSL)
ssh-keygen -t ed25519 -C "your_email@example.com"
# Accept defaults and add a passphrase for better security

Deploy your public key

Easiest method with ssh-copy-id:

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server-ip

If ssh-copy-id isn’t available, copy manually:

cat ~/.ssh/id_ed25519.pub | ssh user@server-ip "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Test login using the key:

ssh user@server-ip

Disable password auth after key login works

Once keys work, tighten authentication to prevent brute-force attacks.

# Edit config
sudo nano /etc/ssh/sshd_config
# Set:
PasswordAuthentication no

# Validate and reload
sudo sshd -t && sudo systemctl reload sshd

Create a Non-Root Admin and Use sudo

Disable root SSH login and operate through a non-root user with sudo privileges.

# Debian/Ubuntu
sudo adduser adminuser
sudo usermod -aG sudo adminuser

# RHEL/CentOS/Fedora
sudo useradd -m adminuser
sudo passwd adminuser
sudo usermod -aG wheel adminuser

Confirm you can SSH as the new user and run sudo before you disable root login.

SSH Hardening Best Practices

• Change the SSH port from 22 to a high, unused port and update firewall/security groups accordingly.
• Install Fail2ban to block repeated failed login attempts.
• Restrict access by IP using firewall rules or sshd_config Match blocks.
• Keep the OS and OpenSSH packages updated.
• Disable root login (PermitRootLogin no) and passwords (after key setup).
• Disable unused features (X11Forwarding no, PermitEmptyPasswords no).
• Use a bastion host (ProxyJump) instead of exposing many servers to the internet.
• Consider 2FA (e.g., Google Authenticator or Duo) for critical systems.

Basic Fail2ban setup:

# Install
# Debian/Ubuntu
sudo apt install -y fail2ban
# RHEL/CentOS/Fedora
sudo dnf install -y fail2ban

# Minimal jail
sudo tee /etc/fail2ban/jail.d/sshd.local >/dev/null <<'EOF'
[sshd]
enabled = true
maxretry = 3
bantime = 1h
findtime = 10m
EOF

sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd

If SELinux is enforcing, ensure your .ssh directory labels are correct after manual changes:

sudo restorecon -R -v /home/youruser/.ssh

Troubleshooting Common SSH Issues

Connection refused or timed out

• Is sshd running? Check systemctl status.
• Is the firewall open on the correct port?
• Did you allow the port in your cloud security group?
• Is the server listening on the right interface? Check “ss -tnlp | grep sshd”.

Permission denied (publickey)

• Verify the correct key is offered: ssh -v user@server-ip.
• Fix permissions: .ssh (700), authorized_keys (600), home dir not group-writable.
• Ensure your public key matches the server’s authorized_keys entry.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Check logs and validate config

sudo journalctl -u sshd -e
sudo sshd -t

Recovering from a bad config

• Use your hosting provider’s console/VNC to revert to the backup sshd_config.
• Keep one SSH session open while testing changes in a new session.
• Temporarily run two ports by adding a second Port line; remove it after confirming the new port works.

Real-World Quick Start (Ubuntu 22.04 Example)

# 1) Install and start
sudo apt update && sudo apt install -y openssh-server
sudo systemctl enable --now sshd

# 2) Allow firewall
sudo ufw allow 22/tcp && sudo ufw reload

# 3) Create admin user and set SSH keys
sudo adduser adminuser
sudo usermod -aG sudo adminuser
# On your local machine:
ssh-keygen -t ed25519
ssh-copy-id adminuser@server-ip

# 4) Harden sshd
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
sudo nano /etc/ssh/sshd_config
# Set: PermitRootLogin no, PubkeyAuthentication yes, PasswordAuthentication no (after testing)
sudo sshd -t && sudo systemctl reload sshd

# 5) Optional: change port and update firewall
# Edit sshd_config: Port 2222
sudo ufw allow 2222/tcp && sudo ufw delete allow 22/tcp && sudo ufw reload
# Test: ssh -p 2222 adminuser@server-ip

Why Your Hosting Choice Matters

Secure SSH isn’t just about commands—it’s about recovery options and reliability. With YouStable VPS, SSH is preinstalled, firewall templates are available, and you get an integrated browser console to fix configs if you lock yourself out. Snapshots and 24/7 support help you test changes safely without risking downtime.

Best Practices Checklist

• Install OpenSSH and enable sshd.
• Open SSH port in both OS firewall and cloud security groups.
• Set up ED25519 keys; test login.
• Disable root login and then password login.
• Limit auth attempts and idle sessions.
• Enable Fail2ban and apply updates regularly.
• Restrict users/IPs; consider changing the default port.
• Document your configuration and keep a secure backup.

FAQs