Level 4 — Initial Server Security

Your server has a public IP. Within minutes of first boot, automated scanners are probing port 22. This chapter is the hardening pass you run before installing anything else.

READ THIS BEFORE EDITING SSH CONFIGURATION

Never change SSH configuration with only one session open.

The safe procedure, every single time:

Keep your current SSH session open. Do not close it. Do not log out. It stays authenticated even if you break the config for new connections.
Make your changes in that session.
Validate the syntax: sudo sshd -t — no output means valid.
Reload: sudo systemctl reload ssh.
Open a brand-new terminal and connect. Do not reuse the old one.
Only when the new connection succeeds may you close the original session.

If step 5 fails, you still have the working session from step 1 to undo the change. If you skip step 1, your only recourse is the provider's web console — and you should already know where that button is (Level 1).

The hardening checklist

We will work through this in order:

[ ] Update all packages
[ ] Set the hostname and timezone
[ ] Remove unnecessary packages and services
[ ] Create the deploy user (done in Level 3)
[ ] Harden SSH configuration
[ ] Configure UFW (Level 5)
[ ] Install and configure fail2ban
[ ] Enable unattended security upgrades
[ ] Add swap (if RAM ≤ 4 GB)
[ ] Verify no unexpected services are listening
[ ] Set up log review habits

1. Update the system

bash

# SERVER
sudo apt update
sudo apt upgrade -y
sudo apt autoremove -y

Command	What it does
`apt update`	Downloads the package index from the repositories. Changes nothing installed. Always run first.
`apt upgrade`	Installs newer versions of installed packages. Will not remove packages.
`apt full-upgrade`	Like upgrade, but may remove packages to resolve dependencies. Needed for release upgrades.
`apt autoremove`	Removes packages installed as dependencies that nothing needs any more (mainly old kernels — this is what keeps `/boot` from filling up).
`apt autoclean`	Deletes cached `.deb` files for packages no longer downloadable

A FRESH VPS IMAGE IS ALREADY OUT OF DATE

Provider images are built periodically. Between the build and your apt upgrade, security patches accumulate. On a brand-new server this first upgrade routinely installs 30+ updates. Do it before anything else.

If a reboot is required

Kernel updates need a reboot to take effect:

bash

# SERVER
[ -f /var/run/reboot-required ] && cat /var/run/reboot-required
sudo reboot

Your SSH session drops. Reconnect after 30–60 seconds.

2. Hostname and timezone

bash

# SERVER
sudo hostnamectl set-hostname prod-app-01
timedatectl set-timezone UTC
timedatectl                      # verify

ALWAYS USE UTC ON SERVERS

Log timestamps, cron schedules, database timestamptz values, and certificate expiry all become vastly easier to reason about when every server is UTC. Convert to local time in the UI, never in the infrastructure. This also removes an entire class of daylight-saving bugs from your cron jobs.

Also add your hostname to /etc/hosts so sudo does not stall on a name lookup:

bash

# SERVER
echo "127.0.1.1 prod-app-01" | sudo tee -a /etc/hosts

Time synchronisation is handled by systemd-timesyncd out of the box on Ubuntu 24.04. Verify:

bash

timedatectl show-timesync --all | head -5
systemctl status systemd-timesyncd

Accurate time matters more than it seems: TLS certificate validation, JWT exp claims, TOTP two-factor codes, and log correlation all break with clock skew.

3. Remove unnecessary packages and services

Every listening service is attack surface. Start by finding out what is actually listening:

bash

# SERVER
sudo ss -tulpn

On a clean Ubuntu 24.04 cloud image you should see roughly:

tcp LISTEN 0.0.0.0:22      sshd
udp UNCONN 127.0.0.54:53   systemd-resolved
udp UNCONN 127.0.0.53:53   systemd-resolved

systemd-resolved on 127.0.0.53 is loopback-only and fine. Anything else bound to 0.0.0.0 deserves investigation.

Things sometimes present that you likely do not need:

bash

# SERVER — check before removing
dpkg -l | grep -E "apache2|sendmail|postfix|rpcbind|telnet|xinetd"

# Remove what you find and do not need
sudo apt purge -y apache2 apache2-utils   # if present and you use Nginx
sudo systemctl disable --now snapd        # if you do not use snaps

DO NOT BLINDLY PURGE

purge removes configuration files too. Check what depends on a package with apt-cache rdepends <pkg> before removing it. Removing postfix breaks local mail delivery, which some cron jobs rely on for reporting. When in doubt, systemctl disable --now <service> first — it is reversible.

4. SSH hardening

The SSH configuration file is at:

/etc/ssh/sshd_config

On Ubuntu 24.04, it ends with an include directive:

Include /etc/ssh/sshd_config.d/*.conf

INCLUDES OVERRIDE — AND THEY COME FIRST

Ubuntu 24.04 places the Include line near the top of sshd_config. In OpenSSH, the first occurrence of a keyword wins. That means settings in /etc/ssh/sshd_config.d/*.conf override anything you write in the main file below them.

Cloud providers drop files there (e.g. 50-cloud-init.conf containing PasswordAuthentication yes). If you edit the main file and your change appears to do nothing, this is why.

Always verify the effective config with sudo sshd -T, and prefer to put your own settings in a drop-in file with a high-sorting name.

The recommended approach: a drop-in file

bash

# SERVER
sudo nano /etc/ssh/sshd_config.d/99-hardening.conf

sshconfig

# /etc/ssh/sshd_config.d/99-hardening.conf

# --- Authentication ---
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
PermitEmptyPasswords no
KbdInteractiveAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

# --- Limit who can log in ---
AllowUsers deploy

# --- Reduce brute-force surface ---
MaxAuthTries 3
MaxSessions 5
LoginGraceTime 20
MaxStartups 10:30:60

# --- Disable features we do not use ---
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no
PermitUserEnvironment no

# --- Keepalives: drop dead connections ---
ClientAliveInterval 300
ClientAliveCountMax 2

# --- Logging ---
LogLevel VERBOSE

Line by line:

Directive	Effect	Why
`PermitRootLogin no`	Root cannot log in over SSH at all	Removes the single most-targeted account. Attackers must guess a username and have a key.
`PasswordAuthentication no`	Keys only	The most important line in this file. Eliminates brute-force entirely.
`AuthenticationMethods publickey`	Belt and braces — only publickey is acceptable	Prevents a PAM misconfiguration from re-enabling a password path
`PermitEmptyPasswords no`	Reject accounts with blank passwords	Default, but be explicit
`AllowUsers deploy`	Only `deploy` may connect; everyone else is rejected before auth	A whitelist beats a blacklist. Add more users space-separated.
`MaxAuthTries 3`	Disconnect after 3 failed attempts per connection	Slows automated attempts; also causes "Too many authentication failures" if your agent offers many keys
`LoginGraceTime 20`	20 seconds to authenticate, then disconnect	Frees slots held by half-open connections
`MaxStartups 10:30:60`	Beyond 10 unauthenticated connections, start dropping 30% of new ones; refuse all at 60	Resists connection-flood DoS
`X11Forwarding no`	No GUI forwarding	Never needed on a server; has had CVEs
`AllowAgentForwarding no`	Clients cannot forward their agent here	Prevents this server being used to pivot with someone's agent (Level 1)
`AllowTcpForwarding no`	No port forwarding through SSH	Prevents using your server as a tunnel/proxy. Turn this back on if you use `ssh -L` to reach PostgreSQL — see below.
`ClientAliveInterval 300`	Ping idle clients every 5 minutes	Cleans up dead sessions
`LogLevel VERBOSE`	Logs key fingerprints used for each login	Lets you audit which key authenticated — essential for incident response

AllowTcpForwarding no BREAKS SSH TUNNELS

The recommended way to reach PostgreSQL from your laptop is an SSH tunnel (ssh -L 5433:127.0.0.1:5432 deploy@server), which requires TCP forwarding. If you use TablePlus/DBeaver/pgAdmin this way, set AllowTcpForwarding yes. That is a reasonable trade: the alternative is exposing 5432, which is far worse.

Should you change the SSH port?

You will read everywhere that moving SSH from 22 to 2222 improves security. Here is the honest assessment:

What it actually does:

Eliminates ~99% of automated log noise. Mass scanners hit port 22 and move on. Your auth.log becomes readable and fail2ban has less to do.
Reduces wasted CPU and bandwidth on junk connections.

What it does not do:

It is not security. nmap -p- your-server finds SSH on any port in under a minute. A targeted attacker is unaffected.
It does not protect a weak password (disabling password auth does).
It does not protect against a stolen key (key passphrases and rotation do).

What it costs:

Every teammate, deploy script, CI config, and monitoring check needs the port.
You must add -p 2222 or a ~/.ssh/config entry forever.
SELinux/AppArmor and some corporate networks that only allow outbound 22 cause friction.
One more thing to remember at 3 a.m. during an incident.

THE VERDICT

Changing the port is log hygiene, not security. It is a legitimate, mild convenience — but do not let it substitute for the things that matter.

Ranked by actual security value:

PasswordAuthentication no — enormous
PermitRootLogin no — large
AllowUsers deploy — meaningful
fail2ban — meaningful (mostly for other services once passwords are off)
Non-standard port — negligible security value, real log-noise value

Do 1–4 first. Do 5 only if the log noise genuinely bothers you.

If you do change it:

sshconfig

# in /etc/ssh/sshd_config.d/99-hardening.conf
Port 2222

bash

# SERVER — open the new port BEFORE reloading sshd
sudo ufw allow 2222/tcp
sudo sshd -t
sudo systemctl reload ssh
# Test from a NEW terminal: ssh -p 2222 deploy@server
# Only after that succeeds:
sudo ufw delete allow 22/tcp

THE ORDER MATTERS

Open the new port in the firewall before reloading sshd, and remove the old rule only after confirming the new port works from a fresh terminal. Doing it in the wrong order locks you out. Also update your provider's cloud firewall.

Apply and verify

bash

# SERVER
sudo sshd -t                     # validate syntax — silence means OK
sudo sshd -T | grep -Ei "permitrootlogin|passwordauth|pubkeyauth|allowusers|port|maxauthtries"
sudo systemctl reload ssh

sshd -T prints the effective configuration with all includes resolved. Trust it over the file contents. Expected output:

port 22
permitrootlogin no
pubkeyauthentication yes
passwordauthentication no
maxauthtries 3
allowusers deploy

reload vs restart

systemctl reload ssh re-reads the config without dropping existing connections — your current session survives even if the new config is broken. restart kills the daemon and restarts it; on Ubuntu, existing sessions usually survive that too, but reload is the safer habit.

Now, from a new terminal:

bash

# LOCAL
ssh deploy@203.0.113.10          # should work
ssh root@203.0.113.10            # should be rejected

Verify password auth is genuinely off:

bash

# LOCAL
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password deploy@203.0.113.10
# Expected: Permission denied (publickey).

5. fail2ban

fail2ban watches log files for repeated failures and temporarily adds firewall rules to block the offending IP.

Why bother if passwords are disabled?

A fair question. With key-only auth, SSH brute force cannot succeed. fail2ban still earns its place:

It stops the resource waste of thousands of connection attempts per day.
It protects other services you will add later — Nginx (nginx-http-auth, nginx-limit-req), and any application login endpoint you write a filter for.
It gives you a running count of who is attacking you, which is genuinely useful intelligence.

It is a defence-in-depth layer, not the primary defence.

Install

bash

# SERVER
sudo apt install -y fail2ban

UBUNTU 24.04 + fail2ban: THE nftables GOTCHA

The fail2ban package in Ubuntu 24.04 defaults to a banaction of iptables-multiport, but 24.04 uses nftables underneath. On some installs the SSH jail fails to start with Failed to execute ban jail 'sshd' action 'iptables-multiport'.

Two fixes, either works:

Set banaction = nftables-multiport in your local config (shown below), or
sudo apt install -y iptables to provide the compatibility shim.

Always check sudo fail2ban-client status sshd after installing — a jail that failed to start is silently useless.

Configure

Never edit /etc/fail2ban/jail.conf — package upgrades overwrite it. Create a local override:

bash

# SERVER
sudo nano /etc/fail2ban/jail.local

ini

[DEFAULT]
# Ignore your own IPs so you never ban yourself
ignoreip = 127.0.0.1/8 ::1

# How long a ban lasts
bantime  = 1h
# Window in which failures are counted
findtime = 10m
# Failures within findtime that trigger a ban
maxretry = 5

# Escalate: each repeat offence multiplies the ban duration
bantime.increment = true
bantime.factor    = 2
bantime.maxtime   = 1w

# Ubuntu 24.04 uses nftables
banaction = nftables-multiport
banaction_allports = nftables-allports

# Read from systemd journal rather than log files (correct for Ubuntu 24.04)
backend = systemd

[sshd]
enabled = true
port    = ssh
maxretry = 3
bantime  = 1h

PUT YOUR OWN IP IN ignoreip

If you have a static IP or office range, add it: ignoreip = 127.0.0.1/8 ::1 198.51.100.42. Fat-fingering your key three times from a dynamic IP and banning yourself for an hour is a genuine and irritating way to lose access. (Your provider's web console remains available, so it is recoverable — but avoid it.)

Start and verify

bash

# SERVER
sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban

sudo fail2ban-client status            # which jails are active
sudo fail2ban-client status sshd       # detail for the SSH jail

Status for the jail: sshd
|- Filter
|  |- Currently failed: 2
|  |- Total failed:     1847
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service
`- Actions
   |- Currently banned: 7
   |- Total banned:     213
   `- Banned IP list:   45.148.10.92 193.32.162.157 ...

"Total failed: 1847" after a day is normal for any public IP. That is what the internet looks like.

Managing bans

bash

# SERVER
sudo fail2ban-client set sshd unbanip 198.51.100.42    # unban an IP
sudo fail2ban-client set sshd banip 198.51.100.99      # ban manually
sudo fail2ban-client unban --all                        # clear all bans
sudo tail -f /var/log/fail2ban.log                      # watch it work

An Nginx jail (add after installing Nginx)

ini

# append to /etc/fail2ban/jail.local

[nginx-http-auth]
enabled  = true
port     = http,https
logpath  = /var/log/nginx/error.log

[nginx-limit-req]
enabled  = true
port     = http,https
logpath  = /var/log/nginx/error.log
findtime = 10m
maxretry = 10
bantime  = 1h

[nginx-botsearch]
enabled  = true
port     = http,https
logpath  = /var/log/nginx/access.log
maxretry = 5

nginx-limit-req pairs with Nginx's limit_req directive (Level 14): Nginx rate-limits and logs, fail2ban then bans persistent offenders outright.

6. Unattended security upgrades

The single highest-value/lowest-effort security control. Most compromises exploit vulnerabilities that were patched months earlier.

bash

# SERVER
sudo apt install -y unattended-upgrades apt-listchanges
sudo dpkg-reconfigure -plow unattended-upgrades   # answer Yes

bash

# SERVER
sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
};

// Do not auto-upgrade these — you control their versions deliberately
Unattended-Upgrade::Package-Blacklist {
    "postgresql-16";
    "docker-ce";
    "nginx";
};

Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";

// Reboot automatically if a kernel update requires it
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";

Unattended-Upgrade::Mail "you@example.com";
Unattended-Upgrade::MailReport "on-change";

Automatic-Reboot "true" IS A DELIBERATE TRADE-OFF

It means your server may restart at 04:00 without warning. Only enable it if you have verified that everything comes back automatically after a reboot:

bash

sudo systemctl is-enabled nginx postgresql redis-server docker
pm2 startup      # and pm2 save — see Level 13
sudo reboot      # then verify everything is up

If you have not tested a reboot, set it to "false" and apply kernel updates during a maintenance window instead. An unattended reboot that leaves your app down is worse than a delayed kernel patch. Test the reboot once, deliberately, before trusting automation with it.

Verify:

bash

# SERVER
sudo unattended-upgrades --dry-run --debug     # simulate
systemctl status unattended-upgrades
cat /var/log/unattended-upgrades/unattended-upgrades.log

7. Swap space

If your server has 4 GB of RAM or less, add swap. Nuxt builds and pnpm install both spike hard.

bash

# SERVER
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Prefer RAM; only swap under real pressure
echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf
sudo sysctl -p /etc/sysctl.d/99-swappiness.conf

free -h

Step	Why
`fallocate -l 2G`	Create a 2 GB file. Rule of thumb: swap = RAM, capped at 4 GB.
`chmod 600`	Swap contains memory contents — potentially secrets. Root-only.
`mkswap` / `swapon`	Format and activate it
`/etc/fstab` entry	Re-enable on reboot
`vm.swappiness=10`	Default is 60 (swap eagerly). 10 means "only under pressure".

SWAP IS AN AIRBAG, NOT AN UPGRADE

Swap on a network-attached SSD is orders of magnitude slower than RAM. It prevents the OOM killer from terminating PostgreSQL during a build spike; it does not let you run a workload that needs more RAM than you have. If you are constantly swapping, resize the server — check with vmstat 1 and watch the si/so columns.

8. Kernel network hardening (optional but cheap)

bash

# SERVER
sudo nano /etc/sysctl.d/99-hardening.conf

ini

# Ignore ICMP redirects and source-routed packets
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_redirects = 0

# Log packets with impossible source addresses
net.ipv4.conf.all.log_martians = 1

# Enable reverse-path filtering (anti-spoofing)
net.ipv4.conf.all.rp_filter = 1

# SYN flood protection
net.ipv4.tcp_syncookies = 1

# Do not act as a router
net.ipv4.ip_forward = 0

# Restrict access to kernel pointers and dmesg
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1

bash

sudo sysctl --system

ip_forward = 0 BREAKS DOCKER

Docker requires IP forwarding to route container traffic. If you run Docker, either omit that line or let Docker set it back (it does so at startup). Verify after installing Docker: sysctl net.ipv4.ip_forward should be 1.

9. Secrets on disk

Covered fully in Level 8, but the baseline now:

bash

# SERVER
chmod 600 /home/deploy/apps/myapp/.env
chown deploy:deploy /home/deploy/apps/myapp/.env

bash

# SERVER — audit: are any secrets world-readable?
find /home /var/www /opt -name "*.env*" -type f ! -perm 600 2>/dev/null
find /home /var/www -name "*.pem" -o -name "id_rsa" -o -name "id_ed25519" 2>/dev/null

Also disable shell history for sensitive sessions when you must paste a secret:

bash

# SERVER — this shell only
unset HISTFILE

Otherwise your database password ends up in ~/.bash_history, which survives reboots and is included in home-directory backups.

10. Verify the result

bash

# SERVER — nothing unexpected should be listening on 0.0.0.0
sudo ss -tulpn | grep -v 127.0.0

# SSH effective config
sudo sshd -T | grep -Ei "permitroot|passwordauth|allowusers|maxauth"

# Firewall
sudo ufw status verbose

# fail2ban jails running
sudo fail2ban-client status

# Automatic upgrades enabled
systemctl is-enabled unattended-upgrades

# Any pending reboot?
[ -f /var/run/reboot-required ] && echo "REBOOT REQUIRED"

# Recent authentication activity
sudo journalctl -u ssh --since "24 hours ago" | grep -Ei "accepted|failed" | tail -30

A LIGHTWEIGHT AUDIT

lynis gives a scored report with prioritised suggestions:

bash

sudo apt install -y lynis
sudo lynis audit system

Do not chase a perfect score — many suggestions are irrelevant to a single-purpose app server. Read the "Warnings" section, ignore most of "Suggestions".

What can go wrong

Symptom	Cause	Fix
Locked out after SSH edit	Config error, or key not in place for the allowed user	Provider web console → revert `/etc/ssh/sshd_config.d/*.conf` → `systemctl restart ssh`
Changes to `sshd_config` have no effect	Overridden by `sshd_config.d/50-cloud-init.conf`	Check `sudo sshd -T`; edit the drop-in or use a `99-` prefixed file
Banned by your own fail2ban	Repeated failed attempts from your IP	Web console → `sudo fail2ban-client set sshd unbanip <ip>`; add to `ignoreip`
fail2ban jail not running	nftables/iptables mismatch on 24.04	Set `banaction = nftables-multiport`, `systemctl restart fail2ban`
Server rebooted unexpectedly at 04:00	`Automatic-Reboot "true"`	Expected. Verify services auto-start, or disable it.
`sudo` hangs for ~10 seconds	Hostname not resolvable	Add `127.0.1.1 <hostname>` to `/etc/hosts`
Docker networking broken after sysctl	`ip_forward = 0`	Remove that line, `sysctl --system`, restart Docker

Production Checklist — Level 4

[ ] apt update && apt upgrade run; rebooted if the kernel changed
[ ] Hostname set, timezone is UTC, time sync confirmed
[ ] sudo ss -tulpn shows only SSH (and later Nginx) on public interfaces
[ ] PermitRootLogin no
[ ] PasswordAuthentication no — verified by attempting a password login and being refused
[ ] AllowUsers deploy restricts SSH to the accounts I intend
[ ] Config verified with sudo sshd -T, not by reading the file
[ ] I tested SSH from a new terminal before closing the old session
[ ] fail2ban installed, sshd jail confirmed running via fail2ban-client status sshd
[ ] My own IP is in fail2ban's ignoreip
[ ] unattended-upgrades enabled and verified with --dry-run
[ ] Automatic reboot is either disabled, or enabled after testing a full reboot recovery
[ ] Swap configured if RAM ≤ 4 GB, with vm.swappiness=10
[ ] .env files are 600
[ ] I know where my provider's web console is and have used it at least once

Next: Level 5 — Firewall →

Level 4 — Initial Server Security ​

The hardening checklist ​

1. Update the system ​

If a reboot is required ​

2. Hostname and timezone ​

3. Remove unnecessary packages and services ​

4. SSH hardening ​

The recommended approach: a drop-in file ​

Should you change the SSH port? ​

Apply and verify ​

5. fail2ban ​

Why bother if passwords are disabled? ​

Install ​

Configure ​

Start and verify ​

Managing bans ​

An Nginx jail (add after installing Nginx) ​

6. Unattended security upgrades ​

7. Swap space ​

8. Kernel network hardening (optional but cheap) ​

9. Secrets on disk ​

10. Verify the result ​

What can go wrong ​

Production Checklist — Level 4 ​

Level 4 — Initial Server Security

The hardening checklist

1. Update the system

If a reboot is required

2. Hostname and timezone

3. Remove unnecessary packages and services

4. SSH hardening

The recommended approach: a drop-in file

Should you change the SSH port?

Apply and verify

5. fail2ban

Why bother if passwords are disabled?

Install

Configure

Start and verify

Managing bans

An Nginx jail (add after installing Nginx)

6. Unattended security upgrades

7. Swap space

8. Kernel network hardening (optional but cheap)

9. Secrets on disk

10. Verify the result

What can go wrong

Production Checklist — Level 4