Level 23 — Server Security, Advanced

Threat scenarios with concrete playbooks: how each attack happens, how to prevent it, how to detect it, and how to recover.

The security model

Defence in depth: each layer assumes the one above it may fail. The database is bound to 127.0.0.1 and firewalled and password-protected — because any single control can be misconfigured.

The complete security checklist

Access

[ ] SSH password authentication disabled
[ ] Root SSH login disabled
[ ] AllowUsers restricts SSH to named accounts
[ ] SSH keys are Ed25519 with passphrases (except scoped CI keys)
[ ] CI keys restricted with command= and no-port-forwarding
[ ] fail2ban running with the sshd jail confirmed active
[ ] One account per human; no shared credentials
[ ] Offboarding rotates every secret the person could reach

Network

[ ] UFW default-deny inbound; only 22, 80, 443 allowed
[ ] Cloud firewall configured as an independent second layer
[ ] PostgreSQL bound to 127.0.0.1, port never opened
[ ] Redis bound to 127.0.0.1 with requirepass
[ ] Application ports bound to 127.0.0.1
[ ] Docker published ports prefixed 127.0.0.1: or absent
[ ] sudo ss -tulpn reviewed after every deploy

System

[ ] unattended-upgrades enabled and verified
[ ] Application runs as a non-root user
[ ] docker group membership limited to trusted operators
[ ] Passwordless sudo scoped to specific absolute command paths
[ ] .env at mode 600
[ ] No unexpected setuid binaries
[ ] journald persistent with retention configured

Application

[ ] TLS 1.2+ only, valid certificate, auto-renewal tested
[ ] Security headers on every response
[ ] Rate limiting on general traffic and stricter on auth endpoints
[ ] Input validation with whitelist: true and forbidNonWhitelisted: true
[ ] Parameterised queries only (Prisma does this)
[ ] Passwords hashed with argon2id or bcrypt (cost ≥ 12)
[ ] JWT secrets ≥ 32 random bytes, unique per purpose
[ ] CORS restricted to known origins — never * with credentials
[ ] Dependencies audited; critical CVEs patched promptly

Operations

[ ] Off-site, encrypted backups with tested restore
[ ] External uptime monitoring
[ ] Error tracking with secret scrubbing
[ ] Log retention long enough for incident investigation
[ ] Documented incident response procedure

Threat scenarios

1. SSH brute force

Threat: Automated attempts to guess credentials for a shell.

How it happens: Botnets scan the entire IPv4 space for port 22 and try common username/password pairs — root:root, admin:admin, ubuntu:ubuntu — thousands of times per hour. Your server sees this from the moment it boots.

Prevent:

bash

# /etc/ssh/sshd_config.d/99-hardening.conf
PasswordAuthentication no        # ← this alone makes brute force impossible
PermitRootLogin no
AllowUsers deploy
MaxAuthTries 3

Plus fail2ban and, optionally, cloud-firewall restriction to known IPs.

Detect:

bash

sudo journalctl -u ssh --since "24 hours ago" | grep -c "Failed password"
sudo grep "Accepted publickey" /var/log/auth.log | tail -20
sudo fail2ban-client status sshd
sudo lastb | head -20                 # failed login attempts
last -20                              # successful logins

WHAT NORMAL LOOKS LIKE

Thousands of failed attempts per day is normal background noise on any public IP. With password auth disabled, none can succeed.

What is not normal: a Accepted publickey line you do not recognise, at a time you were not working, from an IP you do not know. That is the line to alert on — LogLevel VERBOSE records which key fingerprint was used (Level 4).

Recover: If a login succeeded that you cannot account for, treat it as a full compromise — see the recovery procedure at the end of this chapter.

2. Credential theft (your laptop or key)

Threat: An attacker obtains your SSH private key or password-manager contents.

How it happens: Laptop stolen or malware-infected; a key committed to a public repository; a key copied to a shared machine; a phishing page harvesting your password-manager master password.

Prevent:

Passphrase on every SSH private key, plus full-disk encryption
Hardware-backed keys (YubiKey, Secure Enclave) where practical
Separate keys per server so one compromise is contained
Never forward your SSH agent to a machine you do not fully control (Level 1)
2FA on GitHub, GitLab, your VPS provider, and your DNS provider

Detect:

bash

sudo grep "Accepted publickey" /var/log/auth.log | awk '{print $NF, $(NF-2)}' | sort -u
last -50
who

With LogLevel VERBOSE, each accepted login records the key fingerprint. Compare against your known keys.

Recover:

bash

# SERVER — via provider console
# 1. Remove every key you do not recognise
nano ~/.ssh/authorized_keys
# 2. Kill active sessions
sudo pkill -u deploy sshd
# 3. Rotate everything the key could reach

THE PROVIDER ACCOUNT IS THE REAL CROWN JEWEL

Someone with your Hetzner/DigitalOcean login does not need SSH — they can reset the root password, mount your disk, or take a snapshot and download everything.

2FA on the provider account is more important than any server hardening. Same for your domain registrar: control of DNS means the attacker can issue a valid certificate for your domain and intercept all traffic.

3. Stolen API key

Threat: A third-party credential (Stripe, AWS, SMTP) leaks.

How it happens: Committed to Git; logged in an error report; present in a client-side bundle; leaked in a screenshot or support ticket; exfiltrated from a compromised server.

Prevent:

Never commit secrets; enable push protection (Level 7)
Scrub secrets from logs and error reports (Level 21)
Scope keys to the minimum permissions and, where supported, restrict by IP
Use separate keys per environment
Prefer short-lived tokens where the provider offers them

Detect:

bash

# LOCAL — scan history
gitleaks detect --source . --verbose

# SERVER — check the built bundle
grep -riE "sk_live|AKIA[0-9A-Z]{16}" .output/public/ | head

# Provider dashboards: unusual API call volume, calls from unknown IPs

GitHub scans public pushes and notifies major providers automatically — which is why an AWS key pushed publicly is often disabled within minutes.

Recover:

Revoke the key at the provider immediately. This is the only step that stops the bleeding.
Issue a new key, update .env, pm2 reload --update-env.
Review the provider's audit log for what was done with it.
If it was a payment key, check for fraudulent charges.
Purge from Git history — after revoking, not instead of it.

REVOKE FIRST, CLEAN UP SECOND

People often start with git filter-repo to scrub history. That takes an hour, and the key is live and being used the whole time. Revoke in 30 seconds, then clean up at your leisure.

4. Database exposure

Threat: PostgreSQL reachable from the internet.

How it happens: listen_addresses = '*'; a UFW rule added "temporarily" for a GUI client; a Docker ports: "5432:5432" bypassing UFW entirely (Level 5).

Prevent:

bash

# SERVER — verify all three
sudo ss -tulpn | grep 5432                   # must be 127.0.0.1
sudo ufw status | grep 5432                  # must be empty
docker ps --format "{{.Names}} {{.Ports}}" | grep 5432   # must be 127.0.0.1 or absent

Plus: a scoped role, not postgres; scram-sha-256; no trust in pg_hba.conf.

Detect:

bash

# LOCAL — from outside; must time out
nc -vz 203.0.113.10 5432

# SERVER — unexpected connections
sudo -u postgres psql -c "SELECT usename, client_addr, state, backend_start FROM pg_stat_activity WHERE client_addr IS NOT NULL;"
sudo grep "connection authorized" /var/log/postgresql/*.log | awk '{print $NF}' | sort | uniq -c

client_addr should only ever be 127.0.0.1 or NULL.

Recover: Close the exposure, rotate the database password, audit pg_stat_activity and logs for what was accessed, and assume the data was copied. If it contained personal data, you likely have a legal notification obligation with a deadline measured in days.

5. Redis exposure

Threat: Redis reachable and unauthenticated.

THIS IS THE FASTEST PATH FROM EXPOSURE TO FULL COMPROMISE

Redis executes commands from anyone who can reach it. The automated attack:

CONFIG SET dir /home/deploy/.ssh
CONFIG SET dbfilename authorized_keys
SET x "\n\nssh-rsa AAAA... attacker\n\n"
SAVE

The attacker's key is now in your authorized_keys. They log in. Elapsed time from your port becoming visible: minutes.

Variants write to /var/spool/cron/crontabs/root or exploit the Lua sandbox. Shodan indexes tens of thousands of open Redis instances continuously.

Prevent: bind 127.0.0.1 -::1, requirepass with 32+ random bytes, protected-mode yes, CONFIG renamed to "", port never opened (Level 10).

Detect:

bash

# SERVER
sudo ss -tulpn | grep 6379              # must be 127.0.0.1
redis-cli ping                          # must return NOAUTH
cat ~/.ssh/authorized_keys              # any key you do not recognise?
sudo crontab -l; crontab -l             # unexpected jobs?
ps aux --sort=-%cpu | head              # a miner burning CPU?

Recover: Assume full server compromise. Rebuild from backup on a fresh server — see the recovery procedure below.

6. Malicious or compromised dependency

Threat: A package in your tree runs hostile code.

How it happens:

Typosquatting — expres instead of express
Account takeover — a maintainer's npm account is phished and a malicious version published
Protestware — a maintainer deliberately sabotages their own widely-used package
Dependency confusion — a public package shadows your private one
Postinstall scripts — arbitrary code executed at install time, before any of your code runs

npm install EXECUTES ARBITRARY CODE

Postinstall scripts run automatically with your user's privileges. A compromised transitive dependency — five levels deep, that you have never heard of — can read your .env, exfiltrate your SSH keys, and install a backdoor, during pnpm install.

This is why never run pnpm install as root. As deploy the damage is limited to what deploy can reach; as root it is the whole machine.

Prevent:

bash

pnpm install --frozen-lockfile         # exact tested versions only
pnpm audit --audit-level=high
pnpm config set minimum-release-age 1440   # only install packages ≥24h old

Practice	Effect
`--frozen-lockfile`	No surprise version resolution
Lockfile reviewed in PRs	Unexpected new dependencies are visible
Minimum release age	Most malicious versions are pulled within hours
Fewer dependencies	The most effective control there is
Dependabot/Renovate with a delay	Updates arrive, but not instantly
`--ignore-scripts` in CI	Blocks postinstall (needs explicit `prisma generate`)

Detect:

bash

pnpm audit
pnpm why suspicious-package
git diff HEAD~1 pnpm-lock.yaml | grep "^+" | grep resolution

# SERVER — unexpected outbound connections
sudo ss -tupn | grep ESTAB | grep -v "127.0.0.1"

Recover: Pin to a known-good version, rotate every secret the build environment could read (.env, CI secrets, SSH keys on the build host), and audit for persistence.

THE REAL DEFENCE IS FEWER DEPENDENCIES

Every package is a trust relationship with strangers, transitively. A typical Nuxt + NestJS app has 1,000+ packages. Before adding one, ask whether 20 lines of your own code would do. left-pad was a real outage; event-stream was a real compromise.

7. Web application vulnerability

Threat: A bug in your code — SQL injection, XSS, IDOR, SSRF, path traversal, broken access control.

Prevent, by class:

Vulnerability	Defence
SQL injection	Prisma parameterises everything. Never use `$queryRawUnsafe` with user input.
XSS	Vue escapes by default. Never `v-html` with user content. Add a Content-Security-Policy.
IDOR	Check ownership on every record access — `where: { id, userId: currentUser.id }`
SSRF	Validate and allowlist outbound URLs. Block requests to `169.254.169.254` (cloud metadata) and RFC1918 ranges.
Path traversal	Never build paths from user input; validate with `path.resolve` and a prefix check
Mass assignment	`ValidationPipe({ whitelist: true, forbidNonWhitelisted: true })`
Broken auth	Short-lived access tokens, rotating refresh tokens, rate-limited login
CSRF	`SameSite=Lax` cookies plus a CSRF token for state-changing requests

IDOR IS THE MOST COMMON SERIOUS BUG IN CRUD APPLICATIONS

// ❌ Any authenticated user can read any order by guessing an ID
const order = await prisma.order.findUnique({ where: { id: params.id } });

// ✅ Scoped to the requesting user
const order = await prisma.order.findFirst({
  where: { id: params.id, userId: req.user.id },
});

It is invisible in normal use — the UI only ever shows your own IDs — and trivially exploitable by changing a number in a URL. Audit every endpoint that accepts an ID.

Detect:

bash

# SERVER — probing patterns in access logs
sudo grep -iE "union.*select|\.\./|<script|%00|/etc/passwd" /var/log/nginx/access.log | tail -30

# Sudden 4xx spikes from one IP
sudo awk '$9 ~ /^4/ {print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

Recover: Patch, deploy, then determine what was accessed. Rotate any credential that could have been read. If personal data was exposed, follow your notification obligations.

8. Privilege escalation

Threat: An attacker with limited access becomes root.

How it happens: A kernel exploit; a setuid binary with a vulnerability; a writable script that root executes via cron; overly-broad passwordless sudo; docker group membership; a world-writable file in root's PATH.

Prevent:

bash

# SERVER — audit setuid binaries
sudo find / -perm -4000 -type f 2>/dev/null

# World-writable files outside /tmp
sudo find / -xdev -perm -0002 -type f -not -path "/proc/*" -not -path "/tmp/*" 2>/dev/null

# What can deploy run as root?
sudo -l -U deploy

# Are root's cron scripts writable by others?
sudo ls -la /etc/cron.d /etc/cron.daily

PASSWORDLESS SUDO FOR EVERYTHING IS PRIVILEGE ESCALATION BY DESIGN

deploy ALL=(ALL) NOPASSWD: ALL     # ❌

Any code running as deploy — including a malicious postinstall script — becomes root with no prompt. You have converted a dependency compromise into a full server compromise.

Scope it to specific absolute paths, and never to a command that can spawn a shell (vim, less, find, awk, env, systemctl with an arbitrary unit are all escapes). (Level 3)

Detect:

bash

sudo grep "sudo:" /var/log/auth.log | tail -30
sudo grep "COMMAND=" /var/log/auth.log | grep -v "deploy.sh" | tail
sudo ausearch -m avc 2>/dev/null | tail

Recover: Full rebuild. A root compromise cannot be reliably cleaned.

9. Supply chain via CI/CD

Threat: Your pipeline is used to deploy hostile code.

How it happens: A compromised GitHub Action (tag moved to point at malicious code); a leaked CI secret; a malicious PR triggering a workflow with secrets; a compromised maintainer account with push access.

Prevent:

Pin third-party actions to commit SHAs, never tags (Level 18)
permissions: contents: read
Never deploy from pull_request; never use pull_request_target with PR code checkout
Branch protection with required reviews on the deploy branch
Environment protection rules requiring approval
command=-restricted CI SSH key

Detect: Review workflow run history for runs you did not expect. Check the deployed commit SHA on your health endpoint matches what you merged.

Recover: Rotate all CI secrets and the deploy key, audit what was deployed, roll back, and review the commit history for injected changes.

Nginx security configuration

nginx

# /etc/nginx/snippets/security-headers.conf
add_header X-Frame-Options            "SAMEORIGIN"                              always;
add_header X-Content-Type-Options     "nosniff"                                 always;
add_header Referrer-Policy            "strict-origin-when-cross-origin"         always;
add_header Permissions-Policy         "camera=(), microphone=(), geolocation=()" always;
add_header Strict-Transport-Security  "max-age=31536000; includeSubDomains"     always;
add_header Content-Security-Policy    "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https://api.example.com; frame-ancestors 'self'; base-uri 'self'; form-action 'self'" always;

CSP BREAKS THINGS BEFORE IT PROTECTS THINGS

A strict Content-Security-Policy will block your analytics, your inline scripts, and possibly Nuxt's hydration payload — presenting as a blank page with console errors.

Roll it out safely:

Start with Content-Security-Policy-Report-Only and a report-uri
Collect violations for a week
Adjust the policy to permit what is legitimate
Switch to enforcing

Nuxt SSR typically needs 'unsafe-inline' for styles, and nonces or hashes for its inline scripts.

nginx

# Block common probing
location ~ /\.(?!well-known) { deny all; return 404; }     # dotfiles, but allow ACME
location ~* \.(env|git|sql|bak|old|swp)$ { deny all; return 404; }
location = /.env { deny all; return 404; }

THESE PATHS ARE PROBED CONSTANTLY

/.env, /.git/config, /wp-login.php, /phpmyadmin. Returning 404 rather than 403 gives away less information. Your logs will fill with these regardless — that is the internet, not a targeted attack.

Password hashing

import * as argon2 from 'argon2';

const hash = await argon2.hash(password, {
  type: argon2.argon2id,
  memoryCost: 19456,   // 19 MiB — OWASP minimum
  timeCost: 2,
  parallelism: 1,
});

const valid = await argon2.verify(hash, password);

NEVER STORE PASSWORDS WITH MD5, SHA-1, OR PLAIN SHA-256

Those are fast hashes — a GPU computes billions per second, so a leaked database is cracked in hours. Password hashing must be deliberately slow and memory-hard.

argon2id is the current recommendation. bcrypt with cost ≥ 12 remains acceptable and is well-supported.

ARGON2 MEMORY COST × CONCURRENT LOGINS

At 19 MiB per hash, 50 simultaneous logins is ~1 GB of RAM. On a small VPS that is an accidental denial of service. Rate-limit your login endpoint (Level 14) and size memoryCost against your actual RAM.

Regular audits

bash

#!/usr/bin/env bash
# /home/deploy/scripts/security-audit.sh
echo "=== Listening on public interfaces (should be 22, 80, 443 only) ==="
sudo ss -tulpn | grep -vE "127.0.0.1|::1"

echo -e "\n=== Firewall ==="
sudo ufw status verbose

echo -e "\n=== SSH effective config ==="
sudo sshd -T | grep -Ei "permitroot|passwordauth|allowusers|maxauth|pubkeyauth"

echo -e "\n=== Authorized keys ==="
for f in /home/*/.ssh/authorized_keys /root/.ssh/authorized_keys; do
  [ -f "$f" ] && { echo "--- $f"; ssh-keygen -lf "$f" 2>/dev/null; }
done

echo -e "\n=== Sudo privileges ==="
sudo grep -rvE '^\s*#|^\s*$' /etc/sudoers /etc/sudoers.d/ 2>/dev/null

echo -e "\n=== Users with a login shell ==="
awk -F: '$7 ~ /(bash|sh|zsh)$/ {print $1, $7}' /etc/passwd

echo -e "\n=== Setuid binaries ==="
sudo find / -perm -4000 -type f 2>/dev/null

echo -e "\n=== World-writable files ==="
sudo find / -xdev -perm -0002 -type f -not -path "/proc/*" -not -path "/tmp/*" 2>/dev/null | head -20

echo -e "\n=== Pending security updates ==="
apt list --upgradable 2>/dev/null | grep -i security

echo -e "\n=== fail2ban ==="
sudo fail2ban-client status

echo -e "\n=== Failed SSH (24h) ==="
sudo journalctl -u ssh --since "24 hours ago" | grep -c "Failed password"

echo -e "\n=== Successful SSH logins (recent) ==="
last -15

echo -e "\n=== TLS expiry ==="
sudo certbot certificates 2>/dev/null | grep -E "Certificate Name|Expiry"

echo -e "\n=== Cron jobs ==="
for u in root deploy; do echo "--- $u"; sudo crontab -l -u "$u" 2>/dev/null; done
ls -la /etc/cron.d/

Run monthly. Also consider lynis audit system for a scored report.

Compromise recovery

IF YOU BELIEVE THE SERVER IS COMPROMISED, DO NOT TRY TO CLEAN IT

A competent attacker installs multiple persistence mechanisms — a modified binary, a cron job, a systemd unit, an SSH key, a kernel module. Finding all of them is not realistic. Every "cleaned" server that gets re-compromised was cleaned by someone who was confident they had found everything.

Rebuild on a fresh server. It is faster and it actually works.

Immediate (first 15 minutes)

Do not power off. RAM contents are evidence, and a reboot may trigger a persistence mechanism.
Isolate: at the cloud firewall (so you do not lose your own access via UFW), block all inbound except your IP.
Snapshot the disk for forensics before changing anything.
Assume every secret on that server is compromised.

Assess (next hour)

bash

# SERVER
last -50                                       # who logged in
sudo grep "Accepted" /var/log/auth.log | tail -50
ps auxf                                         # unfamiliar processes
sudo ss -tupn                                   # outbound connections
cat ~/.ssh/authorized_keys /root/.ssh/authorized_keys
sudo crontab -l; crontab -l; ls -la /etc/cron.d/
systemctl list-units --type=service --state=running
sudo find / -newermt "3 days ago" -type f -not -path "/proc/*" -not -path "/sys/*" 2>/dev/null | head -50
sudo debsums -c 2>/dev/null                     # modified package files

Rebuild

Provision a new server.
Apply the full hardening from Levels 3–5.
Generate entirely new SSH keys — the old ones are suspect.
Restore the database from a backup taken before the compromise.
Deploy application code from Git, after reviewing recent commits for injected changes.
Rotate every secret: database passwords, JWT secrets, API keys, OAuth secrets, webhook secrets, CI/CD secrets, provider API tokens.
Point DNS at the new server.
Destroy the old server (after preserving the forensic snapshot).

Notify

Users, if personal data was accessed — most jurisdictions impose a deadline (72 hours under GDPR)
Payment processors, if payment data was involved
Your hosting provider, if their infrastructure was implicated

Post-mortem

Write it down: what the entry point was, how long the attacker had access, what they reached, and — most importantly — what control would have prevented it. Then implement that control. An incident without a post-mortem is an incident you will have twice.

Production Checklist — Level 23

[ ] Monthly security audit script run and reviewed
[ ] sudo ss -tulpn shows only 22, 80, 443 on public interfaces
[ ] SSH: no passwords, no root, AllowUsers set, fail2ban active
[ ] PostgreSQL and Redis verified unreachable from outside
[ ] Docker port bindings audited
[ ] Passwordless sudo scoped to specific absolute paths
[ ] No unexpected setuid binaries or world-writable files
[ ] unattended-upgrades running and verified
[ ] Application runs as non-root; pnpm install never run as root
[ ] Security headers on all responses; CSP rolled out in report-only first
[ ] Rate limiting on auth endpoints
[ ] Passwords hashed with argon2id (or bcrypt cost ≥ 12)
[ ] Every record access scoped to the requesting user (IDOR audit done)
[ ] ValidationPipe with whitelist and forbidNonWhitelisted
[ ] Dependencies audited; CI actions pinned to SHAs
[ ] Lockfile changes reviewed in pull requests
[ ] 2FA on VPS provider, registrar, GitHub/GitLab, and password manager
[ ] Off-site encrypted backups with write-only credentials
[ ] Incident response procedure documented and readable off the server
[ ] I understand that a compromised server gets rebuilt, not cleaned

Next: Level 24 — Production Architecture →

Level 23 — Server Security, Advanced ​

The security model ​

The complete security checklist ​

Access ​

Network ​

System ​

Application ​

Operations ​

Threat scenarios ​

1. SSH brute force ​

2. Credential theft (your laptop or key) ​

3. Stolen API key ​

4. Database exposure ​

5. Redis exposure ​

6. Malicious or compromised dependency ​

7. Web application vulnerability ​

8. Privilege escalation ​

9. Supply chain via CI/CD ​

Nginx security configuration ​

Password hashing ​

Regular audits ​

Compromise recovery ​

Immediate (first 15 minutes) ​

Assess (next hour) ​

Rebuild ​

Notify ​

Post-mortem ​

Production Checklist — Level 23 ​

Level 23 — Server Security, Advanced

The security model

The complete security checklist

Access

Network

System

Application

Operations

Threat scenarios

1. SSH brute force

2. Credential theft (your laptop or key)

3. Stolen API key

4. Database exposure

5. Redis exposure

6. Malicious or compromised dependency

7. Web application vulnerability

8. Privilege escalation

9. Supply chain via CI/CD

Nginx security configuration

Password hashing

Regular audits

Compromise recovery

Immediate (first 15 minutes)

Assess (next hour)

Rebuild

Notify

Post-mortem

Production Checklist — Level 23