Level 14 — Nginx

Nginx is the boundary between the hostile internet and your application. It terminates TLS, routes by hostname, serves static files, compresses responses, rate-limits abuse, and returns a controlled error when your app is down.

Web server vs reverse proxy vs load balancer

Nginx is all three, depending on configuration:

Role	What it does	Directive
Web server	Serves files from disk	root + try_files
Reverse proxy	Forwards requests to a backend	proxy_pass
Load balancer	Distributes across several backends	upstream with multiple server lines
TLS terminator	Decrypts HTTPS, forwards plain HTTP internally	listen 443 ssl

Installing Nginx

# SERVER
sudo apt update
sudo apt install -y nginx
sudo systemctl enable --now nginx
sudo systemctl status nginx
curl -I http://127.0.0.1        # 200 from the default page

# SERVER
sudo ufw allow 'Nginx Full'     # opens 80 and 443
sudo ufw status

Where everything lives

Path	Purpose
/etc/nginx/nginx.conf	Main configuration
/etc/nginx/sites-available/	Site configs (all of them)
/etc/nginx/sites-enabled/	Symlinks to the active ones
/etc/nginx/conf.d/	Additional global config snippets
/etc/nginx/snippets/	Reusable fragments to include
/var/log/nginx/access.log	Every request
/var/log/nginx/error.log	Errors and warnings
/var/www/html	Default document root
/etc/nginx/mime.types	Extension → Content-Type mapping

The sites-available / sites-enabled split lets you write a config without activating it, and disable a site by removing a symlink rather than deleting the file.

# SERVER
sudo rm /etc/nginx/sites-enabled/default    # remove the welcome page

REMOVE THE DEFAULT SITE

It is default_server, so it catches any request whose Host header does not match one of your sites — including direct requests to your IP address. Leaving it means bots see an Nginx welcome page and learn your version. It can also shadow your real sites if you misconfigure server_name.

How Nginx picks a server block

Location matching order — this trips people up constantly:

Modifier	Meaning	Priority
=	Exact match	1 (highest)
^~	Prefix; if matched, skip regex checks	2
~ / ~*	Regex (case-sensitive / insensitive)	3, in file order
(none)	Prefix; longest match wins	4

REGEX LOCATIONS BEAT LONGER PREFIXES

location /_nuxt/ { ... }                       # prefix
location ~* \.(js|css)$ { ... }                # regex — WINS for /_nuxt/entry.js

The regex is checked before plain prefixes, so your /_nuxt/ caching rules are ignored. Use ^~ to stop regex evaluation:

location ^~ /_nuxt/ { ... }

Main configuration

# SERVER
sudo nano /etc/nginx/nginx.conf

user www-data;
worker_processes auto;
worker_rlimit_nofile 65535;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 4096;
    multi_accept on;
    use epoll;
}

http {
    ##
    # Basic
    ##
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    keepalive_requests 1000;
    types_hash_max_size 2048;
    server_tokens off;                 # do not advertise the version

    client_max_body_size 20M;          # max upload size
    client_body_timeout 15s;
    client_header_timeout 15s;
    send_timeout 30s;
    reset_timedout_connection on;

    server_names_hash_bucket_size 64;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging
    ##
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" rt=$request_time urt="$upstream_response_time"';

    access_log /var/log/nginx/access.log main;
    error_log  /var/log/nginx/error.log warn;

    ##
    # Compression
    ##
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_min_length 1024;
    gzip_types
        text/plain text/css text/xml text/javascript
        application/json application/javascript application/xml+rss
        application/rss+xml application/atom+xml image/svg+xml
        font/woff font/woff2 application/font-woff;

    ##
    # Rate limiting zones (used per-location below)
    ##
    limit_req_zone  $binary_remote_addr zone=general:10m rate=30r/s;
    limit_req_zone  $binary_remote_addr zone=login:10m   rate=5r/m;
    limit_conn_zone $binary_remote_addr zone=addr:10m;

    ##
    # Virtual hosts
    ##
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Directive	Why
worker_processes auto	One worker per CPU core
worker_connections 4096	Max concurrent connections per worker
server_tokens off	Hides "nginx/1.24.0" from responses and error pages — one less fingerprint for attackers
sendfile on	Kernel-level file transfer, bypassing user space
client_max_body_size 20M	Default is 1M — uploads larger than this get 413
$request_time / $upstream_response_time in the log format	Lets you find slow requests and distinguish "app was slow" from "network was slow"
gzip_comp_level 6	Above 6 costs noticeably more CPU for marginal size gains

client_max_body_size DEFAULTS TO 1 MB

File upload endpoints fail with 413 Request Entity Too Large and the error appears in Nginx's log, not your application's — so people debug the wrong layer for an hour. Set it globally and override per-location if one endpoint needs more.

DO NOT GZIP ALREADY-COMPRESSED CONTENT

JPEG, PNG, WebP, MP4, and ZIP are already compressed. Gzipping them wastes CPU and can make them slightly larger. The gzip_types list above deliberately excludes them.

Reusable snippets

# SERVER
sudo nano /etc/nginx/snippets/proxy-params.conf

proxy_http_version 1.1;

proxy_set_header Host              $host;
proxy_set_header X-Real-IP         $remote_addr;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host  $host;
proxy_set_header X-Forwarded-Port  $server_port;

# WebSocket upgrade support
proxy_set_header Upgrade    $http_upgrade;
proxy_set_header Connection $connection_upgrade;

proxy_connect_timeout 10s;
proxy_send_timeout    60s;
proxy_read_timeout    60s;

proxy_buffering on;
proxy_buffer_size       16k;
proxy_buffers        8  16k;
proxy_busy_buffers_size 32k;

proxy_redirect off;

WITHOUT proxy_set_header Host $host, NGINX SENDS THE UPSTREAM NAME

The default is proxy_set_header Host $proxy_host, which sends 127.0.0.1:3000. Your application then generates redirects and absolute URLs pointing at 127.0.0.1 — users get sent to their own machine. Always set Host $host.

WITHOUT X-Forwarded-Proto, YOUR APP THINKS IT IS ON HTTP

Nginx terminates TLS and forwards plain HTTP, so the app sees http. Consequences: Secure cookies are not set, OAuth redirect URIs are generated as http://, and any "force HTTPS" middleware creates a redirect loop. X-Forwarded-Proto $scheme plus trust proxy in NestJS (Level 12) fixes it.

Security headers:

# SERVER
sudo nano /etc/nginx/snippets/security-headers.conf

add_header X-Frame-Options            "SAMEORIGIN"                     always;
add_header X-Content-Type-Options     "nosniff"                        always;
add_header Referrer-Policy            "strict-origin-when-cross-origin" always;
add_header Permissions-Policy         "camera=(), microphone=(), geolocation=()" always;
add_header Strict-Transport-Security  "max-age=31536000; includeSubDomains" always;

Header	Protects against
X-Frame-Options: SAMEORIGIN	Clickjacking — your site framed inside an attacker's page
X-Content-Type-Options: nosniff	Browsers guessing a content type and executing an upload as JS
Referrer-Policy	Leaking full URLs (with tokens) to third parties
Permissions-Policy	Unwanted access to camera/mic/location
Strict-Transport-Security	Downgrade attacks; forces HTTPS for future visits

add_header IS NOT INHERITED IF THE CHILD BLOCK HAS ITS OWN

If a server block sets headers and a location inside it also calls add_header, the server-level headers are discarded for that location. This silently removes your security headers from exactly the paths you care about.

Always include the snippet in every location that adds its own headers, or use nginx-extras' more_set_headers.

HSTS IS EFFECTIVELY IRREVERSIBLE

max-age=31536000 tells browsers to refuse HTTP for one year. If your certificate later expires or breaks, users get an error page with no click-through option, and you cannot fix it by reverting the header — browsers have already cached it.

Start with max-age=300 (5 minutes). Once HTTPS has been stable for a week, raise it. Do not add preload unless you are certain — removal from the browser preload list takes months.

WebSocket connection mapping:

# SERVER
sudo nano /etc/nginx/conf.d/websocket.conf

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

proxy_set_header Connection "upgrade" UNCONDITIONALLY BREAKS KEEPALIVE

Hardcoding it sends Connection: upgrade on every request, including normal HTTP ones, which confuses upstreams and disables connection reuse. The map sets it only when the client actually requested an upgrade.

The frontend site — app.example.com

# SERVER
sudo nano /etc/nginx/sites-available/app.example.com

upstream nuxt_backend {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=10s;
    keepalive 32;
}

# HTTP → HTTPS redirect, plus the ACME challenge path
server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;

    # Certbot must reach this over plain HTTP
    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name app.example.com;

    # Certbot fills these in
    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    include snippets/security-headers.conf;

    access_log /var/log/nginx/app.example.com.access.log main;
    error_log  /var/log/nginx/app.example.com.error.log warn;

    root /home/deploy/apps/myapp/frontend/.output/public;

    # Hashed build assets — cache aggressively
    location ^~ /_nuxt/ {
        expires 1y;
        add_header Cache-Control "public, max-age=31536000, immutable" always;
        include snippets/security-headers.conf;
        access_log off;
        try_files $uri =404;
    }

    # Other static files — try disk first, then fall through to SSR
    location ~* \.(ico|css|js|gif|jpe?g|png|webp|avif|svg|woff2?|ttf|eot|map)$ {
        expires 30d;
        add_header Cache-Control "public, max-age=2592000" always;
        include snippets/security-headers.conf;
        access_log off;
        try_files $uri @nuxt;
    }

    location = /robots.txt  { try_files $uri @nuxt; access_log off; }
    location = /favicon.ico { try_files $uri @nuxt; access_log off; log_not_found off; }

    # Everything else → Nuxt SSR
    location / {
        limit_req zone=general burst=50 nodelay;
        limit_conn addr 20;
        include snippets/proxy-params.conf;
        include snippets/security-headers.conf;
        proxy_pass http://nuxt_backend;
    }

    location @nuxt {
        include snippets/proxy-params.conf;
        proxy_pass http://nuxt_backend;
    }
}

SERVING _nuxt/ FROM DISK IS A LARGE WIN

Without the location ^~ /_nuxt/ block, every JS and CSS request is proxied to the Node process. Nginx serves files from disk with sendfile at a fraction of the cost, and it takes that entire load off your app. On a page with 30 assets, that is 30 requests your Node process never sees.

Nginx needs traversal permission on the whole path — /home, /home/deploy, etc. all need 755 (Level 3). Verify:

sudo -u www-data stat /home/deploy/apps/myapp/frontend/.output/public/

http2 on; IS THE MODERN SYNTAX

listen 443 ssl http2; is deprecated in Nginx 1.25.1+ and emits a warning. Ubuntu 24.04 ships 1.24, where the old form still works; if you upgrade Nginx, switch to the separate http2 on; directive.

The API site — api.example.com

# SERVER
sudo nano /etc/nginx/sites-available/api.example.com

upstream nest_backend {
    # ip_hash gives sticky sessions — required for Socket.IO polling
    # with multiple upstream servers. With ONE upstream + PM2 cluster,
    # stickiness must be handled inside the app instead.
    server 127.0.0.1:3001 max_fails=3 fail_timeout=10s;
    keepalive 32;
}

server {
    listen 80;
    listen [::]:80;
    server_name api.example.com;

    location /.well-known/acme-challenge/ { root /var/www/certbot; }
    location / { return 301 https://$host$request_uri; }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name api.example.com;

    ssl_certificate     /etc/letsencrypt/live/api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    include snippets/security-headers.conf;

    access_log /var/log/nginx/api.example.com.access.log main;
    error_log  /var/log/nginx/api.example.com.error.log warn;

    client_max_body_size 20M;

    # Socket.IO — long-lived connections, no buffering
    location /socket.io/ {
        include snippets/proxy-params.conf;
        proxy_pass http://nest_backend;

        proxy_read_timeout    7d;
        proxy_send_timeout    7d;
        proxy_connect_timeout 10s;
        proxy_buffering off;
        proxy_cache off;

        # Explicit upgrade headers (also in proxy-params, kept for clarity)
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    # Stricter rate limit on authentication endpoints
    location ~ ^/api/(auth|login|register|password-reset) {
        limit_req zone=login burst=5 nodelay;
        include snippets/proxy-params.conf;
        include snippets/security-headers.conf;
        proxy_pass http://nest_backend;
    }

    location / {
        limit_req zone=general burst=40 nodelay;
        limit_conn addr 30;
        include snippets/proxy-params.conf;
        include snippets/security-headers.conf;
        proxy_pass http://nest_backend;
    }

    location = /health {
        include snippets/proxy-params.conf;
        proxy_pass http://nest_backend;
        access_log off;
    }
}

WebSocket configuration explained

THE THREE THINGS WEBSOCKETS NEED FROM NGINX

1. proxy_http_version 1.1 — HTTP/1.0 has no upgrade mechanism. Without it, the handshake fails silently.
2. Upgrade and Connection headers — these are hop-by-hop headers that proxies strip by default. You must re-add them.
3. A long proxy_read_timeout — the default is 60 seconds. A WebSocket that is idle for 60s gets disconnected, and your client reconnects in a loop. 7d effectively disables it.

Also proxy_buffering off for Socket.IO: buffering delays messages until a buffer fills, which destroys real-time behaviour.

Symptoms of each: handshake 400s (missing 1 or 2), disconnects exactly every 60 seconds (missing 3), messages arriving in bursts (buffering on).

STICKY SESSIONS FOR SOCKET.IO POLLING

Socket.IO's default transport starts with HTTP long-polling: several requests that must all reach the same process. With PM2 cluster mode, Node's round-robin distributes them across workers and you get Session ID unknown errors.

Solutions, in order of preference:

1. Force WebSocket-only transport — transports: ['websocket'] on both client and server. No multi-request handshake, so no affinity needed. Simplest and works well in 2026.
2. @socket.io/sticky in the app with PM2 cluster mode.
3. Run several PM2 fork-mode instances on different ports and use ip_hash in the Nginx upstream.

And in every case: the Redis adapter is still required so broadcasts reach clients on other workers (Level 10).

Enabling sites

# SERVER
sudo ln -s /etc/nginx/sites-available/app.example.com /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/api.example.com /etc/nginx/sites-enabled/

sudo nginx -t              # ⭐ ALWAYS test before reloading
sudo systemctl reload nginx

ALWAYS RUN nginx -t BEFORE RELOADING

sudo nginx -t
# nginx: configuration file /etc/nginx/nginx.conf test is successful

A syntax error plus systemctl restart nginx means Nginx fails to start and your entire site is down until you fix it. nginx -t catches it while the old config is still serving traffic.

Make it one command so you cannot forget:

sudo nginx -t && sudo systemctl reload nginx

reload NEVER DROPS A CONNECTION

reload starts new workers with the new config, lets old workers finish their current requests, then retires them. Zero downtime.

restart stops the master process — brief downtime, and if the config is broken it does not come back.

Use reload for config changes. Use restart only after an Nginx package upgrade.

Caching static content

location ^~ /_nuxt/ {
    expires 1y;
    add_header Cache-Control "public, max-age=31536000, immutable" always;
}

This is safe only because Nuxt hashes filenames (entry.a1b2c3.js). Content change → new hash → new URL. immutable tells the browser not to even send a revalidation request.

NEVER CACHE HTML AGGRESSIVELY

location / {
    expires 1y;    # ❌ CATASTROPHIC
}

Users get a cached HTML page referencing JS bundles that no longer exist. The site breaks for them and a hard refresh is the only fix — which you cannot ask thousands of users to do. Rolling back the deploy does not help, because the bad HTML is already in their browser cache.

HTML must be no-cache (or short-lived). Only hashed assets get long caching.

Content	Cache-Control
Hashed JS/CSS (/_nuxt/)	public, max-age=31536000, immutable
Images, fonts	public, max-age=2592000 (30 days)
HTML / SSR pages	no-cache or public, max-age=0, must-revalidate
API JSON	no-store (or a short max-age for public data)
Anything authenticated	private, no-store

Rate limiting

# In http {} — define the zone
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;

# In location {} — apply it
location /api/auth/login {
    limit_req zone=login burst=5 nodelay;
}

Part	Meaning
$binary_remote_addr	Key by client IP (binary form uses less memory)
zone=login:10m	Named shared zone, 10 MB ≈ 160,000 IPs
rate=5r/m	5 requests per minute sustained
burst=5	Allow a burst of 5 above the rate
nodelay	Serve the burst immediately rather than queueing it

RATE LIMITING KEYS ON THE CLIENT IP — WHICH MUST BE CORRECT

If you later add Cloudflare or another proxy in front, $remote_addr becomes the proxy's IP and you rate-limit everyone as one client — a single abusive user locks out all traffic.

With Cloudflare, use real_ip so Nginx recovers the true client IP:

set_real_ip_from 173.245.48.0/20;    # ... all Cloudflare ranges
real_ip_header CF-Connecting-IP;

Never trust X-Forwarded-For from an untrusted source — clients can set it to anything.

Nginx returns 503 by default when limiting. 429 is more correct:

limit_req_status 429;
limit_conn_status 429;

PAIR RATE LIMITING WITH fail2ban

Nginx logs rejections to the error log. The nginx-limit-req jail (Level 4) bans IPs that trip the limit repeatedly, so the traffic stops at the firewall instead of costing you an Nginx worker.

Logs

# SERVER
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log
sudo tail -f /var/log/nginx/api.example.com.error.log

Useful analysis:

# Top client IPs
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Status code distribution
sudo awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# All 5xx responses
sudo grep -E ' (50[0-9]) ' /var/log/nginx/access.log | tail -50

# Slowest requests (rt= is at the end of our log_format)
sudo awk '{print $NF, $0}' /var/log/nginx/access.log | sort -rn | head -20

# Most requested paths
sudo awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Log rotation is configured by the package at /etc/logrotate.d/nginx — daily, 14 days retained, compressed. Verify it is working:

ls -la /var/log/nginx/
sudo logrotate -d /etc/logrotate.d/nginx    # dry run

Troubleshooting

502 Bad Gateway

Nginx could not connect to your upstream. The problem is almost always your application, not Nginx.

# SERVER — diagnose in order
sudo tail -20 /var/log/nginx/error.log      # the specific reason
pm2 list                                     # is the app running?
sudo ss -tulpn | grep -E '3000|3001'         # is it listening? on what address?
curl -i http://127.0.0.1:3001/health         # can Nginx's host reach it?
pm2 logs api --err --lines 50 --nostream     # why did it die?

Error log message	Cause	Fix
connect() failed (111: Connection refused)	App not running or wrong port	Start the app; check proxy_pass port
connect() failed (113: No route to host)	Wrong address	Use 127.0.0.1
no live upstreams	All upstreams marked failed by max_fails	Fix the app; upstreams recover after fail_timeout
upstream prematurely closed connection	App crashed mid-request	Check app logs for the exception
(13: Permission denied)	SELinux/AppArmor, or socket permissions	Check sudo journalctl -u nginx

THE 30-SECOND 502 DIAGNOSIS

curl -i http://127.0.0.1:3001/health

Works → the app is fine; the problem is Nginx configuration (wrong port, wrong upstream). Fails → the problem is your application. Go read pm2 logs.

This single command eliminates half the possibilities immediately.

504 Gateway Timeout

The app accepted the connection but did not respond within proxy_read_timeout (60s).

# SERVER
pm2 monit                     # is the process pegged at 100% CPU?
sudo -u postgres psql -c "SELECT pid, now()-query_start AS d, left(query,80) FROM pg_stat_activity WHERE state!='idle' ORDER BY d DESC;"

Causes: a slow database query (missing index), an external API call with no timeout, an infinite loop, or a blocked event loop.

RAISING proxy_read_timeout IS NOT A FIX

It converts a 504 into a request that ties up a worker for five minutes. Under load, all workers end up blocked and the whole site stops responding. Fix the slow operation. If work genuinely takes minutes, move it to a background queue (Level 10) and return a job ID immediately.

403 Forbidden on static files

# SERVER
sudo tail /var/log/nginx/error.log
# "Permission denied" or "directory index of ... is forbidden"

sudo -u www-data stat /home/deploy/apps/myapp/frontend/.output/public/
namei -l /home/deploy/apps/myapp/frontend/.output/public/index.html

namei -l shows the permissions of every component of the path — instantly revealing which directory is blocking www-data. Every directory needs x for others; /home/deploy must be 755 (Level 3).

413 Request Entity Too Large

client_max_body_size is too small. Set it in http, server, or the specific location.

Wrong site being served

# SERVER
sudo nginx -T | grep -A2 server_name          # dump the full effective config
curl -H "Host: app.example.com" http://127.0.0.1 -I

Usually: the default_server block is catching the request because server_name does not match, or you forgot the sites-enabled symlink, or you edited sites-available and never reloaded.

nginx -T (CAPITAL T) DUMPS THE ENTIRE RESOLVED CONFIGURATION

Including every included file. When you cannot work out which directive is winning, this is the answer.

Production Checklist — Level 14

• [ ] Nginx installed, enabled, and starting on boot
• [ ] Default site removed from sites-enabled
• [ ] server_tokens off
• [ ] client_max_body_size set appropriately for uploads
• [ ] gzip enabled with a sensible gzip_types list
• [ ] Custom log_format including $request_time and $upstream_response_time
• [ ] Per-site access and error logs
• [ ] proxy_set_header Host $host on every proxy location
• [ ] X-Forwarded-Proto $scheme set, and trust proxy configured in NestJS
• [ ] WebSocket upgrade map defined and used
• [ ] proxy_read_timeout 7d and proxy_buffering off on /socket.io/
• [ ] Static /_nuxt/ served from disk with ^~ and long cache headers
• [ ] HTML not cached
• [ ] Security headers included in every location that has its own add_header
• [ ] HSTS starts at a low max-age until HTTPS is proven stable
• [ ] Rate limiting on general traffic and stricter limits on auth endpoints
• [ ] limit_req_status 429
• [ ] sudo nginx -t passes and is run before every reload
• [ ] reload used for config changes, not restart
• [ ] sudo -u www-data stat <static path> succeeds
• [ ] Log rotation verified
• [ ] I know that curl http://127.0.0.1:3001/health is the first step in diagnosing a 502

---

Next: Level 15 — DNS →