Level 8 — Environment Variables and Secrets

Configuration that differs between your laptop, staging, and production, plus the credentials that must never appear in Git. Getting this wrong is the most common cause of both outages and breaches.

Environment variables

An environment variable is a key-value pair in a process's environment. Child processes inherit it. It is the standard way to configure software without changing code.

bash

# SERVER
echo $HOME                       # read one
printenv                         # list all for this shell
env                              # same
DATABASE_URL=postgres://... node app.js    # set for one command only
export NODE_ENV=production       # set for this shell and its children

In Node:

process.env.DATABASE_URL   // string | undefined

EVERY process.env VALUE IS A STRING OR UNDEFINED

process.env.PORT          // "3001", not 3001
process.env.DEBUG         // "false" — which is TRUTHY in JavaScript
Number(process.env.PORT)  // 3001
process.env.DEBUG === 'true'   // the correct boolean check

if (process.env.DEBUG) is true when DEBUG=false. Validate and coerce at startup — see the Zod example below.

Inspecting a running process's environment

bash

# SERVER
sudo cat /proc/$(pgrep -f "dist/main.js" | head -1)/environ | tr '\0' '\n'
pm2 env 0                        # PM2's view of process 0's environment

THIS IS ALSO A SECURITY BOUNDARY

Any user who can read /proc/<pid>/environ sees your database password. On Linux this is restricted to the process owner and root — another concrete reason your app must not run as root, and why other humans should not share the deploy account.

The `.env` file

A .env file is a list of KEY=value lines loaded into process.env at startup — by dotenv, by Nuxt/Nest natively, or by PM2.

bash

# /home/deploy/apps/myapp/.env
NODE_ENV=production
PORT=3001

DATABASE_URL=postgresql://myapp:S0me-Str0ng-P4ss@127.0.0.1:5432/myapp_production
REDIS_URL=redis://:An0ther-Str0ng-P4ss@127.0.0.1:6379/0

JWT_SECRET=8f2a4c9e1b7d3f5a6c8e0b2d4f6a8c0e2b4d6f8a0c2e4b6d8f0a2c4e6b8d0f2a
JWT_EXPIRES_IN=15m
REFRESH_TOKEN_SECRET=1a3c5e7b9d1f3a5c7e9b1d3f5a7c9e1b3d5f7a9c1e3b5d7f9a1c3e5b7d9f1a3c

SMTP_HOST=smtp.postmarkapp.com
SMTP_PORT=587
SMTP_USER=abc123-def456
SMTP_PASS=xyz789-uvw012

STRIPE_SECRET_KEY=sk_live_51Abc...
STRIPE_WEBHOOK_SECRET=whsec_...

CORS_ORIGIN=https://app.example.com

Rules for `.env` on a server

bash

# SERVER
chmod 600 /home/deploy/apps/myapp/.env
chown deploy:deploy /home/deploy/apps/myapp/.env

Rule	Why
Mode `600`	Only the owner can read it. Anything looser and every user on the box has your database password.
Owned by the app user	So the app can read it and nobody else can
Never in Git	Covered in Level 7
Not inside the web root	If it were under a directory Nginx serves, a misconfiguration could serve it over HTTP
Backed up separately	It is not in Git, so it is not in your code backup. If the server dies, you need it. (Level 22)

THE .env FILE IS NOT IN GIT — SO IT IS NOT IN YOUR BACKUP

This catches people during disaster recovery. You restore the database, re-clone the repo, and then discover you no longer know the JWT secret — invalidating every session — or the Stripe webhook secret.

Store production .env contents in a password manager (1Password, Bitwarden) as a secure note, or in a dedicated secret manager. Update it whenever you change a value. Verify it during your restore drill.

`.env.example` — commit this one

bash

# .env.example — COMMITTED to Git
NODE_ENV=development
PORT=3001

DATABASE_URL=postgresql://user:password@localhost:5432/myapp_dev
REDIS_URL=redis://localhost:6379/0

JWT_SECRET=generate-with-openssl-rand-hex-32
JWT_EXPIRES_IN=15m

SMTP_HOST=
SMTP_PORT=587
SMTP_USER=
SMTP_PASS=

STRIPE_SECRET_KEY=sk_test_...
CORS_ORIGIN=http://localhost:3000

Real keys, fake values. It documents what the application needs, so a new developer (or you, in six months, rebuilding from scratch) knows exactly what to fill in.

KEEP .env.example IN SYNC — ENFORCE IT

Add a CI check that fails if .env keys and .env.example keys diverge. Missing an environment variable in production is a classic cause of a deploy that starts and then 500s on the first request.

Multiple environments

File	Purpose	In Git?
`.env`	Local development	❌ No
`.env.example`	Template of required keys	✅ Yes
`.env.production`	Production values	❌ Never
`.env.staging`	Staging values	❌ Never
`.env.test`	Test values (usually harmless)	⚠️ Only if it contains no real credentials

"IT'S ONLY STAGING"

Staging databases routinely contain a copy of production data, staging often shares an API key with production, and staging is usually less monitored. Treat staging secrets with production discipline — attackers certainly do.

Better still: staging should have its own credentials for everything, so a staging leak does not touch production.

Build-time vs runtime vs public — the critical distinction

This is where security accidents happen.

Category	Read when	Ends up in	Example	Secret?
Build-time	During `pnpm build`	Baked into the JS bundle	`NUXT_PUBLIC_API_BASE`	❌ Never
Runtime (server)	When the process starts	Memory only	`DATABASE_URL`	✅ Yes
Public frontend	In the browser	Visible to every user	`NUXT_PUBLIC_SITE_URL`	❌ Never
CI secrets	During the pipeline	CI logs (masked)	`SSH_PRIVATE_KEY`	✅ Yes

ANY VARIABLE PREFIXED FOR CLIENT USE IS PUBLIC — PERMANENTLY

In Nuxt, runtimeConfig.public and NUXT_PUBLIC_* variables are serialised into the HTML sent to every visitor. Same for NEXT_PUBLIC_* in Next.js and VITE_* in Vite.

// nuxt.config.ts
export default defineNuxtConfig({
  runtimeConfig: {
    // Server-only — NOT sent to the browser
    apiSecret: process.env.API_SECRET,
    databaseUrl: process.env.DATABASE_URL,

    public: {
      // SENT TO EVERY BROWSER — view-source will show these
      apiBase: process.env.NUXT_PUBLIC_API_BASE,
      siteUrl: process.env.NUXT_PUBLIC_SITE_URL,
    },
  },
})

Putting a secret in public publishes it to the internet. Worse, if it was there at build time, it is compiled into the JS bundle — rotating the secret afterwards requires a rebuild and redeploy, and the old bundle may still be cached in CDNs and browsers.

Verify before every release:

bash

grep -ri "sk_live\|secret\|password" .output/public/ | head
curl -s https://app.example.com | grep -o '__NUXT__.*' | head -c 2000

NUXT RUNTIME CONFIG IS OVERRIDABLE AT RUNTIME

runtimeConfig values can be replaced by environment variables at process start without rebuilding, using the NUXT_-prefixed uppercase snake-case name:

runtimeConfig.apiSecret ← NUXT_API_SECRET
runtimeConfig.public.apiBase ← NUXT_PUBLIC_API_BASE

This means you can build one artifact and deploy it to staging and production with different config — a significant operational win. Do not use process.env directly in Nuxt components; use useRuntimeConfig() so this works.

Generating strong secrets

bash

# Any of these — 256 bits of entropy
openssl rand -hex 32
openssl rand -base64 32
head -c 32 /dev/urandom | base64
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

# A password safe for use inside a connection string URL (no @ : / ? # &)
openssl rand -base64 32 | tr -d '/+=' | head -c 32

URL-UNSAFE CHARACTERS BREAK CONNECTION STRINGS

postgresql://user:p@ss@host/db — the parser sees the first @ as the host separator and fails with a confusing error. Either generate passwords without @ : / ? # & %, or URL-encode them (@ → %40).

The tr -d '/+=' above removes the base64 characters that cause trouble.

Rules for secrets:

Rule	Reason
≥ 32 bytes of entropy for signing keys	Brute force must be infeasible
Different secret per environment	A staging leak must not affect production
Different secret per purpose	JWT signing ≠ refresh tokens ≠ session encryption ≠ cookie secret
Generated by a CSPRNG, never by a human	Human-chosen "secrets" are guessable
Never logged	Scrub them from error reports and log lines

NEVER USE A DEFAULT OR PLACEHOLDER SECRET

const secret = process.env.JWT_SECRET || 'dev-secret';   // ❌ CATASTROPHIC

If JWT_SECRET is missing in production — a typo in the .env, a variable not passed through by PM2 — the app silently signs tokens with 'dev-secret'. Anyone who has ever seen your source can now forge an admin token.

Fail loudly instead:

const secret = process.env.JWT_SECRET;
if (!secret) throw new Error('JWT_SECRET is required');

A crash at startup is infinitely better than a silent authentication bypass.

Validating environment at startup

Fail fast, at boot, with a clear message — never at 3 a.m. on a user's request.

// src/config/env.ts
import { z } from 'zod';

const envSchema = z.object({
  NODE_ENV: z.enum(['development', 'test', 'production']),
  PORT: z.coerce.number().int().positive().default(3001),

  DATABASE_URL: z.string().url().startsWith('postgresql://'),
  REDIS_URL: z.string().url().startsWith('redis://'),

  JWT_SECRET: z.string().min(32, 'JWT_SECRET must be at least 32 characters'),
  JWT_EXPIRES_IN: z.string().default('15m'),
  REFRESH_TOKEN_SECRET: z.string().min(32),

  SMTP_HOST: z.string().min(1),
  SMTP_PORT: z.coerce.number().int(),
  SMTP_USER: z.string().min(1),
  SMTP_PASS: z.string().min(1),

  CORS_ORIGIN: z.string().url(),
});

const parsed = envSchema.safeParse(process.env);

if (!parsed.success) {
  console.error('❌ Invalid environment configuration:');
  console.error(parsed.error.flatten().fieldErrors);
  process.exit(1);
}

export const env = parsed.data;

// main.ts — import this FIRST, before anything else
import { env } from './config/env';

THIS SINGLE FILE PREVENTS AN ENTIRE CATEGORY OF OUTAGE

The failure mode it eliminates: you add a feature needing STRIPE_SECRET_KEY, deploy, and the app starts fine — then crashes for real users when they hit checkout. With validation, the deploy fails immediately with STRIPE_SECRET_KEY: Required, PM2 keeps the previous version running, and nobody notices.

Type-safety is a bonus: env.PORT is number, not string | undefined.

Passing environment variables to PM2

Three approaches; only two are correct.

❌ Wrong — relying on shell exports

bash

export DATABASE_URL=...
pm2 start app.js

The variable exists only in that shell. After a reboot, pm2 resurrect starts the app without it.

✅ Option A — PM2 reads the `.env` file

// ecosystem.config.cjs
require('dotenv').config({ path: '/home/deploy/apps/myapp/.env' });

module.exports = {
  apps: [{
    name: 'api',
    script: 'dist/main.js',
    cwd: '/home/deploy/apps/myapp/backend',
    env: {
      NODE_ENV: 'production',
      PORT: 3001,
      DATABASE_URL: process.env.DATABASE_URL,
      JWT_SECRET: process.env.JWT_SECRET,
      // ...
    },
  }],
};

✅ Option B — the app loads its own `.env` (simpler)

// ecosystem.config.cjs
module.exports = {
  apps: [{
    name: 'api',
    script: 'dist/main.js',
    cwd: '/home/deploy/apps/myapp/backend',
    env: { NODE_ENV: 'production' },
  }],
};

with NestJS loading it:

ConfigModule.forRoot({ isGlobal: true, envFilePath: '.env' });

Option B keeps secrets out of PM2's saved process list (~/.pm2/dump.pm2), which is a real advantage — that file is 600 but it is one more copy of your secrets on disk.

pm2 restart DOES NOT RELOAD ENVIRONMENT VARIABLES

Changing .env and running pm2 restart api leaves the old values in place. PM2 caches the environment from when the process was first created.

bash

pm2 restart api --update-env     # ✅ re-reads the environment
pm2 reload  api --update-env     # ✅ zero-downtime version
pm2 delete api && pm2 start ecosystem.config.cjs   # ✅ guaranteed clean

"I updated the database password and the app still uses the old one" is this, every time. Verify with pm2 env 0.

CI/CD secrets

Your pipeline needs credentials to reach the server. These live in the CI platform's encrypted store, never in the workflow file.

GitHub Secrets

Repository → Settings → Secrets and variables → Actions.

Scope	Use for
Repository secrets	Available to all workflows in the repo
Environment secrets	Scoped to a named environment (`production`) with optional approval gates and branch restrictions
Organization secrets	Shared across repos, with a repo allowlist

Typical set for deployment:

Secret	Contents
`SSH_PRIVATE_KEY`	Private key whose public half is in `deploy`'s `authorized_keys`
`SSH_HOST`	`203.0.113.10`
`SSH_USER`	`deploy`
`SSH_PORT`	`22`
`DEPLOY_PATH`	`/home/deploy/apps/myapp`
`SSH_KNOWN_HOSTS`	Output of `ssh-keyscan 203.0.113.10`

yaml

- name: Deploy
  env:
    HOST: ${{ secrets.SSH_HOST }}
  run: ssh $HOST "..."

USE ENVIRONMENT SECRETS WITH PROTECTION RULES

Create a production environment (Settings → Environments) and attach the deploy secrets to it, with "Required reviewers" enabled. Now a deployment to production needs a human click, and a compromised branch cannot deploy on its own.

CI SECRETS ARE MASKED, NOT HIDDEN

GitHub replaces secret values with *** in logs. This fails when:

The value is transformed — base64-encoded, uppercased, JSON-embedded
It is printed character by character
A set -x in your script echoes the command line
A tool writes it to a file that a later step uploads as an artifact

Never echo a secret, even to debug. To check a secret exists:

yaml

- run: |
    if [ -z "${{ secrets.SSH_HOST }}" ]; then echo "SSH_HOST is not set"; exit 1; fi
    echo "SSH_HOST is set (length: ${#HOST})"
  env:
    HOST: ${{ secrets.SSH_HOST }}

pull_request FROM A FORK CANNOT ACCESS SECRETS — AND THAT IS DELIBERATE

Anyone can open a PR from a fork. If that workflow could read your secrets, they would simply add a step that prints them. Deployment must trigger on push to a protected branch, never on pull_request.

Never use pull_request_target with a checkout of the PR's code — that combination executes untrusted code with secrets available, and is a well-documented way to lose your entire secret store.

GitLab CI/CD Variables

Settings → CI/CD → Variables.

Option	Meaning
Protected	Only available on protected branches/tags. Enable for all deploy secrets.
Masked	Hidden in job logs (requires ≥8 chars, no newlines, base64 alphabet)
File type	Value is written to a temp file and the variable holds the path — use this for SSH private keys
Environment scope	Restrict to `production`, `staging`, etc.

USE "FILE" TYPE FOR SSH KEYS IN GITLAB

Multi-line values cannot be masked, and writing a key with echo "$KEY" > id mangles newlines. File-type variables solve both:

yaml

script:
  - chmod 600 "$SSH_PRIVATE_KEY"           # the variable IS the path
  - ssh -i "$SSH_PRIVATE_KEY" deploy@$SSH_HOST "..."

Secret rotation

Rotation limits how long a leaked credential is useful. It is also the only real remedy after an incident.

Secret	Routine rotation	Rotate immediately when
Database password	6–12 months	Anyone with access leaves; suspected breach
JWT signing secret	6–12 months	Suspected breach (invalidates all sessions)
API keys (Stripe, SMTP)	Per provider guidance	Anyone with access leaves; key seen in logs
SSH deploy keys	12 months	Team member leaves; server rebuilt
CI/CD secrets	12 months	Any repository compromise
TLS certificates	Every 60 days	Automatic via Certbot

Rotating a database password without downtime

bash

# SERVER
# 1. Set the new password (PostgreSQL applies it to NEW connections only)
sudo -u postgres psql -c "ALTER USER myapp WITH PASSWORD 'new-strong-password';"

# 2. Update .env
nano /home/deploy/apps/myapp/.env

# 3. Reload with the new environment — existing connections finish, new ones use the new password
pm2 reload ecosystem.config.cjs --update-env

# 4. Verify
pm2 logs api --lines 50
curl -fsS http://127.0.0.1:3001/health

ROTATING JWT_SECRET LOGS EVERYONE OUT

Existing tokens were signed with the old secret and become invalid instantly. For a planned rotation, support two secrets during a transition window: sign with the new one, accept either for verification, then drop the old one after your longest token lifetime has elapsed.

const secrets = [process.env.JWT_SECRET, process.env.JWT_SECRET_PREVIOUS].filter(Boolean);

After a suspected compromise, skip the graceful path — invalidate everything immediately.

Rotation checklist

[ ] Generate the new value with a CSPRNG
[ ] Update it at the source (database, API provider)
[ ] Update the server .env
[ ] Update the CI/CD secret store
[ ] Update your password manager record
[ ] Reload the application with --update-env
[ ] Verify the app works with the new value
[ ] Revoke the old value at the source
[ ] Note the date so the next rotation is scheduled

Beyond `.env` files

A .env file on disk is adequate for a single server and one or two people. Consider more when you have several servers, several people, or a compliance requirement.

Approach	Pros	Cons	Fits
`.env` on the server	Simple, no dependency, no network call	Manual sync; on disk in plaintext; hard to audit	1 server, 1–3 people
SOPS + age/GPG	Encrypted secrets committed to Git; versioned and reviewable	Key distribution to manage	Small teams wanting Git history
Doppler / Infisical	Central UI, audit log, sync to servers and CI	Third-party dependency; cost	Growing teams
HashiCorp Vault	Dynamic short-lived credentials, fine-grained policy, full audit	Substantial operational overhead	Larger orgs, compliance
Cloud secret manager	Managed, IAM-integrated	Ties you to that cloud	Already on AWS/GCP

SOPS IS THE BEST NEXT STEP FROM .env

SOPS encrypts only the values in a YAML/JSON/env file, leaving keys readable. You commit the encrypted file to Git — so secrets are versioned, reviewable in diffs, and restored automatically with your repo — and decrypt at deploy time with a key held on the server.

bash

sops -e .env.production > .env.production.enc   # commit this
sops -d .env.production.enc > .env              # on the server at deploy time

It solves the "secrets are not in my backup" problem elegantly, without running new infrastructure.

Auditing for leaked secrets

bash

# LOCAL — scan the full Git history
gitleaks detect --source . --verbose

# Search working tree for common patterns
grep -rIn --exclude-dir={node_modules,.git,.output,dist} \
  -E "(sk_live_|AKIA[0-9A-Z]{16}|-----BEGIN.*PRIVATE KEY-----|password\s*=\s*['\"])" .

# SERVER — did anything end up in the built bundle?
grep -ri "sk_live\|JWT_SECRET\|DATABASE_URL" .output/public/ | head

# SERVER — are secrets in logs?
sudo grep -ri "password\|secret\|token" /var/log/nginx/access.log | head
pm2 logs --lines 1000 --nostream | grep -i "password\|secret" | head

# SERVER — secrets in shell history?
grep -i "password\|secret\|token" ~/.bash_history | head

SECRETS IN URLs END UP IN LOGS

Never put a token in a query string: GET /api/data?token=abc123. It lands in Nginx access logs, browser history, Referer headers sent to third parties, and any proxy in between. Use the Authorization header.

Similarly, make sure your error reporter (Sentry etc.) is configured to scrub Authorization headers, cookies, and request bodies containing password fields.

Production Checklist — Level 8

[ ] .env is 600, owned by deploy, and outside any web-served directory
[ ] .env and .env.* are gitignored; .env.example is committed and current
[ ] No secrets in Git history — verified with gitleaks
[ ] Every secret generated with openssl rand, ≥32 bytes, unique per environment and per purpose
[ ] No || 'default-secret' fallbacks anywhere in the codebase
[ ] Environment validated at startup with Zod (or equivalent); the app refuses to boot on bad config
[ ] Nothing secret is in runtimeConfig.public / NUXT_PUBLIC_*
[ ] Built bundle grepped for secrets before release
[ ] PM2 reloads use --update-env, verified with pm2 env 0
[ ] CI secrets stored in GitHub/GitLab encrypted stores, marked protected/masked
[ ] Deploy workflows trigger on push to a protected branch, never on pull_request from forks
[ ] Production .env backed up in a password manager, and included in the restore drill
[ ] A rotation schedule exists and rotation is part of offboarding
[ ] Tokens are never passed in URLs
[ ] Error reporting scrubs authorization headers and password fields

Next: Level 9 — PostgreSQL →

Level 8 — Environment Variables and Secrets ​

Environment variables ​

Inspecting a running process's environment ​

The .env file ​

Rules for .env on a server ​

.env.example — commit this one ​

Multiple environments ​

Build-time vs runtime vs public — the critical distinction ​

Generating strong secrets ​

Validating environment at startup ​

Passing environment variables to PM2 ​

❌ Wrong — relying on shell exports ​

✅ Option A — PM2 reads the .env file ​

✅ Option B — the app loads its own .env (simpler) ​

CI/CD secrets ​

GitHub Secrets ​

GitLab CI/CD Variables ​

Secret rotation ​

Rotating a database password without downtime ​

Rotation checklist ​

Beyond .env files ​

Auditing for leaked secrets ​

Production Checklist — Level 8 ​

Level 8 — Environment Variables and Secrets

Environment variables

Inspecting a running process's environment

The `.env` file

Rules for `.env` on a server

`.env.example` — commit this one

Multiple environments

Build-time vs runtime vs public — the critical distinction

Generating strong secrets

Validating environment at startup

Passing environment variables to PM2

❌ Wrong — relying on shell exports

✅ Option A — PM2 reads the `.env` file

✅ Option B — the app loads its own `.env` (simpler)

CI/CD secrets

GitHub Secrets

GitLab CI/CD Variables

Secret rotation

Rotating a database password without downtime

Rotation checklist

Beyond `.env` files

Auditing for leaked secrets

Production Checklist — Level 8