diff --git a/plans/homelab-release-runbook.md b/plans/homelab-release-runbook.md new file mode 100644 index 0000000..0a598be --- /dev/null +++ b/plans/homelab-release-runbook.md @@ -0,0 +1,264 @@ +# Homelab Release Runbook — New Project → Local Host → Public + +**Audience:** Work Lumen (and future me). This is the authoritative, battle-tested +procedure for taking a new alpha app from zero to (a) running on TrueNAS for LAN +testing, then (b) publicly reachable over HTTPS — using only Gitea Actions + frp + +IONOS Apache. No expensive per-project DevOps rediscovery. + +**Status:** Proven twice end-to-end (InspectFlow, CannaManage). Supersedes the +stale [`homelab-proxy-architecture.md`](homelab-proxy-architecture.md:1) (that doc +describes a WireGuard plan that was **never used** — the VPS is OpenVZ and cannot +run WireGuard; we use **frp** instead). + +--- + +## 0. The mental model (read this once) + +``` + PUBLIC (optional, additive) + browser ─HTTPS─► IONOS Apache ─ProxyPass─► VPS frps ─frp tunnel─► TrueNAS frpc ─► frontend:PORT + (82.165.206.45, Let's Encrypt) (85.214.154.199) (192.168.188.119) + + LOCAL-FIRST (always works on its own) + LAN browser ──────────────────────────────────────────────────► TrueNAS 192.168.188.119:HOSTPORT +``` + +Two **decoupled** phases: + +1. **Local phase** — `git push` to `main` → Gitea Actions self-hosted runner on + TrueNAS builds + runs the stack in-place. App is live at + `http://192.168.188.119:`. **Zero VPS / IONOS involvement.** This is + where every project starts and stays during early alpha. + +2. **Public phase** — purely *additive*. Run [`homelab-publish.sh`](../scripts/homelab-publish.sh:1) + once to wire the frp tunnel + IONOS vhost + Let's Encrypt cert. Nothing about + the app changes; you only add a tunnel from `frontend:PORT` out to the world. + To "unpublish", stop the frpc proxy block — local phase keeps working. + +**Why frontend-only tunnelling:** the Next.js frontend proxies `/api/backend/*` +to the backend server-side (see the route.ts catch-all proxy in the template), so +only the frontend host port needs to be exposed. The backend and db never touch +the public path. + +--- + +## 1. Fixed infrastructure (already exists — do not rebuild) + +| Component | Where | Detail | +|---|---|---| +| Gitea | TrueNAS `192.168.188.119:30008` | source of truth; `http://192.168.188.119:30008/` | +| act_runner | TrueNAS container, **instance-level** | auto-picks-up **any** new repo — no per-repo runner registration needed | +| frps (frp server) | VPS `85.214.154.199:7000` | token auth; bind ports in the 300xx range | +| frpc (frp client) | TrueNAS `systemd` service | config `/mnt/VM_SSD_Pool/frp/frpc.toml`; reload `systemctl restart frpc` | +| IONOS Apache | `82.165.206.45` (apex `plate-software.de`) | terminates HTTPS, ProxyPass to VPS frps; acme.sh for certs | +| DNS | IONOS | each subdomain `A` record → **82.165.206.45** (the IONOS box, NOT the VPS) | + +SSH aliases assumed: `ssh truenas`, `ssh ionos`, `ssh vps` (confirm in `~/.ssh/config`). + +--- + +## 2. Port & subdomain registry ⚠️ SINGLE SOURCE OF TRUTH — update on every new project + +**frp remote ports (on VPS frps).** Each project gets exactly one. Allocate the +next free one and record it here *before* writing any config. + +| Project | frp remotePort | subdomain | frontend hostPort (LAN) | backend hostPort (LAN, debug) | Status | +|---|---|---|---|---|---| +| gitea | 30008 | (direct) | — | — | live | +| inspectflow | 30009 | inspectflow.plate-software.de | (Caddy-fronted) | — | live | +| cannamanage | 30010 | cannamanage.plate-software.de | 3000 | 8081→8080 | live | +| **— next free —** | **30011** | — | — | — | — | + +**Allocation rules:** +- **frp remotePort**: strictly increment from the table. Next = **30011**. +- **frontend hostPort**: each compose project runs on its **own bridge network**, so + internal ports (3000 / 8080 / 5432) never collide across stacks. Only *published* + host ports can clash. Pick a unique LAN host port per project (e.g. 3000, 3001, …) + for the frontend; keep backend published only if you need LAN debugging (use a + unique port like 8081, 8082, …). **db must NOT publish a host port** (see §6). +- **subdomain**: `.plate-software.de`, A-record → 82.165.206.45. + +--- + +## 3. NEW PROJECT — local phase (every project does this) + +Goal: app live at `http://192.168.188.119:` via push-to-deploy. + +1. **Create the repo from the template.** + - Generate from `homelab-app-template` in Gitea (or copy its `.gitea/`, + `docker-compose.truenas.yml`, frontend proxy route, `.env.example`). + - The template ships a working `.gitea/workflows/deploy.yml` and TrueNAS compose + override. You only fill in placeholders. + +2. **Fill placeholders** (template uses `__PROJECT__`, `__FRONTEND_PORT__`, + `__BACKEND_PORT__`): + - compose `-p` project name = `__PROJECT__` + - frontend `ports: "__FRONTEND_PORT__:3000"` + - backend `ports: "__BACKEND_PORT__:8080"` (or drop entirely if no LAN debug) + - container names `__PROJECT__-frontend`, `__PROJECT__-backend`, `__PROJECT__-db` + +3. **Set Gitea Actions secrets** (repo → Settings → Actions → Secrets). Minimum: + - `AUTH_SECRET` — `openssl rand -base64 32` (NextAuth) + - `JWT_SECRET` — `openssl rand -base64 32` (backend, if it issues JWTs) + - `DB_PASSWORD` — `openssl rand -base64 24` + ```bash + # quick generate + for s in AUTH_SECRET JWT_SECRET DB_PASSWORD; do echo "$s=$(openssl rand -base64 32)"; done + ``` + +4. **Push to `main`.** The instance-level act_runner picks it up automatically. + Watch the run: + ```bash + # list recent runs via Gitea API (token in ~/.config or use web UI) + curl -s -H "Authorization: token $GITEA_TOKEN" \ + "http://192.168.188.119:30008/api/v1/repos///actions/tasks" | jq '.workflow_runs[:5]' + ``` + Or just open `http://192.168.188.119:30008///actions`. + +5. **Verify locally:** + ```bash + curl -I http://192.168.188.119:/ # expect 200 or 307→/login + ``` + +✅ At this point the app is fully usable on the LAN. You can stop here for as long +as you want. The next section is **optional** and additive. + +--- + +## 4. GO PUBLIC — the switch (run once per project, when ready) + +Everything below is automated by [`homelab-publish.sh`](../scripts/homelab-publish.sh:1). +Run it and skip to §5 to verify. The manual steps are documented here so the script +is auditable and so you can debug if a step fails. + +**Prereq:** DNS A-record `.plate-software.de → 82.165.206.45` exists and +has propagated (`dig +short .plate-software.de` must return 82.165.206.45). + +### 4a. frp tunnel (TrueNAS frpc → VPS frps) +Append a proxy block to `/mnt/VM_SSD_Pool/frp/frpc.toml` on TrueNAS: +```toml +[[proxies]] +name = "__PROJECT__" +type = "tcp" +localIP = "127.0.0.1" +localPort = # the LAN host port the frontend publishes +remotePort = # from the registry, e.g. 30011 +``` +Reload: `ssh truenas 'systemctl restart frpc'`. + +### 4b. IONOS Apache vhost +Create `/etc/apache2/sites-available/.plate-software.de.conf`: +```apache + + ServerName .plate-software.de + Alias /.well-known/acme-challenge/ /var/www/html/.well-known/acme-challenge/ + ProxyPass /.well-known/acme-challenge/ ! + RewriteEngine On + RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/ + RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L] + + + ServerName .plate-software.de + SSLEngine on + SSLCertificateFile /root/.acme.sh/.plate-software.de_ecc/.plate-software.de.cer + SSLCertificateKeyFile /root/.acme.sh/.plate-software.de_ecc/.plate-software.de.key + SSLCertificateChainFile /root/.acme.sh/.plate-software.de_ecc/ca.cer + ProxyPreserveHost On + ProxyPass / http://85.214.154.199:/ + ProxyPassReverse / http://85.214.154.199:/ + RequestHeader set X-Forwarded-Proto https + RequestHeader set X-Real-IP %{REMOTE_ADDR}s + +``` +Enable HTTP vhost first (needed for the ACME challenge), then issue the cert, then +the 443 vhost will have a valid cert to load: +```bash +a2ensite .plate-software.de.conf +apache2ctl configtest && systemctl reload apache2 +``` + +### 4c. Let's Encrypt cert (acme.sh) — ⚠️ force the right CA +acme.sh defaults to **ZeroSSL**, which stalled on us (order stuck at +`retryafter=86400`). Always pin Let's Encrypt: +```bash +acme.sh --set-default-ca --server letsencrypt +acme.sh --issue -d .plate-software.de -w /var/www/html --server letsencrypt +``` +Then reload Apache so the 443 vhost picks up the freshly issued cert: +```bash +apache2ctl configtest && systemctl reload apache2 +``` + +--- + +## 5. Verify public + +⚠️ Your workstation DNS may be stale-cached. Force-resolve to be sure you're +testing the real path, not a cache: +```bash +curl -I --resolve .plate-software.de:443:82.165.206.45 \ + https://.plate-software.de/ # expect 307 → /login, valid TLS + +# full smoke (login → an authed endpoint) +curl -s --resolve .plate-software.de:443:82.165.206.45 \ + -c /tmp/cj https://.plate-software.de/api/auth/... # adapt per app +``` +Check the latest deploy run is green and the db port is closed: +```bash +ssh truenas 'docker exec -db sh -c "netstat -tln | grep 5432 || true"' # internal only +ss -tln | grep 5432 # on TrueNAS host: should be EMPTY (no host publish) +``` + +--- + +## 6. Security baseline (end-of-alpha minimum) + +- **db is internal-only.** Never publish Postgres to the LAN. In + `docker-compose.truenas.yml` use `ports: !override []` on the db service to drop + any inherited host publish. The backend reaches it as `db:5432` on the compose + net; deploy-time role reconcile uses `docker exec`. +- **Secrets via Gitea Actions secrets**, never committed. Injected at job `env` + level in deploy.yml. +- **Postgres password rotation gotcha:** `POSTGRES_PASSWORD` only applies on first + volume init. A persistent volume keeps the *old* role password. The deploy.yml + includes an `ALTER USER ... WITH PASSWORD` reconcile step (guarded by + `if [ -n "$DB_PASSWORD" ]`) so rotating the secret actually takes effect. +- **Frontend verify** in deploy.yml uses a container-loopback node probe + (`docker exec -frontend node -e "require('http').get(...)"`), NOT a host + wget — the host probe gave transient false-failures while the container was still + recreating. + +--- + +## 7. Auth gotchas (NextAuth v5 over the HTTPS→HTTP proxy boundary) + +- Use **`auth()`**, not `getToken()`. `getToken`'s `__Secure-` cookie + autodetection breaks across the HTTPS-frontend → HTTP-internal boundary. +- Frontend env (set in compose override): `NEXTAUTH_URL` / `AUTH_URL` = + `https://.plate-software.de`, `AUTH_TRUST_HOST=true`, + `BACKEND_URL=http://backend:8080`. +- The server-side proxy route (`/api/backend/[...path]/route.ts`) injects the + Bearer token from the session and streams bodies with `duplex: "half"`. This is + the systemic fix that unblocked CannaManage — it ships in the template. + +--- + +## 8. Quick reference — "do the whole public switch" + +```bash +# 1. allocate port in §2 registry (next = 30011), commit the runbook update +# 2. create DNS A-record .plate-software.de → 82.165.206.45 (IONOS panel) +# 3. run the switch script: +./scripts/homelab-publish.sh +# 4. verify (§5) +``` + +--- + +## Appendix — files this runbook references + +- Template repo: `homelab-app-template` (Gitea) — scaffold for new projects +- Switch script: [`scripts/homelab-publish.sh`](../scripts/homelab-publish.sh:1) +- Proven examples: CannaManage [`deploy.yml`] + [`docker-compose.truenas.yml`], + InspectFlow (Caddy-fronted variant) +- Handover notes: `lumen-exchange/from-homelab/2026-06-22-cannamanage-public-hosting-LIVE.md` diff --git a/scripts/homelab-publish.sh b/scripts/homelab-publish.sh new file mode 100755 index 0000000..2975ef6 --- /dev/null +++ b/scripts/homelab-publish.sh @@ -0,0 +1,175 @@ +#!/usr/bin/env bash +# +# homelab-publish.sh — flip a locally-hosted TrueNAS app to public HTTPS. +# +# The "go public" switch from homelab-release-runbook.md §4. Idempotent and +# additive: it never touches the running app stack, only adds the frp tunnel + +# IONOS vhost + Let's Encrypt cert. Re-running is safe. To unpublish, remove the +# frpc proxy block (or run with `--down`) — local-phase hosting keeps working. +# +# Usage: +# ./homelab-publish.sh [subdomain] +# ./homelab-publish.sh myapp 3001 30011 myapp.plate-software.de +# +# --down remove the frpc block + disable the IONOS vhosts (keep the cert) +# --dry print what would happen without changing anything +# +# Prereqs (see runbook §1): +# - SSH aliases `truenas` and `ionos` configured in ~/.ssh/config +# - DNS A-record → 82.165.206.45 already propagated +# - frpc on TrueNAS (systemd) reading /mnt/VM_SSD_Pool/frp/frpc.toml +# - acme.sh + apache2 on IONOS; /var/www/html webroot for ACME challenge +# +set -euo pipefail + +# ── Fixed infra constants (runbook §1) ────────────────────────────────────── +FRPC_TOML="/mnt/VM_SSD_Pool/frp/frpc.toml" +VPS_IP="85.214.154.199" # frps host the IONOS vhost proxies to +APACHE_SITES="/etc/apache2/sites-available" +ACME_WEBROOT="/var/www/html" +SSH_TRUENAS="${SSH_TRUENAS:-truenas}" +SSH_IONOS="${SSH_IONOS:-ionos}" + +# ── Args ──────────────────────────────────────────────────────────────────── +DOWN=0; DRY=0; POS=() +for a in "$@"; do + case "$a" in + --down) DOWN=1 ;; + --dry) DRY=1 ;; + *) POS+=("$a") ;; + esac +done +set -- "${POS[@]:-}" + +PROJECT="${1:-}" +FRONTEND_PORT="${2:-}" +FRP_REMOTE_PORT="${3:-}" +SUBDOMAIN="${4:-${PROJECT}.plate-software.de}" + +if [[ -z "$PROJECT" || -z "$FRONTEND_PORT" || -z "$FRP_REMOTE_PORT" ]]; then + echo "usage: $0 [subdomain] [--down] [--dry]" >&2 + exit 2 +fi + +VHOST_CONF="${APACHE_SITES}/${SUBDOMAIN}.conf" +log() { printf '\033[1;34m▶ %s\033[0m\n' "$*"; } +ok() { printf '\033[1;32m✓ %s\033[0m\n' "$*"; } +warn() { printf '\033[1;33m! %s\033[0m\n' "$*"; } +run() { if [[ $DRY -eq 1 ]]; then echo " [dry] $*"; else eval "$@"; fi; } + +# ── Marker for the frpc block so we can find/replace/remove idempotently ───── +BEGIN="# >>> homelab-publish ${PROJECT} >>>" +END="# <<< homelab-publish ${PROJECT} <<<" + +frpc_block() { + cat </dev/null; apache2ctl configtest && systemctl reload apache2\"" + ok "vhost disabled (cert left in place for quick re-publish)" + exit 0 +fi + +# ── Pre-flight: DNS must point at IONOS ────────────────────────────────────── +log "Pre-flight: checking DNS for ${SUBDOMAIN}" +RESOLVED="$(dig +short "${SUBDOMAIN}" A | tail -n1 || true)" +if [[ "$RESOLVED" != "82.165.206.45" ]]; then + warn "DNS for ${SUBDOMAIN} = '${RESOLVED:-}' (expected 82.165.206.45)." + warn "Create the A-record in the IONOS panel and let it propagate first." + [[ $DRY -eq 1 ]] || exit 1 +else + ok "DNS → 82.165.206.45" +fi + +# ── 1. frp tunnel (TrueNAS frpc) ───────────────────────────────────────────── +log "frp: adding/updating proxy block on TrueNAS (${FRPC_TOML})" +BLOCK="$(frpc_block)" +# Remove any existing block for this project, then append the fresh one. +run "$SSH_TRUENAS \"sed -i '/${BEGIN}/,/${END}/d' ${FRPC_TOML}\"" +if [[ $DRY -eq 1 ]]; then + echo " [dry] append to ${FRPC_TOML}:"; echo "${BLOCK}" | sed 's/^/ /' +else + printf '%s\n' "${BLOCK}" | $SSH_TRUENAS "cat >> ${FRPC_TOML}" +fi +run "$SSH_TRUENAS \"systemctl restart frpc\"" +ok "frp tunnel ${FRONTEND_PORT} → frps:${FRP_REMOTE_PORT}" + +# ── 2. IONOS: HTTP vhost first (for ACME), then cert, then 443 ─────────────── +log "IONOS: writing vhost ${VHOST_CONF}" +VHOST="$(cat < + ServerName ${SUBDOMAIN} + Alias /.well-known/acme-challenge/ ${ACME_WEBROOT}/.well-known/acme-challenge/ + ProxyPass /.well-known/acme-challenge/ ! + RewriteEngine On + RewriteCond %{REQUEST_URI} !^/\\.well-known/acme-challenge/ + RewriteRule ^(.*)\$ https://%{HTTP_HOST}\$1 [R=301,L] + + + ServerName ${SUBDOMAIN} + SSLEngine on + SSLCertificateFile /root/.acme.sh/${SUBDOMAIN}_ecc/${SUBDOMAIN}.cer + SSLCertificateKeyFile /root/.acme.sh/${SUBDOMAIN}_ecc/${SUBDOMAIN}.key + SSLCertificateChainFile /root/.acme.sh/${SUBDOMAIN}_ecc/ca.cer + ProxyPreserveHost On + ProxyPass / http://${VPS_IP}:${FRP_REMOTE_PORT}/ + ProxyPassReverse / http://${VPS_IP}:${FRP_REMOTE_PORT}/ + RequestHeader set X-Forwarded-Proto https + RequestHeader set X-Real-IP %{REMOTE_ADDR}s + +EOF +)" +if [[ $DRY -eq 1 ]]; then + echo " [dry] write ${VHOST_CONF}:"; echo "${VHOST}" | sed 's/^/ /' +else + printf '%s\n' "${VHOST}" | $SSH_IONOS "cat > ${VHOST_CONF}" +fi + +# Enable HTTP side first so the ACME challenge can be served. If the cert files +# don't exist yet, the 443 block would fail configtest — so issue the cert, then +# enable. acme.sh writes the files; we (re)issue idempotently. +log "IONOS: issuing/renewing Let's Encrypt cert (pinned CA — ZeroSSL stalls)" +run "$SSH_IONOS \"acme.sh --set-default-ca --server letsencrypt\"" +# Enable the site so the :80 ACME alias is live for webroot validation. +run "$SSH_IONOS \"a2ensite ${SUBDOMAIN}.conf >/dev/null 2>&1 || true\"" +# Issue only if not already present (idempotent); --force on renew handled by acme cron. +run "$SSH_IONOS \"test -f /root/.acme.sh/${SUBDOMAIN}_ecc/${SUBDOMAIN}.cer || acme.sh --issue -d ${SUBDOMAIN} -w ${ACME_WEBROOT} --server letsencrypt\"" +ok "cert present for ${SUBDOMAIN}" + +log "IONOS: enabling vhost + reloading Apache" +run "$SSH_IONOS \"a2ensite ${SUBDOMAIN}.conf >/dev/null && apache2ctl configtest && systemctl reload apache2\"" +ok "Apache serving https://${SUBDOMAIN}" + +# ── 3. Verify (force-resolve to dodge stale workstation DNS cache) ─────────── +log "Verifying https://${SUBDOMAIN} (force-resolved to 82.165.206.45)" +if [[ $DRY -eq 0 ]]; then + CODE="$(curl -s -o /dev/null -w '%{http_code}' --max-time 15 \ + --resolve "${SUBDOMAIN}:443:82.165.206.45" "https://${SUBDOMAIN}/" || echo 000)" + if [[ "$CODE" =~ ^(200|301|302|307|308)$ ]]; then + ok "Public HTTPS up — HTTP ${CODE}" + else + warn "Got HTTP ${CODE} — check: frpc running? app on :${FRONTEND_PORT}? frps:${FRP_REMOTE_PORT} reachable?" + fi +fi + +echo +ok "DONE. ${SUBDOMAIN} → frps:${FRP_REMOTE_PORT} → frontend:${FRONTEND_PORT}" +echo " Remember: record this in homelab-release-runbook.md §2 registry."