From 70842cb57600b67579457acd4dc964926a9d955d Mon Sep 17 00:00:00 2001 From: daniel-c-harvey Date: Tue, 23 Jun 2026 08:15:56 -0400 Subject: [PATCH] docs: add production install checklist --- deploy/PROD-INSTALL.md | 127 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 deploy/PROD-INSTALL.md diff --git a/deploy/PROD-INSTALL.md b/deploy/PROD-INSTALL.md new file mode 100644 index 0000000..668963b --- /dev/null +++ b/deploy/PROD-INSTALL.md @@ -0,0 +1,127 @@ +# DeepDrftHome — Production Installation Checklist + +Fresh-box checklist for deploying the DeepDrftHome solution (DeepDrftPublic, DeepDrftManager, DeepDrftAPI) to a new production host. Every package, directory, and service is treated as absent. Gated steps — those requiring a decision, a secret, or a network action from the operator — are marked **[GATE]**. + +> This document is a reference you run from. It can drift from the actual `deploy/` scripts (`bootstrap.sh`, `install.sh`, `setup-step10-creds.sh`, the systemd units, and the nginx templates) — when those change, update this checklist to match. + +## Phase 0 — Prerequisites (build/admin machine) + +- [ ] Confirm DNS A/AAAA records for `deepdrft.com` and `app.deepdrft.com` point at the new host's IP — certbot's HTTP-01 challenge fails if DNS hasn't propagated. **[GATE]** +- [ ] Generate a CI deploy ed25519 key on your local machine (not the host): `ssh-keygen -t ed25519 -C "gitea-ci-deepdrft-prod" -f ~/.ssh/gitea_deepdrft_prod`. Public key → installer prompt; private key → Gitea secret `DEEPDRFT_PROD_SSH_DEPLOY`. **[GATE]** +- [ ] Download the latest `deepdrft-install.tar.gz` release asset (built by `package-install.yml` on a `deploy/` push to `master`). If none exists, push a no-op change to `deploy/` on `master` and wait for the artifact. **[GATE]** +- [ ] `scp deepdrft-install.tar.gz root@:/tmp/` + +## Phase 1 — Bootstrap / installer (run as root) + +- [ ] Run: `INSTALL_PKG_PATH=/tmp/deepdrft-install.tar.gz bash bootstrap.sh` (installs OS prereqs, hands off to `install.sh`). +- [ ] The installer is interactive — have ready: app user (`deepdrft` / `/deepdrft`), PG role (`deepdrft`), DB names (`deepdrft-meta`, `deepdrft-auth`), public domain, app subdomain (default `app.`), ports (5000/5001/5002), certbot email, PG password (twice), CI deploy public key. **[GATE]** + +Automated Steps 0–10: apt preflight (postgresql, nginx, rsync, openssl, jq, wget) → create user → `enable-linger` → directory layout → deploy scripts to `/opt/deepdrft/bin/` → systemd units enabled (not started) → credentials (Step 6, prompts) → PostgreSQL role + DBs → `authorized_keys` forced-command → nginx vhosts → summary. + +## Phase 2 — Credentials (Step 6, interactive) **[GATE]** + +Writes 6 files to `/deepdrft/.config/credentials/` (mode 600): `filedatabase.json` (no prompt; vault path hardcoded), `apikey.json` (auto-generate or paste), `connections.json` (PG password), `authblocks.json` (JWT secret, issuer/audience, SMTP host+token+From, admin user/email/password, support email), `api-public.json` + `api-manager.json` (auto-built). JWT issuer/audience must match `appsettings.json` `AuthBlocks:Jwt`. + +Verify: `sudo -u deepdrft ls -la /deepdrft/.config/credentials/` → 6 × mode 600, owned `deepdrft:deepdrft`. + +## Phase 3 — CorsSettings **[GATE]** + +`DeepDrftAPI/appsettings.json` `CorsSettings.AllowedOrigins` must include the Manager origin `https://app.deepdrft.com`. The API throws on startup if origins are empty; a missing Manager origin causes silent 401s on CMS auth. (Confirm this is present before building.) + +## Phase 4 — Gitea secrets **[GATE]** + +Add `DEEPDRFT_PROD_SSH_DEPLOY` = full private key contents. (The `dev`/beta host uses `DEEPDRFT_DCH7_SSH_DEPLOY`.) + +## Phase 5 — TLS (after DNS propagates) **[GATE]** + +``` +host deepdrft.com +host app.deepdrft.com +snap install --classic certbot +ln -sf /snap/bin/certbot /usr/bin/certbot +certbot --nginx --email --agree-tos --no-eff-email -d deepdrft.com -d app.deepdrft.com +nginx -t && systemctl reload nginx +certbot renew --dry-run +``` + +The installer's vhosts are HTTP-only (`listen 80`); certbot rewrites them in place to add the 443 blocks. + +## Phase 6 — Verify SSH forced-command chain (before first deploy) **[GATE]** + +- Layer 1: `ssh -i key deepdrft@host deploy-public` → prints the `[deploy-public]` prefix then a missing-archive error (non-zero exit expected). +- Layer 2: `ssh -i key deepdrft@host id` → `ssh-wrapper: unknown command: id` (no shell). +- Layer 3: rsync a smoke file → lands in `/deepdrft/staging/`. +- Layer 4: `deploy-public`, `deploy-manager`, `deploy-api` each print their prefix. + +Do not proceed until all four pass. + +## Phase 7 — First deploy (push `master`) + +Three workflows trigger by path filter: +- `deploy-api.yml`: `DeepDrftAPI/`, `DeepDrftData/`, `DeepDrftContent/`, `DeepDrftModels/` +- `deploy-public.yml`: `DeepDrftPublic/`, `DeepDrftPublic.Client/`, `DeepDrftShared.Client/`, `DeepDrftModels/` +- `deploy-manager.yml`: `DeepDrftManager/`, `DeepDrftShared.Client/`, `DeepDrftModels/` + +Root-file-only changes trigger none. `deploy-api` builds + publishes self-contained linux-x64, runs `ef migrations bundle`, rsyncs, then `deploy-api.sh` applies the EF bundle to `deepdrft-meta`, swaps `bin/`, restarts the unit. `deploy-public` also installs the `wasm-tools` workload. Watch the three parallel Gitea jobs. **[GATE]** + +## Phase 8 — EF migration verification **[GATE]** + +The EF bundle runs before the binary swap (metadata DB). The AuthBlocks `deepdrft-auth` schema self-migrates on first boot and seeds the admin. Verify `\dt` on both DBs plus `__EFMigrationsHistory`. On failure: `journalctl --user -u deepdrftapi`. + +## Phase 9 — Service health + +Check `deepdrftapi` / `deepdrftpublic` / `deepdrftmanager` via `systemctl --user status` and `journalctl`. Common failures: missing/wrong credential key; unreadable creds (must be 600 `deepdrft:deepdrft`); PostgreSQL not on peer auth; wrong vault path; OOM on droplets < 2 GB (add swap). + +## Phase 10 — Smoke-test per host + +**API (port 5002, internal):** +- `curl localhost:5002/api/track/page` → empty list on fresh install. +- `curl localhost:5002/api/stats/home` → zeros. +- `POST /api/auth/login` with admin creds → a JWT. + +**Public site:** +- `curl https://deepdrft.com/` renders. +- `curl https://deepdrft.com/robots.txt` → `Allow: /` (NOT `Disallow: /` — that means the env isn't Production). +- `curl https://deepdrft.com/sitemap.xml` → XML (static roots present even with 0 releases). +- Confirm the unit carries `Environment=ASPNETCORE_ENVIRONMENT=Production`. + +**Manager (CMS):** +- `curl https://app.deepdrft.com/` renders. +- `curl https://app.deepdrft.com/robots.txt` → `Disallow: /` (always uncrawlable). +- `curl -o /dev/null -w "%{http_code}" https://app.deepdrft.com/` → 200. + +**Blazor WebSocket:** +- `curl -H "Upgrade: websocket" -H "Connection: Upgrade" https://deepdrft.com/_blazor` → 101 (not 502/504 from nginx). + +## Phase 11 — Hardening + +- [ ] Change the admin password immediately — the seed credentials are stored plaintext on disk. **[GATE]** +- [ ] Firewall (UFW): allow 22/80/443, deny the rest. Port 5002 (API) is internal-only. +- [ ] `apt-get install -y unattended-upgrades && dpkg-reconfigure -plow unattended-upgrades` +- [ ] Review `pg_hba.conf` — no `host all all 0.0.0.0/0 md5` line; the `deepdrft` role connects via Unix socket (peer auth). +- [ ] Back up `~/api/deepdrft/vaults` — it is not in any deploy artifact or EF bundle; a server wipe loses all audio permanently. **[GATE]** + +## Phase 12 — Iteration notes + +- EF migrations auto-apply before the binary swap on every deploy — no manual `dotnet ef database update`. +- AuthBlocks self-migrates on each API start. +- The FileDatabase vault is never touched by deploys; new vault types are created by the app on first access. +- Credential rotation: rerun `setup-step10-creds.sh --force` on the host as `deepdrft`, then restart the affected services. +- Each deploy script moves the current `bin/` to `bin.prev/` — manual rollback: `mv ~/public/bin.prev ~/public/bin && systemctl --user restart deepdrftpublic.service` (substitute `manager` / `api/deepdrft`). +- Two distinct staging dirs: `~/staging/` (CI rsync jail) vs `~/api/deepdrft/vaults/staging/` (large-audio upload staging). Do not conflate them. + +## Host path reference + +| Path | What | +|---|---| +| `/deepdrft/.config/credentials/` | 6 × JSON credential files (600) | +| `/deepdrft/.config/systemd/user/` | 3 × `.service` unit files | +| `/deepdrft/public/bin/` | DeepDrftPublic publish output | +| `/deepdrft/manager/bin/` | DeepDrftManager publish output | +| `/deepdrft/api/deepdrft/bin/` | DeepDrftAPI publish output | +| `/deepdrft/api/deepdrft/vaults/` | FileDatabase vault — never delete, never in deploy | +| `/deepdrft/staging/` | rsync jail root (CI artifact drop zone) | +| `/opt/deepdrft/bin/ssh-wrapper` | Forced-command dispatcher | +| `/opt/deepdrft/bin/deploy-*.sh` | Per-service deploy scripts | +| `/etc/nginx/sites-available/deepdrft.com.conf` | Public nginx vhost | +| `/etc/nginx/sites-available/app.deepdrft.com.conf` | Manager nginx vhost |