deepdrft/deploy/PROD-INSTALL.md

# DeepDrftHome — Production Installation Checklist

Fresh-box checklist for deploying the DeepDrftHome solution (DeepDrftPublic, DeepDrftManager, DeepDrftAPI) to a new production host. Every package, directory, and service is treated as absent. Gated steps — those requiring a decision, a secret, or a network action from the operator — are marked **[GATE]**.

> This document is a reference you run from. It can drift from the actual `deploy/` scripts (`bootstrap.sh`, `install.sh`, `setup-step10-creds.sh`, the systemd units, and the nginx templates) — when those change, update this checklist to match.

## Phase 0 — Prerequisites (build/admin machine)

- [ ] Confirm DNS A/AAAA records for `deepdrft.com` and `app.deepdrft.com` point at the new host's IP — certbot's HTTP-01 challenge fails if DNS hasn't propagated. **[GATE]**
- [ ] Generate a CI deploy ed25519 key on your local machine (not the host): `ssh-keygen -t ed25519 -C "gitea-ci-deepdrft-prod" -f ~/.ssh/gitea_deepdrft_prod`. Public key → installer prompt; private key → Gitea secret `DEEPDRFT_PROD_SSH_DEPLOY`. **[GATE]**
- [ ] Download the latest `deepdrft-install.tar.gz` release asset (built by `package-install.yml` on a `deploy/` push to `master`). If none exists, push a no-op change to `deploy/` on `master` and wait for the artifact. **[GATE]**
- [ ] `scp deepdrft-install.tar.gz root@<host>:/tmp/`

## Phase 1 — Bootstrap / installer (run as root)

- [ ] Run: `INSTALL_PKG_PATH=/tmp/deepdrft-install.tar.gz bash bootstrap.sh` (installs OS prereqs, hands off to `install.sh`).
- [ ] The installer is interactive — have ready: app user (`deepdrft` / `/deepdrft`), PG role (`deepdrft`), DB names (`deepdrft-meta`, `deepdrft-auth`), public domain, app subdomain (default `app.<public>`), ports (5000/5001/5002), certbot email, PG password (twice), CI deploy public key. **[GATE]**

Automated Steps 0–10: apt preflight (postgresql, nginx, rsync, openssl, jq, wget) → create user → `enable-linger` → directory layout → deploy scripts to `/opt/deepdrft/bin/` → systemd units enabled (not started) → credentials (Step 6, prompts) → PostgreSQL role + DBs → `authorized_keys` forced-command → nginx vhosts → summary.

## Phase 2 — Credentials (Step 6, interactive) **[GATE]**

Writes 6 files to `/deepdrft/.config/credentials/` (mode 600): `filedatabase.json` (no prompt; vault path hardcoded), `apikey.json` (auto-generate or paste), `connections.json` (PG password), `authblocks.json` (JWT secret, issuer/audience, SMTP host+token+From, admin user/email/password, support email), `api-public.json` + `api-manager.json` (auto-built). JWT issuer/audience must match `appsettings.json` `AuthBlocks:Jwt`.

Verify: `sudo -u deepdrft ls -la /deepdrft/.config/credentials/` → 6 × mode 600, owned `deepdrft:deepdrft`.

## Phase 3 — CorsSettings **[GATE]**

`DeepDrftAPI/appsettings.json` `CorsSettings.AllowedOrigins` must include the Manager origin `https://app.deepdrft.com`. The API throws on startup if origins are empty; a missing Manager origin causes silent 401s on CMS auth. (Confirm this is present before building.)

## Phase 4 — Gitea secrets **[GATE]**

Add `DEEPDRFT_PROD_SSH_DEPLOY` = full private key contents. (The `dev`/beta host uses `DEEPDRFT_DCH7_SSH_DEPLOY`.)

## Phase 5 — TLS (after DNS propagates) **[GATE]**

```
host deepdrft.com
host app.deepdrft.com
snap install --classic certbot
ln -sf /snap/bin/certbot /usr/bin/certbot
certbot --nginx --email <email> --agree-tos --no-eff-email -d deepdrft.com -d app.deepdrft.com
nginx -t && systemctl reload nginx
certbot renew --dry-run
```

The installer's vhosts are HTTP-only (`listen 80`); certbot rewrites them in place to add the 443 blocks.

## Phase 6 — Verify SSH forced-command chain (before first deploy) **[GATE]**

- Layer 1: `ssh -i key deepdrft@host deploy-public` → prints the `[deploy-public]` prefix then a missing-archive error (non-zero exit expected).
- Layer 2: `ssh -i key deepdrft@host id` → `ssh-wrapper: unknown command: id` (no shell).
- Layer 3: rsync a smoke file → lands in `/deepdrft/staging/`.
- Layer 4: `deploy-public`, `deploy-manager`, `deploy-api` each print their prefix.

Do not proceed until all four pass.

## Phase 7 — First deploy (push `master`)

Three workflows trigger by path filter:
- `deploy-api.yml`: `DeepDrftAPI/`, `DeepDrftData/`, `DeepDrftContent/`, `DeepDrftModels/`
- `deploy-public.yml`: `DeepDrftPublic/`, `DeepDrftPublic.Client/`, `DeepDrftShared.Client/`, `DeepDrftModels/`
- `deploy-manager.yml`: `DeepDrftManager/`, `DeepDrftShared.Client/`, `DeepDrftModels/`

Root-file-only changes trigger none. `deploy-api` builds + publishes self-contained linux-x64, runs `ef migrations bundle`, rsyncs, then `deploy-api.sh` applies the EF bundle to `deepdrft-meta`, swaps `bin/`, restarts the unit. `deploy-public` also installs the `wasm-tools` workload. Watch the three parallel Gitea jobs. **[GATE]**

## Phase 8 — EF migration verification **[GATE]**

The EF bundle runs before the binary swap (metadata DB). The AuthBlocks `deepdrft-auth` schema self-migrates on first boot and seeds the admin. Verify `\dt` on both DBs plus `__EFMigrationsHistory`. On failure: `journalctl --user -u deepdrftapi`.

## Phase 9 — Service health

Check `deepdrftapi` / `deepdrftpublic` / `deepdrftmanager` via `systemctl --user status` and `journalctl`. Common failures: missing/wrong credential key; unreadable creds (must be 600 `deepdrft:deepdrft`); PostgreSQL not on peer auth; wrong vault path; OOM on droplets < 2 GB (add swap).

## Phase 10 — Smoke-test per host

**API (port 5002, internal):**
- `curl localhost:5002/api/track/page` → empty list on fresh install.
- `curl localhost:5002/api/stats/home` → zeros.
- `POST /api/auth/login` with admin creds → a JWT.

**Public site:**
- `curl https://deepdrft.com/` renders.
- `curl https://deepdrft.com/robots.txt` → `Allow: /` (NOT `Disallow: /` — that means the env isn't Production).
- `curl https://deepdrft.com/sitemap.xml` → XML (static roots present even with 0 releases).
- Confirm the unit carries `Environment=ASPNETCORE_ENVIRONMENT=Production`.

**Manager (CMS):**
- `curl https://app.deepdrft.com/` renders.
- `curl https://app.deepdrft.com/robots.txt` → `Disallow: /` (always uncrawlable).
- `curl -o /dev/null -w "%{http_code}" https://app.deepdrft.com/` → 200.

**Blazor WebSocket:**
- `curl -H "Upgrade: websocket" -H "Connection: Upgrade" https://deepdrft.com/_blazor` → 101 (not 502/504 from nginx).

## Phase 11 — Hardening

- [ ] Change the admin password immediately — the seed credentials are stored plaintext on disk. **[GATE]**
- [ ] Firewall (UFW): allow 22/80/443, deny the rest. Port 5002 (API) is internal-only.
- [ ] `apt-get install -y unattended-upgrades && dpkg-reconfigure -plow unattended-upgrades`
- [ ] Review `pg_hba.conf` — no `host all all 0.0.0.0/0 md5` line; the `deepdrft` role connects via Unix socket (peer auth).
- [ ] Back up `~/api/deepdrft/vaults` — it is not in any deploy artifact or EF bundle; a server wipe loses all audio permanently. **[GATE]**

## Phase 12 — Iteration notes

- EF migrations auto-apply before the binary swap on every deploy — no manual `dotnet ef database update`.
- AuthBlocks self-migrates on each API start.
- The FileDatabase vault is never touched by deploys; new vault types are created by the app on first access.
- Credential rotation: rerun `setup-step10-creds.sh --force` on the host as `deepdrft`, then restart the affected services.
- Each deploy script moves the current `bin/` to `bin.prev/` — manual rollback: `mv ~/public/bin.prev ~/public/bin && systemctl --user restart deepdrftpublic.service` (substitute `manager` / `api/deepdrft`).
- Two distinct staging dirs: `~/staging/` (CI rsync jail) vs `~/api/deepdrft/vaults/staging/` (large-audio upload staging). Do not conflate them.

## Host path reference

| Path | What |
|---|---|
| `/deepdrft/.config/credentials/` | 6 × JSON credential files (600) |
| `/deepdrft/.config/systemd/user/` | 3 × `.service` unit files |
| `/deepdrft/public/bin/` | DeepDrftPublic publish output |
| `/deepdrft/manager/bin/` | DeepDrftManager publish output |
| `/deepdrft/api/deepdrft/bin/` | DeepDrftAPI publish output |
| `/deepdrft/api/deepdrft/vaults/` | FileDatabase vault — never delete, never in deploy |
| `/deepdrft/staging/` | rsync jail root (CI artifact drop zone) |
| `/opt/deepdrft/bin/ssh-wrapper` | Forced-command dispatcher |
| `/opt/deepdrft/bin/deploy-*.sh` | Per-service deploy scripts |
| `/etc/nginx/sites-available/deepdrft.com.conf` | Public nginx vhost |
| `/etc/nginx/sites-available/app.deepdrft.com.conf` | Manager nginx vhost |