deepdrft/deploy/PROD-INSTALL.md

DeepDrftHome — Production Installation Checklist

Fresh-box checklist for deploying the DeepDrftHome solution (DeepDrftPublic, DeepDrftManager, DeepDrftAPI) to a new production host. Every package, directory, and service is treated as absent. Gated steps — those requiring a decision, a secret, or a network action from the operator — are marked [GATE].

This document is a reference you run from. It can drift from the actual deploy/ scripts (bootstrap.sh, install.sh, setup-step10-creds.sh, the systemd units, and the nginx templates) — when those change, update this checklist to match.

Phase 0 — Prerequisites (build/admin machine)

• Confirm DNS A/AAAA records for deepdrft.com and app.deepdrft.com point at the new host's IP — certbot's HTTP-01 challenge fails if DNS hasn't propagated. [GATE]
• Generate a CI deploy ed25519 key on your local machine (not the host): ssh-keygen -t ed25519 -C "gitea-ci-deepdrft-prod" -f ~/.ssh/gitea_deepdrft_prod. Public key → installer prompt; private key → Gitea secret DEEPDRFT_PROD_SSH_DEPLOY. [GATE]
• Download the latest deepdrft-install.tar.gz release asset (built by package-install.yml on a deploy/ push to master). If none exists, push a no-op change to deploy/ on master and wait for the artifact. [GATE]
• scp deepdrft-install.tar.gz root@<host>:/tmp/

Phase 1 — Bootstrap / installer (run as root)

• Run: INSTALL_PKG_PATH=/tmp/deepdrft-install.tar.gz bash bootstrap.sh (installs OS prereqs, hands off to install.sh).
• The installer is interactive — have ready: app user (deepdrft / /deepdrft), PG role (deepdrft), DB names (deepdrft-meta, deepdrft-auth), public domain, app subdomain (default app.<public>), ports (5000/5001/5002), certbot email, PG password (twice), CI deploy public key. [GATE]

Automated Steps 0–10: apt preflight (postgresql, nginx, rsync, openssl, jq, wget) → create user → enable-linger → directory layout → deploy scripts to /opt/deepdrft/bin/ → systemd units enabled (not started) → credentials (Step 6, prompts) → PostgreSQL role + DBs → authorized_keys forced-command → nginx vhosts → summary.

Phase 2 — Credentials (Step 6, interactive) [GATE]

Writes 6 files to /deepdrft/.config/credentials/ (mode 600): filedatabase.json (no prompt; vault path hardcoded), apikey.json (auto-generate or paste), connections.json (PG password), authblocks.json (JWT secret, issuer/audience, SMTP host+token+From, admin user/email/password, support email), api-public.json + api-manager.json (auto-built). JWT issuer/audience must match appsettings.json AuthBlocks:Jwt.

Verify: sudo -u deepdrft ls -la /deepdrft/.config/credentials/ → 6 × mode 600, owned deepdrft:deepdrft.

Phase 3 — CorsSettings [GATE]

DeepDrftAPI/appsettings.json CorsSettings.AllowedOrigins must include the Manager origin https://app.deepdrft.com. The API throws on startup if origins are empty; a missing Manager origin causes silent 401s on CMS auth. (Confirm this is present before building.)

Phase 4 — Gitea secrets [GATE]

Add DEEPDRFT_PROD_SSH_DEPLOY = full private key contents. (The dev/beta host uses DEEPDRFT_DCH7_SSH_DEPLOY.)

Phase 5 — TLS (after DNS propagates) [GATE]

host deepdrft.com
host app.deepdrft.com
snap install --classic certbot
ln -sf /snap/bin/certbot /usr/bin/certbot
certbot --nginx --email <email> --agree-tos --no-eff-email -d deepdrft.com -d app.deepdrft.com
nginx -t && systemctl reload nginx
certbot renew --dry-run

The installer's vhosts are HTTP-only (listen 80); certbot rewrites them in place to add the 443 blocks.

Phase 6 — Verify SSH forced-command chain (before first deploy) [GATE]

• Layer 1: ssh -i key deepdrft@host deploy-public → prints the [deploy-public] prefix then a missing-archive error (non-zero exit expected).
• Layer 2: ssh -i key deepdrft@host id → ssh-wrapper: unknown command: id (no shell).
• Layer 3: rsync a smoke file → lands in /deepdrft/staging/.
• Layer 4: deploy-public, deploy-manager, deploy-api each print their prefix.

Do not proceed until all four pass.

Phase 7 — First deploy (push master)

Three workflows trigger by path filter:

• deploy-api.yml: DeepDrftAPI/, DeepDrftData/, DeepDrftContent/, DeepDrftModels/
• deploy-public.yml: DeepDrftPublic/, DeepDrftPublic.Client/, DeepDrftShared.Client/, DeepDrftModels/
• deploy-manager.yml: DeepDrftManager/, DeepDrftShared.Client/, DeepDrftModels/

Root-file-only changes trigger none. deploy-api builds + publishes self-contained linux-x64, runs ef migrations bundle, rsyncs, then deploy-api.sh applies the EF bundle to deepdrft-meta, swaps bin/, restarts the unit. deploy-public also installs the wasm-tools workload. Watch the three parallel Gitea jobs. [GATE]

Phase 8 — EF migration verification [GATE]

The EF bundle runs before the binary swap (metadata DB). The AuthBlocks deepdrft-auth schema self-migrates on first boot and seeds the admin. Verify \dt on both DBs plus __EFMigrationsHistory. On failure: journalctl --user -u deepdrftapi.

Phase 9 — Service health

Check deepdrftapi / deepdrftpublic / deepdrftmanager via systemctl --user status and journalctl. Common failures: missing/wrong credential key; unreadable creds (must be 600 deepdrft:deepdrft); PostgreSQL not on peer auth; wrong vault path; OOM on droplets < 2 GB (add swap).

Phase 10 — Smoke-test per host

API (port 5002, internal):

• curl localhost:5002/api/track/page → empty list on fresh install.
• curl localhost:5002/api/stats/home → zeros.
• POST /api/auth/login with admin creds → a JWT.

Public site:

• curl https://deepdrft.com/ renders.
• curl https://deepdrft.com/robots.txt → Allow: / (NOT Disallow: / — that means the env isn't Production).
• curl https://deepdrft.com/sitemap.xml → XML (static roots present even with 0 releases).
• Confirm the unit carries Environment=ASPNETCORE_ENVIRONMENT=Production.

Manager (CMS):

• curl https://app.deepdrft.com/ renders.
• curl https://app.deepdrft.com/robots.txt → Disallow: / (always uncrawlable).
• curl -o /dev/null -w "%{http_code}" https://app.deepdrft.com/ → 200.

Blazor WebSocket:

• curl -H "Upgrade: websocket" -H "Connection: Upgrade" https://deepdrft.com/_blazor → 101 (not 502/504 from nginx).

Phase 11 — Hardening

• Change the admin password immediately — the seed credentials are stored plaintext on disk. [GATE]
• Firewall (UFW): allow 22/80/443, deny the rest. Port 5002 (API) is internal-only.
• apt-get install -y unattended-upgrades && dpkg-reconfigure -plow unattended-upgrades
• Review pg_hba.conf — no host all all 0.0.0.0/0 md5 line; the deepdrft role connects via Unix socket (peer auth).
• Back up ~/api/deepdrft/vaults — it is not in any deploy artifact or EF bundle; a server wipe loses all audio permanently. [GATE]

Phase 12 — Iteration notes

• EF migrations auto-apply before the binary swap on every deploy — no manual dotnet ef database update.
• AuthBlocks self-migrates on each API start.
• The FileDatabase vault is never touched by deploys; new vault types are created by the app on first access.
• Credential rotation: rerun setup-step10-creds.sh --force on the host as deepdrft, then restart the affected services.
• Each deploy script moves the current bin/ to bin.prev/ — manual rollback: mv ~/public/bin.prev ~/public/bin && systemctl --user restart deepdrftpublic.service (substitute manager / api/deepdrft).
• Two distinct staging dirs: ~/staging/ (CI rsync jail) vs ~/api/deepdrft/vaults/staging/ (large-audio upload staging). Do not conflate them.

Host path reference

Path	What
/deepdrft/.config/credentials/	6 × JSON credential files (600)
/deepdrft/.config/systemd/user/	3 × .service unit files
/deepdrft/public/bin/	DeepDrftPublic publish output
/deepdrft/manager/bin/	DeepDrftManager publish output
/deepdrft/api/deepdrft/bin/	DeepDrftAPI publish output
/deepdrft/api/deepdrft/vaults/	FileDatabase vault — never delete, never in deploy
/deepdrft/staging/	rsync jail root (CI artifact drop zone)
/opt/deepdrft/bin/ssh-wrapper	Forced-command dispatcher
/opt/deepdrft/bin/deploy-*.sh	Per-service deploy scripts
/etc/nginx/sites-available/deepdrft.com.conf	Public nginx vhost
/etc/nginx/sites-available/app.deepdrft.com.conf	Manager nginx vhost