# Runbook Sources of truth: `docker-compose.yml`, `main/check.sh`, `main/docs-check.sh`, `main/app/cli.py`, `main/alembic/`, `backups/`, rian's crontab. How to work on, check, deploy, and recover the app. Rules live in `agents.md`; this file is the procedure. Nothing here is a status: for what is running, ask the app (`/api/health`, `/api/stats`, `python -m app.cli status`). ## Open a session Always from the workspace root, which is where the checkpoint hook and the project commands find `.logs/`: `claude`. Rian's shortcut: ```bash alias dfp='cd /srv/apps/dutyfreeprofessor && claude' ``` A session opened in the old WordPress folder writes no diary; that folder's `CLAUDE.md` redirects here. ## The dev loop - Python lives in `.venv-dev/` at the workspace root; the frontend in `main/web/` (Node is build-time only in production). - **Dev database:** the `dfp-devdb` container on `127.0.0.1:5433`. Point `DATABASE_URL` at it for any local run and for every migration rehearsal. **Never run a migration against the live database** (`dutyfreeprofessor-db`, reachable only inside the compose network). - Local API: from `main/`, `../.venv-dev/bin/uvicorn app.main:app --reload` with `DATABASE_URL` set to the dev database; the SPA dev server is `npm run dev` in `main/web/`. - **Before every commit and every deploy:** `main/check.sh`. It runs the pure-logic tests, regenerates the typed API client from the backend's OpenAPI schema, typechecks the SPA against it, and runs the doc gates (`main/docs-check.sh`). Seconds, not the minutes a Docker build takes to find the same error. - Tests are the bugs that shipped: `main/tests/README.md`. ## Migrations `alembic heads` must show one head before you write one. Migrations are schema-only (create tables, add nullable columns or columns with a `server_default`); every backfill is an idempotent `python -m app.cli` command run after the deploy, never Python inside Alembic. The container runs `alembic upgrade head` on start (see the Dockerfile `CMD`), so a deploy applies whatever is at head. Ownership order for the current wave: build plan §2. After a deploy carrying migration #1 (`0a1b2c3d4e5f`), run the backfills it expects, in this order, each safe to repeat: `backfill accounts`, `vertical`, `stuck_runs`, `locations`, `orphan_tiles`, `image_provenance`. Each prints one line saying what it changed. After a deploy carrying migration #3 (`f3a4b5c6d7e8`), in this order, each safe to repeat and each rehearsed twice on a copy of the dump: `backfill implausible_sizes`, `backfill brands`, `backfill sizes`, `backfill categories`, `rederive`, `backfill merges`. Run them with no collection in progress: `rederive` re-keys every product and `merges` re-points listings. After 0.34.0: `backfill perfume` (renames the stored category Fragrance to Perfume; it has run on the live database, and a second run reports zero). After a deploy carrying migration #4 (`b5c6d7e8f9a0`, the account system), in this order, each safe to repeat and each rehearsed twice on a copy of the dump: `backfill accounts` (the seeded owner row gains its username), `backfill levels` (the two launch levels, insert if absent), then the account commands under Accounts below, then `backfill authors --map Adam=adam --map Mark=mark --map rian=rian` (links typed comment and to-do names to the three accounts; add `--include-defaulted` only to link the two tables where the page defaulted an anonymous actor to a name), and `backfill overrides` (reports what exists; writes nothing). Every backfill takes `--check` to print what a run would change and write nothing. After a deploy carrying migration #6 (`c6d7e8f9a0b1`, lines, variations and aliases at every level; Stream M), in this order, each safe to repeat and each rehearsed on a copy of the dump: `backfill lines`, `backfill variations`, then `rederive` and `backfill merges` (the key changed shape, so every product is re-keyed and whatever now agrees on all four fields folds), then `suggest` (fills the merge queue at brand, line and product level). Run them with no collection in progress. After a deploy carrying migration `e9f0a1b2c3d4` (the side panel: read state, the follow-up mark, the move provenance; Stream R2b), in this order, each safe to repeat: `backfill threads` (a second run reports zero threads and zero comments; it ends by ringing the legacy mentions, the standalone form being `backfill legacy_mentions`: a mention row for every old comment naming a current account that never rang, under the write path's own dedupe key, so the row emitted by hand on staging on 13 Sep is not doubled), `discussion rethread --check` (the 13 Sep map, `import/rethread-2026-09-13.json`, found at `/srv/import/` inside the container: read the lines; rian may edit the file first), then `discussion rethread --apply` once (a second `--apply` reports every entry as already done). Nothing is deleted but the one test comment, and that is the soft delete. After a deploy carrying migration `f0a1b2c3d4e5` (the discussion workflow: outcome, the closing record, archive, asks, acknowledgements), once: `backfill thread_reads_seed` (everything before that day is read for everyone; a second run inserts nothing). After a deploy carrying migration `b3c4d5e6f7a8` (the quantity model, the listed columns and the decided layer; Stream L), in this order, each safe to repeat and each rehearsed twice on a copy of the dump, with no collection in progress (`ls -lt .logs/runs/ | head && pgrep -af app.cli` first, and `pg_dump` before the deploy as always): ``` docker exec dutyfreeprofessor-app alembic current # must print b3c4d5e6f7a8 (head); stop if not docker exec dutyfreeprofessor-app python -m app.cli backfill quantities --check docker exec dutyfreeprofessor-app python -m app.cli backfill quantities docker exec dutyfreeprofessor-app python -m app.cli backfill variations docker exec dutyfreeprofessor-app python -m app.cli backfill lines docker exec dutyfreeprofessor-app python -m app.cli rederive docker exec dutyfreeprofessor-app python -m app.cli backfill merges --check # read the groups first docker exec dutyfreeprofessor-app python -m app.cli backfill merges docker exec dutyfreeprofessor-app python -m app.cli backfill listed --check docker exec dutyfreeprofessor-app python -m app.cli backfill listed docker exec dutyfreeprofessor-app python -m app.cli suggest docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-brands.json --check docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-brands.json docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-lines.json ``` `quantities` runs before `rederive` because the v4 key reads the quantity columns; `lines` before `rederive` because a gram pack now leaves the line key; `merges --check` prints every group so the fold is read before it runs; `listed` after `merges` so the listed columns land on the surviving listings. Then open `/collectors`, Listings, turn every column on; then Merge, Desk. **Recrawl after the Stream L chain.** Extime Paris is the one stale shop that can be recrawled (its 4 Sep crawl predates `raw_records`, so its listed columns are NULL until the fragments exist): rian runs it on bwlive inside an announced window (`.logs/runs/window-2026-09-15-extime-fragments.md` has the commands: collect, verify, then the staging refresh, `backfill listed` and `suggest`). The four Heinemann-family shops refused by their robots.txt and Dubai, which refuses our declared identity, are never contacted again by a collector (`COLLECTORS.md` Posture per source); their prices stay frozen at their last date and the collectors page says "refused since ". The merge session is `/collectors`, the Merge view: one pair at a time, by level (brand, line, product) and score, each side beside the other with its listings, prices, barcodes and images. On the keyboard: `y` or Enter confirms (the chosen spelling becomes the house's, the line's or the product's name), `n` rejects (the pair is never shown again), `s` or the right arrow skips, `1` and `2` pick a spelling, `b`, `l` and `p` switch level. Every decision is recorded against the signed-in account (`merge_candidates.decided_by`, `product_merges.merged_by`, the `decided_by` on `brands`, `product_lines` and `variation_aliases`) and in the audit log. A brand or line confirm re-runs the suggestions, so the pairs it puts within reach follow at once. Refresh (the button, or `app.cli suggest`) after a collection. The merge desk (Stream L) is the Merge view's other face, "Desk": a brands table and a lines table, one row per house or line with its best suggested pair, the reason, the score and a readable why; tick the rows you agree with ("Tick all at or above" a score, "Tick all proposed"), then Approve ticked or Keep apart ticked. The batch (`POST /api/collectors/merge/batch`, up to 200) applies each pair as the signed-in person's decision in id order, reports a refused pair (a barcode veto without a note, two houses' lines, an already decided pair) and applies the rest, regenerates the suggestions once at the end and commits once; decided rows show who and when. A proposals file is a session's own reading of the data, filed with `python -m app.cli propose --file /srv/import/proposals/-.json [--check]` (`import/` is mounted read-only at `/srv/import`): a new pair is inserted with reason `proposed`, which the rules never withdraw or rescore; a pair the rules already hold undecided is endorsed (its why joins the row and "Tick all proposed" finds it); nothing is merged by filing. The files for the 15 Sep deploy are `import/proposals/2026-09-15-brands.json` and `2026-09-15-lines.json`. Other commands, all `python -m app.cli ...` inside the container: `awards --rebuild [--file]` (re-match competition medals from `import/winners.json`; ids survive a rebuild), `images import` (product imagery from the openly licensed source), `discussion import` (the decision cards on `/discuss` from `import/discussion.json`), `discussion inbox --for claude [--since ]` (what rian said to the next session on `/issues`; `ACCOUNTS.md` Notifications), `sources` (list, and the per-source kill switch), `locations --show/--hide` (site visibility), `indexnow --since | --hours N [--dry-run]` (after a collection, once `INDEXNOW_KEY` is set and the host is not gated), `audit`, `verify`, `verify-status`, `verify-clear` (`QUALITY.md`), `articles`, `subscribers` (Content, below). The generated reference at the end of this file is the authority for flags. Environment (`app/config.py`, read from `.app.env`): `APP_ENV` (production unless set), `DATABASE_URL`, `PUBLIC_BASE_URL` (the origin every absolute URL is built from; required in production), `INDEXNOW_KEY`, the feature switches `FEATURE_MY_AIRPORTS`, `FEATURE_FEATURED_HOME`, `FEATURE_RICH_CARDS`, `FEATURE_TEASERS`, `FEATURE_CLIENT_TODOS`, the social profiles `SOCIAL_INSTAGRAM_URL` and `SOCIAL_YOUTUBE_URL`, `BROWSER_URL` (the render sidecar; empty means no browser), `FETCH_SERVICE_KEY` (absent by design), `ACCOUNT_OWNER` and `SITE_ACCESS` (the account system, under Accounts below), and `MAIL_PROVIDER`, `MAIL_FROM`, `RESEND_API_KEY` (mail for invites and resets; `none` until a provider is staged). A `REPLACE_WITH_` placeholder reads as absent everywhere. Host-side scripts in `main/scripts/` (run with the workspace Python, from the workspace root): `items.py` (the running list), `plan-set.py` (task status on `/plan`), `docmap.py` (the generated doc blocks), `seed-todos.py` (the client to-do rows, applied inside the container), `review-checklist.py` (the thirty-page human check), `beauty-candidates.py` (the targeted list), `export_competition_winners.py` (the winners file from the competition network), `render-probe.py` (one rendered page through the sidecar). Each carries its usage in its docstring. ## The three tokens Rian grants each, one holder at a time (build plan §2): - **Deploy.** Sessions build and test locally and never deploy. Say "green and ready". - **Migration.** One head, serialized, schema-only, as above. - **Crawl.** Long runs announce a window in `.logs/runs/`; no deploy inside it. ## Deploy (rian) A deploy recreates the app container and **kills any running collection**. 1. Check nothing is running: `ls -lt .logs/runs/ | head` and `pgrep -af app.cli`. 2. If the deploy carries a migration, dump first: `docker exec dutyfreeprofessor-db pg_dump -U dfp -Fc dfp > backups/dfp-$(date +%F)-.dump` 3. Bump `main/app/version.py`, and turn the `## Unreleased` heading in `main/CHANGELOG.md` into that version with today's date. `docs-check.sh` fails if the two disagree, or if the NOW block in `brief.md` still names the old version. 4. `main/check.sh`, then `srv-gw deploy --project dutyfreeprofessor --build`. 5. Confirm: `curl -s https://dutyfreeprofessor.demoing.info/api/health` shows the new version; `srv-gw security-audit` after any change to compose or ports. ## Collecting Inside the container: `docker exec dutyfreeprofessor-app python -m app.cli collect [--source ] [--limit N] [--delay S]`. One process per host so each host's crawl delay applies independently. Every run re-reads robots.txt; a refusal ends the run (`main/docs/COLLECTORS.md`). Log long runs to `.logs/runs/`. The per-source kill switch is the `enabled` column on `sources` (`/api/sources`, the sources page); a collector can be stopped without a deploy, which is a legal control, not a convenience. **Opening hours** (Stream G, `docs/COLLECTORS.md` Opening hours): `python -m app.cli hours collect [--airport IATA] [--check]` reads the airports whose operator platform has a collector (`services/hours/registry.py`), under the robots policy, and appends one collected row each; announce it in `.logs/runs/` like any collection. An airport the run leaves to hand population gets `python -m app.cli hours set --file [--by ]`, a dated hand row that outranks every collected one after it. `hours show [IATA]` prints the current line and its provenance per airport. Every command takes `--check`. ## Content (articles in, subscribers out) Adam's text arrives on `/todo` as Word, Markdown or plain-text uploads, stored under `uploads//` (mounted read-write at `/srv/uploads`). Inside the container: ```bash docker exec dutyfreeprofessor-app python -m app.cli articles import --todo --author rian docker exec dutyfreeprofessor-app python -m app.cli articles import --upload --kind airport_writeup --airport LHR docker exec dutyfreeprofessor-app python -m app.cli articles list --all # drafts too docker exec dutyfreeprofessor-app python -m app.cli articles publish [ ...] ``` A file that is not an upload goes into `import/editorial/` on the host (create the folder on first use; read-only in the container at `/srv/import/editorial/`) and is imported by path. Imports are drafts and idempotent on slug: fix the text, run the same command, nothing is republished or unpublished. Word images are counted and left out; put them under `public/` with versioned names and reference them from the text. `--airport` and `--category` give the keyed kinds one slug per airport or category, so a second hand-in replaces the first. The subscriber list is personal data and leaves the database only through the owner's shell, to a file only rian can read; never into `backups/` or `uploads/` (group-readable, served): ```bash docker exec dutyfreeprofessor-app python -m app.cli subscribers stats docker exec dutyfreeprofessor-app python -m app.cli subscribers export > ~/subscribers-$(date +%F).csv && chmod 600 ~/subscribers-*.csv docker exec dutyfreeprofessor-app python -m app.cli subscribers unsubscribe
``` ## CLI reference Generated from `python -m app.cli --help` by `main/scripts/docmap.py`; do not edit by hand. ```text usage: app.cli [-h] {collect,awards,images,discussion,backfill,rederive,suggest,propose,status,locations,verify,verify-status,verify-clear,audit,indexnow,articles,subscribers,accounts,sessions,audit-log,notify,staging-refresh,hours} ... positional arguments: {collect,awards,images,discussion,backfill,rederive,suggest,propose,status,locations,verify,verify-status,verify-clear,audit,indexnow,articles,subscribers,accounts,sessions,audit-log,notify,staging-refresh,hours} collect run collectors into the database awards import competition medals onto products images attach openly licensed product imagery discussion the decision cards import, and the inbox a session reads backfill run one idempotent data move after a deploy rederive recompute match keys under the current identity rules suggest fill the merge queue at brand, line and product level from the rules propose file a proposals JSON as merge candidates a person approves on the desk status show catalog and collector status locations list locations and toggle site visibility (collectors keep running) verify re-read a sample of published listings, one host at a time verify-status per-source rolling pass rate, blocks, spot-checked date verify-clear a human clears failing checks (who and when are recorded) audit no-network data audit: metrics against thresholds, review lists indexnow tell the IndexNow engines which pages changed articles editorial text: import hand-ins, list, publish subscribers email capture: export the list (owner only), counts accounts accounts: list, owner-check, bench, and the credential commands sessions sessions: revoke one account's or everyone's, prune old rows audit-log the audit log: prune the noisy actions to a bounded history notify the notification digest mail (host cron, every thirty minutes) staging-refresh the client-written rows out before a restore, and back in after hours opening hours: collect from operator sites, set by hand, show options: -h, --help show this help message and exit usage: app.cli collect [-h] [--source SOURCE] [--limit LIMIT] [--delay DELAY] options: -h, --help show this help message and exit --source SOURCE collector slug; default is every enabled collector --limit LIMIT max listings per source --delay DELAY politeness delay in seconds usage: app.cli awards [-h] [--file FILE] [--rebuild] options: -h, --help show this help message and exit --file FILE winners JSON (default /srv/import/winners.json) --rebuild drop competition-network medals and re-match from scratch usage: app.cli images [-h] [--limit LIMIT] [--delay DELAY] [--recheck] [--barcode-only] options: -h, --help show this help message and exit --limit LIMIT --delay DELAY --recheck re-check products already looked up --barcode-only skip the name-search fallback usage: app.cli discussion [-h] {import,inbox,rethread} ... positional arguments: {import,inbox,rethread} import import the decisions/realities list onto /discuss inbox what rian said to @claude on the running list, oldest first rethread apply a re-threading map once: topics, moves, soft deletes (T18) options: -h, --help show this help message and exit usage: app.cli backfill [-h] [--check] [--map TYPED=USERNAME] [--rename] [--include-defaulted] {accounts,authors,brands,categories,hours_seed,image_provenance,implausible_sizes,legacy_mentions,levels,lines,listed,locations,merges,orphan_tiles,overrides,perfume,quantities,sizes,stuck_runs,thread_reads_seed,threads,variations,vertical} positional arguments: {accounts,authors,brands,categories,hours_seed,image_provenance,implausible_sizes,legacy_mentions,levels,lines,listed,locations,merges,orphan_tiles,overrides,perfume,quantities,sizes,stuck_runs,thread_reads_seed,threads,variations,vertical} options: -h, --help show this help message and exit --check print what would change; write nothing --map TYPED=USERNAME authors only: link this typed name to this account (repeatable) --rename authors only: also rewrite each mapped spelling to the account's display name --include-defaulted authors only: also link feature_priorities and quote_selections, where the page defaulted an anonymous actor to a name usage: app.cli rederive [-h] options: -h, --help show this help message and exit usage: app.cli suggest [-h] [--check] options: -h, --help show this help message and exit --check print what would change; write nothing usage: app.cli propose [-h] --file FILE [--by BY] [--check] options: -h, --help show this help message and exit --file FILE the proposals file (import/proposals/-.json) --by BY who proposed (default: the file's proposed_by) --check report the counts; write nothing usage: app.cli status [-h] options: -h, --help show this help message and exit usage: app.cli locations [-h] [--show CODE [CODE ...]] [--hide CODE [CODE ...]] [--only CODE [CODE ...]] options: -h, --help show this help message and exit --show CODE [CODE ...] make these visible --hide CODE [CODE ...] hide these from the site --only CODE [CODE ...] make exactly these visible and hide everything else usage: app.cli verify [-h] [--n N] [--source SOURCE] [--seed SEED] [--mode {on_demand,after_collection,weekly}] options: -h, --help show this help message and exit --n N listings per source (20 after a collection, 200 weekly) --source SOURCE collector slug; repeatable; default every eligible source --seed SEED sampling seed (stored; random by default) --mode {on_demand,after_collection,weekly} usage: app.cli verify-status [-h] options: -h, --help show this help message and exit usage: app.cli verify-clear [-h] --check CHECK [CHECK ...] --note NOTE [--by BY] options: -h, --help show this help message and exit --check CHECK [CHECK ...] check ids from verify-status --note NOTE what was looked at and why it is fine --by BY account display name (default rian) usage: app.cli audit [-h] [--out OUT] [--json] [--seed SEED] [--no-store] [--fail-on-over] options: -h, --help show this help message and exit --out OUT directory for the JSON snapshot (default: the workspace's .logs/verification when present) --json print the whole snapshot instead of the summary --seed SEED sampling seed, so a review list is reproducible --no-store do not write audit_snapshots --fail-on-over exit 1 when any metric is over its threshold usage: app.cli indexnow [-h] [--since SINCE] [--hours HOURS] [--dry-run] [--show SHOW] options: -h, --help show this help message and exit --since SINCE ISO timestamp; pages with observations after it are submitted --hours HOURS or: pages changed in the last N hours (default 24) --dry-run list the URLs, submit nothing --show SHOW how many URLs a dry run prints usage: app.cli articles [-h] {import,list,publish,unpublish} ... positional arguments: {import,list,publish,unpublish} import a Markdown, text or Word file into the articles table (draft) list published articles (--all includes drafts) publish make articles public unpublish take articles back to draft options: -h, --help show this help message and exit usage: app.cli subscribers [-h] {export,stats,unsubscribe} ... positional arguments: {export,stats,unsubscribe} export CSV of active subscribers to stdout or --out FILE (mode 600) stats counts by source; no addresses unsubscribe record a withdrawal for one address options: -h, --help show this help message and exit usage: app.cli accounts [-h] {create,set-password,invite-link,reset-link,disable,enable,unlock,list,owner-check,bench} ... positional arguments: {create,set-password,invite-link,reset-link,disable,enable,unlock,list,owner-check,bench} create an account row and its level; no password yet set-password set a password (prompted); clears the lock; signs out everywhere invite-link print a one-time welcome link (48 h) reset-link print a one-time reset link (48 h) disable disable an account and revoke its sessions enable re-enable a disabled account unlock clear a lockout list every account with status, level and whether a password is set owner-check is ACCOUNT_OWNER set, present and able to sign in? bench the argon2 cost on this host options: -h, --help show this help message and exit usage: app.cli sessions [-h] {revoke,prune} ... positional arguments: {revoke,prune} revoke sign one account out everywhere, or everyone (--all, the break-glass) prune delete expired sessions, old revoked ones, used and old tokens options: -h, --help show this help message and exit usage: app.cli audit-log [-h] {prune} ... positional arguments: {prune} prune delete login and logout rows older than --keep-days options: -h, --help show this help message and exit usage: app.cli notify [-h] {digest} ... positional arguments: {digest} digest one mail per person for the notifications not yet delivered options: -h, --help show this help message and exit usage: app.cli staging-refresh [-h] {export,apply} ... positional arguments: {export,apply} export every client-written row as one JSON document on stdout apply put an exported document's rows back (by natural key, never losing one) options: -h, --help show this help message and exit usage: app.cli hours [-h] {collect,set,show} ... positional arguments: {collect,set,show} collect read the airports that have a collector, under robots set write an airport's hours by hand, dated and named to who entered them show the current hours and their provenance per airport options: -h, --help show this help message and exit ``` ## Production (bwlive) The live host is a DigitalOcean droplet in Toronto, `bwlive` on the tailnet (see `deploy/` for every script named here). It mirrors the dev server's shape: the same compose file plus `docker-compose.production.yml`, Caddy alone on 80 and 443, the app on the bridge port, Postgres on the internal network with no published port. The app gates itself (`SITE_ACCESS`); Caddy carries no gate. - **Reach it:** `ssh deploy@bwlive` (keys only, from the tailnet or the dev server; root is off). Break-glass if the tailnet is gone: DigitalOcean's recovery console after a root password reset. - **Deploy:** `deploy/production.sh` from this repository on the dev server. It dumps the production database first, rsyncs `main/`, `import/`, `public/`, `uploads/` and the compose files over Tailscale, builds there, recreates, and waits for `/api/health`. Add `--caddy` when `deploy/caddy/` changed. It never copies `.app.env` or `data/`. A deploy kills a running collection there too: check `.logs/runs/` first. - **First deploy only:** `deploy/production.sh --seed-db backups/dfp-nightly-.dump` restores the dev server's nightly dump into the empty production database. - **Roll back:** check out the previous commit and run `deploy/production.sh` again; a migration that rode along is reversed by restoring the pre-deploy dump it took (`backups/dfp--pre-deploy.dump` on the droplet). - **Restore from dump** (rehearse into a scratch database first, as on dev): `docker exec -i dutyfreeprofessor-db pg_restore -U dfp -d dfp --clean --if-exists --no-owner < backups/.dump` - **Backups:** the deploy user's crontab dumps nightly at 03:15 UTC into `backups/`, 14 days; the dev server pulls that directory into `backups/production/` at 04:15 local (`deploy/install-backup-cron.sh` installs both). DigitalOcean snapshots, if enabled, are the whole-box tool; the dumps are the everyday one. - **Secrets:** `.app.env` on the droplet (0600, deploy user). Machine secrets were generated there; the owner's values are filled in his own shell, never in chat. Caddy's Cloudflare DNS token lives in `/srv/caddy/cloudflare.env`, scoped to the one zone. Rotate by editing the file and recreating the container. - **Certificates:** a Cloudflare Origin CA certificate (15 years) in `/srv/caddy/origin/` on the droplet, trusted only by Cloudflare's edge, so the zone must stay proxied and Full (strict). Renew by creating a new one in the dashboard and pasting it over the two files, then `docker compose up -d` in `/srv/caddy`. The DNS-challenge module stays configured for any un-proxied host (`deploy/caddy/`). - **Egress:** `deploy/egress-test.py` re-runs the retailer check from any machine; compare against a same-day run from the dev server before trusting a new address. - **The sidecar fence:** `deploy/sidecar-firewall.sh`, owner-applied, keeps the browser sidecar off private ranges and the cloud metadata service. ## Backups and restore - Nightly `pg_dump` from rian's crontab at 03:15 into `backups/dfp-nightly-.dump`, kept for 14 days; failures append to `.logs/runs/backup.log`. Manual dumps before risky work follow the same shape with a reason in the name. - Restore (rehearse on `dfp-devdb` first): `docker exec -i dutyfreeprofessor-db pg_restore -U dfp -d dfp --clean --if-exists < backups/.dump` - `data/postgres/` is the live cluster and is not the backup. ## Staging refresh Adam comments on staging, so a refresh from production must keep every table people write on the site (`CLIENT_WRITTEN_TABLES` in `app/services/discussion.py`: the decision cards and their comments, the feature ranking, the quote selections and requests, the to-dos and their uploads, rian's running-list states; the list grows with the schema and the refresh reads it, never a copy). One host command, from the workspace root: ``` python3 main/scripts/staging-refresh.py --from backups/production/.dump --check python3 main/scripts/staging-refresh.py --from backups/production/.dump ``` `--check` exports every client-written row to `backups/refresh/-client-rows.json`, prints the counts and the commands the real run would issue, and restores nothing. The real run exports again, drops and recreates the `public` schema in the database container, restores the dump, runs `alembic upgrade head` in the app container (production may be behind staging), `backfill accounts` and `backfill levels`, then puts the exported rows back through `app.cli staging-refresh apply` and revokes every session (production's session rows are not staging's; sign in again). The report says, per table, what was restored, what was already there, what was updated or kept on a natural-key conflict (the newer row wins, a tie keeps production's) and what was renumbered (a comment whose id production has since used comes back under a fresh id; the deep link to the old id then lands on production's comment). Who-columns are remapped by username, so account ids need not agree between hosts; an author whose account does not exist on the restored side keeps the typed name with an empty link, and `backfill authors --map` relinks once the account exists. A second run of `apply` on the same file changes nothing. The export file is kept: whatever fails after it, the rows are on disk, and `app.cli staging-refresh apply --from - < ` inside the container puts them back. The seeded to-do text is production's after a refresh; re-run `seed-todos.py` on staging if it should differ. The app container is not stopped; a request during the restore fails once. Rehearse on `dfp-devdb` first: `--db dfp_r --db-container dfp-devdb --app-exec "../.venv-dev/bin/python -m app.cli" --alembic-exec "../.venv-dev/bin/alembic"` with `DATABASE_URL` pointed at `dfp_r`. Rehearsed 11 Sep with Adam's three comments of that day as the rows that had to survive (`tests/test_staging_refresh.py` pins the policy on SQLite). ## Secrets `.app.env` (mode 660) holds the database password and any service keys; the compose file reads it for both containers. Never print it, never commit it, never paste a value into chat. ### Accounts The site's own login (`ACCOUNTS.md` is the mechanism; this is the procedure). Two lines in `.app.env`, staged as placeholders in `.app.env.example` at the workspace root: ``` ACCOUNT_OWNER=REPLACE_WITH_USERNAME # the super admin; the owner is code, not a row SITE_ACCESS=members # sign in to see anything; `public` is the go-live flip ``` Set them in your own shell; delete `OWNER_PASSWORD` and `SESSION_SECRET` if present (ignored either way). Tighten the file so only you and the gateway read it: `setfacl -b .app.env && chmod 600 .app.env && setfacl -m u:srv-gateway:r .app.env` (a bare `chmod 600` sets the ACL mask to nothing and `srv-gw deploy` fails, because the gateway's compose reads `env_file` as `srv-gateway`); check with `sudo -u srv-gateway test -r .app.env && echo readable` (reads nothing); `srv-gw fix-permissions` widens it back to 660 plus the group ACL, so re-run the `setfacl` line after any permissions repair; `grep -c REPLACE_WITH_ .app.env` prints 0. Then recreate the container. With the owner unset the startup log says CRITICAL, nobody is super admin and the site stays up; `python -m app.cli accounts owner-check` says the same. **Seeding a host**, inside the container after the deploy that carries migration #4, in order: ``` python -m app.cli backfill accounts python -m app.cli backfill levels python -m app.cli accounts create --username adam --email --display-name Adam --level admin python -m app.cli accounts create --username mark --email --display-name Mark --level admin python -m app.cli accounts set-password --username rian # prompted twice, never argv python -m app.cli accounts invite-link --username adam # printed once; send it yourself python -m app.cli accounts invite-link --username mark python -m app.cli backfill authors --map Adam=adam --map Mark=mark --map rian=rian ``` A welcome link is `https:///welcome#`: the token is the fragment, so it never reaches a request line. The container's stdout is an access log of every request line, so nothing secret may ever sit in a URL path or query; the CLI prints a link once to your terminal and to nothing else. Links work for 48 hours and once. `accounts reset-link` is the same for a forgotten password until a mail provider exists (`MAIL_PROVIDER=resend` with `RESEND_API_KEY` and `MAIL_FROM` in an app-only env file, never the shared one); then `/forgot` on the site and "Send reset link" in the People panel work too. Other commands: `accounts list | disable | enable | unlock`, `sessions revoke --username | --all` (the break-glass: everyone out at once), `sessions prune`, `audit-log prune`, `accounts bench` (the argon2 cost on this host). The generated reference at the end of this file has the flags. **Rollout on the dev host** (the staging order, in full, in the accounts plan §7): dump first because a migration rides; the two lines above; the tightening; bump the version and deploy; seed as above and `awards --rebuild` if it is still pending; the curl list from the host against `http://172.17.0.1:` (every POST with `-H 'Origin: https://'`); the browser round trip; `srv-gw security-audit`; then, owner only, `srv-gw id-gate --site --mode public`, after which the app's login is the only barrier (rollback: `srv-gw id-gate --site `); the curls again through the public host; `srv-gw security-audit` again. Production repeats the seeding with its own passwords and links minted on its own `PUBLIC_BASE_URL`; the curl list runs through the domain credential before it comes off. **Transfer of ownership:** set `ACCOUNT_OWNER` to the new owner's username, create that account (`accounts create`, `set-password`), recreate the container; the previous owner's row is an ordinary account from then on, and nothing in the codebase points at this server. **One worker.** The per-address login throttle and the argon2 semaphore live in the process: one uvicorn worker is a launch assumption, and a `login_attempts` table comes before a second. **Members-only and the flip.** `SITE_ACCESS=members` answers every anonymous page with a 302 to `/login`, serves `Disallow: /` and no sitemap, and stamps `X-Robots-Tag: noindex, nofollow` on every response. The flip to `public` is the go-live checklist's step (with the IndexNow key and the uptime checks), one env line and a container recreate; it is never set earlier. ### Mail The provider is Resend behind three lines, `MAIL_PROVIDER=resend`, `MAIL_FROM=` and `RESEND_API_KEY`, and they belong in an **app-only env file**, never `.app.env`, which the database container reads too (`docker-compose.yml` gives both containers the same `env_file`; the split is Stream W's W3 and its compose change gets a security review and `srv-gw security-audit` first). Until those lines exist every mail path is inert: invites and resets go by CLI link, the People panel says so, and `notify digest` reports "no mail provider" and stamps nothing. When they do: every send counts against a daily cap in `email_sends` (6 auth mails per account: welcome, reset, access notice; 25 invites per host; 40 notification mails per account), a spent cap refuses before the provider is called (`MAIL_CAP` in the People panel), and the notification digest goes out from host cron on the machine that runs the app: ``` */30 * * * * docker exec dutyfreeprofessor-app python -m app.cli notify digest >> /srv/apps/dutyfreeprofessor/.logs/runs/notify-digest.log 2>&1 ``` One mail per person per run for the rows not yet delivered (a row already read in the app is stamped delivered and not mailed; a person who turned mail off under Account, or has no email, is skipped and their rows wait); `notify digest --check` prints what would go out and writes nothing. Nothing about a message is logged beyond the provider's status. ## What lives where Workspace root: docs, `.logs/`, compose, `import/` (read-only into the container, so the `/plan` page and the awards and discussion imports update without a deploy), `public/` (explainer pages served at `/.html`), `data/`, `backups/`. Code: `main/`.