# Runbook Sources of truth: `docker-compose.yml`, `main/check.sh`, `main/docs-check.sh`, `main/app/cli.py`, `main/alembic/`, `backups/`, rian's crontab. How to work on, check, deploy, and recover the app. Rules live in `agents.md`; this file is the procedure. Nothing here is a status: for what is running, ask the app (`/api/health`, `/api/stats`, `python -m app.cli status`). ## Open a session Always from the workspace root, which is where the checkpoint hook and the project commands find `.logs/`: `claude`. Rian's shortcut: ```bash alias dfp='cd /srv/apps/dutyfreeprofessor && claude' ``` A session opened in the old WordPress folder writes no diary; that folder's `CLAUDE.md` redirects here. ## The dev loop - Python lives in `.venv-dev/` at the workspace root; the frontend in `main/web/` (Node is build-time only in production). - **Dev database:** the `dfp-devdb` container on `127.0.0.1:5433`. Point `DATABASE_URL` at it for any local run and for every migration rehearsal. **Never run a migration against the live database** (`dutyfreeprofessor-db`, reachable only inside the compose network). - Local API: from `main/`, `../.venv-dev/bin/uvicorn app.main:app --reload` with `DATABASE_URL` set to the dev database; the SPA dev server is `npm run dev` in `main/web/`. - **Before every commit and every deploy:** `main/check.sh`. It runs the pure-logic tests, regenerates the typed API client from the backend's OpenAPI schema, typechecks the SPA against it, and runs the doc gates (`main/docs-check.sh`). Seconds, not the minutes a Docker build takes to find the same error. - Tests are the bugs that shipped: `main/tests/README.md`. ## Migrations `alembic heads` must show one head before you write one. Migrations are schema-only (create tables, add nullable columns or columns with a `server_default`); every backfill is an idempotent `python -m app.cli` command run after the deploy, never Python inside Alembic. The container runs `alembic upgrade head` on start (see the Dockerfile `CMD`), so a deploy applies whatever is at head. Ownership order for the current wave: build plan §2. After a deploy carrying migration #1 (`0a1b2c3d4e5f`), run the backfills it expects, in this order, each safe to repeat: `backfill accounts`, `vertical`, `stuck_runs`, `locations`, `orphan_tiles`, `image_provenance`. Each prints one line saying what it changed. After a deploy carrying migration #3 (`f3a4b5c6d7e8`), in this order, each safe to repeat and each rehearsed twice on a copy of the dump: `backfill implausible_sizes`, `backfill brands`, `backfill sizes`, `backfill categories`, `rederive`, `backfill merges`. Run them with no collection in progress: `rederive` re-keys every product and `merges` re-points listings. After 0.34.0: `backfill perfume` (renames the stored category Fragrance to Perfume; it has run on the live database, and a second run reports zero). After a deploy carrying migration #4 (`b5c6d7e8f9a0`, the account system), in this order, each safe to repeat and each rehearsed twice on a copy of the dump: `backfill accounts` (the seeded owner row gains its username), `backfill levels` (the two launch levels, insert if absent), then the account commands under Accounts below, then `backfill authors --map Adam=adam --map Mark=mark --map rian=rian` (links typed comment and to-do names to the three accounts; add `--include-defaulted` only to link the two tables where the page defaulted an anonymous actor to a name), and `backfill overrides` (reports what exists; writes nothing). Every backfill takes `--check` to print what a run would change and write nothing. After a deploy carrying migration #6 (`c6d7e8f9a0b1`, lines, variations and aliases at every level; Stream M), in this order, each safe to repeat and each rehearsed on a copy of the dump: `backfill lines`, `backfill variations`, then `rederive` and `backfill merges` (the key changed shape, so every product is re-keyed and whatever now agrees on all four fields folds), then `suggest` (fills the merge queue at brand, line and product level). Run them with no collection in progress. The merge session is `/collectors`, the Merge view: one pair at a time, by level (brand, line, product) and score, each side beside the other with its listings, prices, barcodes and images. On the keyboard: `y` or Enter confirms (the chosen spelling becomes the house's, the line's or the product's name), `n` rejects (the pair is never shown again), `s` or the right arrow skips, `1` and `2` pick a spelling, `b`, `l` and `p` switch level. Every decision is recorded against the signed-in account (`merge_candidates.decided_by`, `product_merges.merged_by`, the `decided_by` on `brands`, `product_lines` and `variation_aliases`) and in the audit log. A brand or line confirm re-runs the suggestions, so the pairs it puts within reach follow at once. Refresh (the button, or `app.cli suggest`) after a collection. Other commands, all `python -m app.cli ...` inside the container: `awards --rebuild [--file]` (re-match competition medals from `import/winners.json`; ids survive a rebuild), `images import` (product imagery from the openly licensed source), `discussion import` (the decision cards on `/discuss` from `import/discussion.json`), `sources` (list, and the per-source kill switch), `locations --show/--hide` (site visibility), `indexnow --since | --hours N [--dry-run]` (after a collection, once `INDEXNOW_KEY` is set and the host is not gated), `audit`, `verify`, `verify-status`, `verify-clear` (`QUALITY.md`), `articles`, `subscribers` (Content, below). The generated reference at the end of this file is the authority for flags. Environment (`app/config.py`, read from `.app.env`): `APP_ENV` (production unless set), `DATABASE_URL`, `PUBLIC_BASE_URL` (the origin every absolute URL is built from; required in production), `INDEXNOW_KEY`, the feature switches `FEATURE_MY_AIRPORTS`, `FEATURE_FEATURED_HOME`, `FEATURE_RICH_CARDS`, `FEATURE_TEASERS`, `FEATURE_CLIENT_TODOS`, the social profiles `SOCIAL_INSTAGRAM_URL` and `SOCIAL_YOUTUBE_URL`, `BROWSER_URL` (the render sidecar; empty means no browser), `FETCH_SERVICE_KEY` (absent by design), `ACCOUNT_OWNER` and `SITE_ACCESS` (the account system, under Accounts below), and `MAIL_PROVIDER`, `MAIL_FROM`, `RESEND_API_KEY` (mail for invites and resets; `none` until a provider is staged). A `REPLACE_WITH_` placeholder reads as absent everywhere. Host-side scripts in `main/scripts/` (run with the workspace Python, from the workspace root): `items.py` (the running list), `plan-set.py` (task status on `/plan`), `docmap.py` (the generated doc blocks), `seed-todos.py` (the client to-do rows, applied inside the container), `review-checklist.py` (the thirty-page human check), `beauty-candidates.py` (the targeted list), `export_competition_winners.py` (the winners file from the competition network), `render-probe.py` (one rendered page through the sidecar). Each carries its usage in its docstring. ## The three tokens Rian grants each, one holder at a time (build plan §2): - **Deploy.** Sessions build and test locally and never deploy. Say "green and ready". - **Migration.** One head, serialized, schema-only, as above. - **Crawl.** Long runs announce a window in `.logs/runs/`; no deploy inside it. ## Deploy (rian) A deploy recreates the app container and **kills any running collection**. 1. Check nothing is running: `ls -lt .logs/runs/ | head` and `pgrep -af app.cli`. 2. If the deploy carries a migration, dump first: `docker exec dutyfreeprofessor-db pg_dump -U dfp -Fc dfp > backups/dfp-$(date +%F)-.dump` 3. Bump `main/app/version.py`, and turn the `## Unreleased` heading in `main/CHANGELOG.md` into that version with today's date. `docs-check.sh` fails if the two disagree, or if the NOW block in `brief.md` still names the old version. 4. `main/check.sh`, then `srv-gw deploy --project dutyfreeprofessor --build`. 5. Confirm: `curl -s https://dutyfreeprofessor.demoing.info/api/health` shows the new version; `srv-gw security-audit` after any change to compose or ports. ## Collecting Inside the container: `docker exec dutyfreeprofessor-app python -m app.cli collect [--source ] [--limit N] [--delay S]`. One process per host so each host's crawl delay applies independently. Every run re-reads robots.txt; a refusal ends the run (`main/docs/COLLECTORS.md`). Log long runs to `.logs/runs/`. The per-source kill switch is the `enabled` column on `sources` (`/api/sources`, the sources page); a collector can be stopped without a deploy, which is a legal control, not a convenience. ## Content (articles in, subscribers out) Adam's text arrives on `/todo` as Word, Markdown or plain-text uploads, stored under `uploads//` (mounted read-write at `/srv/uploads`). Inside the container: ```bash docker exec dutyfreeprofessor-app python -m app.cli articles import --todo --author rian docker exec dutyfreeprofessor-app python -m app.cli articles import --upload --kind airport_writeup --airport LHR docker exec dutyfreeprofessor-app python -m app.cli articles list --all # drafts too docker exec dutyfreeprofessor-app python -m app.cli articles publish [ ...] ``` A file that is not an upload goes into `import/editorial/` on the host (create the folder on first use; read-only in the container at `/srv/import/editorial/`) and is imported by path. Imports are drafts and idempotent on slug: fix the text, run the same command, nothing is republished or unpublished. Word images are counted and left out; put them under `public/` with versioned names and reference them from the text. `--airport` and `--category` give the keyed kinds one slug per airport or category, so a second hand-in replaces the first. The subscriber list is personal data and leaves the database only through the owner's shell, to a file only rian can read; never into `backups/` or `uploads/` (group-readable, served): ```bash docker exec dutyfreeprofessor-app python -m app.cli subscribers stats docker exec dutyfreeprofessor-app python -m app.cli subscribers export > ~/subscribers-$(date +%F).csv && chmod 600 ~/subscribers-*.csv docker exec dutyfreeprofessor-app python -m app.cli subscribers unsubscribe
``` ## CLI reference Generated from `python -m app.cli --help` by `main/scripts/docmap.py`; do not edit by hand. ```text usage: app.cli [-h] {collect,awards,images,discussion,backfill,rederive,suggest,status,locations,verify,verify-status,verify-clear,audit,indexnow,articles,subscribers,accounts,sessions,audit-log} ... positional arguments: {collect,awards,images,discussion,backfill,rederive,suggest,status,locations,verify,verify-status,verify-clear,audit,indexnow,articles,subscribers,accounts,sessions,audit-log} collect run collectors into the database awards import competition medals onto products images attach openly licensed product imagery discussion import the decisions/realities list backfill run one idempotent data move after a deploy rederive recompute match keys under the current identity rules suggest fill the merge queue at brand, line and product level from the rules status show catalog and collector status locations list locations and toggle site visibility (collectors keep running) verify re-read a sample of published listings, one host at a time verify-status per-source rolling pass rate, blocks, spot-checked date verify-clear a human clears failing checks (who and when are recorded) audit no-network data audit: metrics against thresholds, review lists indexnow tell the IndexNow engines which pages changed articles editorial text: import hand-ins, list, publish subscribers email capture: export the list (owner only), counts accounts accounts: list, owner-check, bench, and the credential commands sessions sessions: revoke one account's or everyone's, prune old rows audit-log the audit log: prune the noisy actions to a bounded history options: -h, --help show this help message and exit usage: app.cli collect [-h] [--source SOURCE] [--limit LIMIT] [--delay DELAY] options: -h, --help show this help message and exit --source SOURCE collector slug; default is every enabled collector --limit LIMIT max listings per source --delay DELAY politeness delay in seconds usage: app.cli awards [-h] [--file FILE] [--rebuild] options: -h, --help show this help message and exit --file FILE winners JSON (default /srv/import/winners.json) --rebuild drop competition-network medals and re-match from scratch usage: app.cli images [-h] [--limit LIMIT] [--delay DELAY] [--recheck] [--barcode-only] options: -h, --help show this help message and exit --limit LIMIT --delay DELAY --recheck re-check products already looked up --barcode-only skip the name-search fallback usage: app.cli discussion [-h] [--file FILE] options: -h, --help show this help message and exit --file FILE discussion JSON (default /srv/import/discussion.json) usage: app.cli backfill [-h] [--check] [--map TYPED=USERNAME] [--include-defaulted] {accounts,authors,brands,categories,image_provenance,implausible_sizes,levels,lines,locations,merges,orphan_tiles,overrides,perfume,sizes,stuck_runs,variations,vertical} positional arguments: {accounts,authors,brands,categories,image_provenance,implausible_sizes,levels,lines,locations,merges,orphan_tiles,overrides,perfume,sizes,stuck_runs,variations,vertical} options: -h, --help show this help message and exit --check print what would change; write nothing --map TYPED=USERNAME authors only: link this typed name to this account (repeatable) --include-defaulted authors only: also link feature_priorities and quote_selections, where the page defaulted an anonymous actor to a name usage: app.cli rederive [-h] options: -h, --help show this help message and exit usage: app.cli suggest [-h] [--check] options: -h, --help show this help message and exit --check print what would change; write nothing usage: app.cli status [-h] options: -h, --help show this help message and exit usage: app.cli locations [-h] [--show CODE [CODE ...]] [--hide CODE [CODE ...]] [--only CODE [CODE ...]] options: -h, --help show this help message and exit --show CODE [CODE ...] make these visible --hide CODE [CODE ...] hide these from the site --only CODE [CODE ...] make exactly these visible and hide everything else usage: app.cli verify [-h] [--n N] [--source SOURCE] [--seed SEED] [--mode {on_demand,after_collection,weekly}] options: -h, --help show this help message and exit --n N listings per source (20 after a collection, 200 weekly) --source SOURCE collector slug; repeatable; default every eligible source --seed SEED sampling seed (stored; random by default) --mode {on_demand,after_collection,weekly} usage: app.cli verify-status [-h] options: -h, --help show this help message and exit usage: app.cli verify-clear [-h] --check CHECK [CHECK ...] --note NOTE [--by BY] options: -h, --help show this help message and exit --check CHECK [CHECK ...] check ids from verify-status --note NOTE what was looked at and why it is fine --by BY account display name (default rian) usage: app.cli audit [-h] [--out OUT] [--json] [--seed SEED] [--no-store] [--fail-on-over] options: -h, --help show this help message and exit --out OUT directory for the JSON snapshot (default /srv/apps/dutyfreeprofessor/.claude/worktrees/stream-m- fdcfa4/.logs/verification when present) --json print the whole snapshot instead of the summary --seed SEED sampling seed, so a review list is reproducible --no-store do not write audit_snapshots --fail-on-over exit 1 when any metric is over its threshold usage: app.cli indexnow [-h] [--since SINCE] [--hours HOURS] [--dry-run] [--show SHOW] options: -h, --help show this help message and exit --since SINCE ISO timestamp; pages with observations after it are submitted --hours HOURS or: pages changed in the last N hours (default 24) --dry-run list the URLs, submit nothing --show SHOW how many URLs a dry run prints usage: app.cli articles [-h] {import,list,publish,unpublish} ... positional arguments: {import,list,publish,unpublish} import a Markdown, text or Word file into the articles table (draft) list published articles (--all includes drafts) publish make articles public unpublish take articles back to draft options: -h, --help show this help message and exit usage: app.cli subscribers [-h] {export,stats,unsubscribe} ... positional arguments: {export,stats,unsubscribe} export CSV of active subscribers to stdout or --out FILE (mode 600) stats counts by source; no addresses unsubscribe record a withdrawal for one address options: -h, --help show this help message and exit usage: app.cli accounts [-h] {create,set-password,invite-link,reset-link,disable,enable,unlock,list,owner-check,bench} ... positional arguments: {create,set-password,invite-link,reset-link,disable,enable,unlock,list,owner-check,bench} create an account row and its level; no password yet set-password set a password (prompted); clears the lock; signs out everywhere invite-link print a one-time welcome link (48 h) reset-link print a one-time reset link (48 h) disable disable an account and revoke its sessions enable re-enable a disabled account unlock clear a lockout list every account with status, level and whether a password is set owner-check is ACCOUNT_OWNER set, present and able to sign in? bench the argon2 cost on this host options: -h, --help show this help message and exit usage: app.cli sessions [-h] {revoke,prune} ... positional arguments: {revoke,prune} revoke sign one account out everywhere, or everyone (--all, the break-glass) prune delete expired sessions, old revoked ones, used and old tokens options: -h, --help show this help message and exit usage: app.cli audit-log [-h] {prune} ... positional arguments: {prune} prune delete login and logout rows older than --keep-days options: -h, --help show this help message and exit ``` ## Production (bwlive) The live host is a DigitalOcean droplet in Toronto, `bwlive` on the tailnet (see `deploy/` for every script named here). It mirrors the dev server's shape: the same compose file plus `docker-compose.production.yml`, Caddy alone on 80 and 443, the app on the bridge port, Postgres on the internal network with no published port. The app gates itself (`SITE_ACCESS`); Caddy carries no gate. - **Reach it:** `ssh deploy@bwlive` (keys only, from the tailnet or the dev server; root is off). Break-glass if the tailnet is gone: DigitalOcean's recovery console after a root password reset. - **Deploy:** `deploy/production.sh` from this repository on the dev server. It dumps the production database first, rsyncs `main/`, `import/`, `public/`, `uploads/` and the compose files over Tailscale, builds there, recreates, and waits for `/api/health`. Add `--caddy` when `deploy/caddy/` changed. It never copies `.app.env` or `data/`. A deploy kills a running collection there too: check `.logs/runs/` first. - **First deploy only:** `deploy/production.sh --seed-db backups/dfp-nightly-.dump` restores the dev server's nightly dump into the empty production database. - **Roll back:** check out the previous commit and run `deploy/production.sh` again; a migration that rode along is reversed by restoring the pre-deploy dump it took (`backups/dfp--pre-deploy.dump` on the droplet). - **Restore from dump** (rehearse into a scratch database first, as on dev): `docker exec -i dutyfreeprofessor-db pg_restore -U dfp -d dfp --clean --if-exists --no-owner < backups/.dump` - **Backups:** the deploy user's crontab dumps nightly at 03:15 UTC into `backups/`, 14 days; the dev server pulls that directory into `backups/production/` at 04:15 local (`deploy/install-backup-cron.sh` installs both). DigitalOcean snapshots, if enabled, are the whole-box tool; the dumps are the everyday one. - **Secrets:** `.app.env` on the droplet (0600, deploy user). Machine secrets were generated there; the owner's values are filled in his own shell, never in chat. Caddy's Cloudflare DNS token lives in `/srv/caddy/cloudflare.env`, scoped to the one zone. Rotate by editing the file and recreating the container. - **Certificates:** a Cloudflare Origin CA certificate (15 years) in `/srv/caddy/origin/` on the droplet, trusted only by Cloudflare's edge, so the zone must stay proxied and Full (strict). Renew by creating a new one in the dashboard and pasting it over the two files, then `docker compose up -d` in `/srv/caddy`. The DNS-challenge module stays configured for any un-proxied host (`deploy/caddy/`). - **Egress:** `deploy/egress-test.py` re-runs the retailer check from any machine; compare against a same-day run from the dev server before trusting a new address. - **The sidecar fence:** `deploy/sidecar-firewall.sh`, owner-applied, keeps the browser sidecar off private ranges and the cloud metadata service. ## Backups and restore - Nightly `pg_dump` from rian's crontab at 03:15 into `backups/dfp-nightly-.dump`, kept for 14 days; failures append to `.logs/runs/backup.log`. Manual dumps before risky work follow the same shape with a reason in the name. - Restore (rehearse on `dfp-devdb` first): `docker exec -i dutyfreeprofessor-db pg_restore -U dfp -d dfp --clean --if-exists < backups/.dump` - `data/postgres/` is the live cluster and is not the backup. ## Secrets `.app.env` (mode 660) holds the database password and any service keys; the compose file reads it for both containers. Never print it, never commit it, never paste a value into chat. ### Accounts The site's own login (`ACCOUNTS.md` is the mechanism; this is the procedure). Two lines in `.app.env`, staged as placeholders in `.app.env.example` at the workspace root: ``` ACCOUNT_OWNER=REPLACE_WITH_USERNAME # the super admin; the owner is code, not a row SITE_ACCESS=members # sign in to see anything; `public` is the go-live flip ``` Set them in your own shell; delete `OWNER_PASSWORD` and `SESSION_SECRET` if present (ignored either way). Tighten the file so only you and the gateway read it: `setfacl -b .app.env && chmod 600 .app.env && setfacl -m u:srv-gateway:r .app.env` (a bare `chmod 600` sets the ACL mask to nothing and `srv-gw deploy` fails, because the gateway's compose reads `env_file` as `srv-gateway`); check with `sudo -u srv-gateway test -r .app.env && echo readable` (reads nothing); `srv-gw fix-permissions` widens it back to 660 plus the group ACL, so re-run the `setfacl` line after any permissions repair; `grep -c REPLACE_WITH_ .app.env` prints 0. Then recreate the container. With the owner unset the startup log says CRITICAL, nobody is super admin and the site stays up; `python -m app.cli accounts owner-check` says the same. **Seeding a host**, inside the container after the deploy that carries migration #4, in order: ``` python -m app.cli backfill accounts python -m app.cli backfill levels python -m app.cli accounts create --username adam --email --display-name Adam --level admin python -m app.cli accounts create --username mark --email --display-name Mark --level admin python -m app.cli accounts set-password --username rian # prompted twice, never argv python -m app.cli accounts invite-link --username adam # printed once; send it yourself python -m app.cli accounts invite-link --username mark python -m app.cli backfill authors --map Adam=adam --map Mark=mark --map rian=rian ``` A welcome link is `https:///welcome#`: the token is the fragment, so it never reaches a request line. The container's stdout is an access log of every request line, so nothing secret may ever sit in a URL path or query; the CLI prints a link once to your terminal and to nothing else. Links work for 48 hours and once. `accounts reset-link` is the same for a forgotten password until a mail provider exists (`MAIL_PROVIDER=resend` with `RESEND_API_KEY` and `MAIL_FROM` in an app-only env file, never the shared one); then `/forgot` on the site and "Send reset link" in the People panel work too. Other commands: `accounts list | disable | enable | unlock`, `sessions revoke --username | --all` (the break-glass: everyone out at once), `sessions prune`, `audit-log prune`, `accounts bench` (the argon2 cost on this host). The generated reference at the end of this file has the flags. **Rollout on the dev host** (the staging order, in full, in the accounts plan §7): dump first because a migration rides; the two lines above; the tightening; bump the version and deploy; seed as above and `awards --rebuild` if it is still pending; the curl list from the host against `http://172.17.0.1:` (every POST with `-H 'Origin: https://'`); the browser round trip; `srv-gw security-audit`; then, owner only, `srv-gw id-gate --site --mode public`, after which the app's login is the only barrier (rollback: `srv-gw id-gate --site `); the curls again through the public host; `srv-gw security-audit` again. Production repeats the seeding with its own passwords and links minted on its own `PUBLIC_BASE_URL`; the curl list runs through the domain credential before it comes off. **Transfer of ownership:** set `ACCOUNT_OWNER` to the new owner's username, create that account (`accounts create`, `set-password`), recreate the container; the previous owner's row is an ordinary account from then on, and nothing in the codebase points at this server. **One worker.** The per-address login throttle and the argon2 semaphore live in the process: one uvicorn worker is a launch assumption, and a `login_attempts` table comes before a second. **Members-only and the flip.** `SITE_ACCESS=members` answers every anonymous page with a 302 to `/login`, serves `Disallow: /` and no sitemap, and stamps `X-Robots-Tag: noindex, nofollow` on every response. The flip to `public` is the go-live checklist's step (with the IndexNow key and the uptime checks), one env line and a container recreate; it is never set earlier. ## What lives where Workspace root: docs, `.logs/`, compose, `import/` (read-only into the container, so the `/plan` page and the awards and discussion imports update without a deploy), `public/` (explainer pages served at `/.html`), `data/`, `backups/`. Code: `main/`.