# Runbook

Sources of truth: `docker-compose.yml`, `main/check.sh`, `main/docs-check.sh`, `main/app/cli.py`,
`main/alembic/`, `backups/`, rian's crontab.

How to work on, check, deploy, and recover the app. Rules live in `agents.md`; this file is
the procedure. Nothing here is a status: for what is running, ask the app
(`/api/health`, `/api/stats`, `python -m app.cli status`).

## Open a session
Always from the workspace root, which is where the checkpoint hook and the project commands
find `.logs/`: `claude`. Rian's shortcut:

```bash
alias dfp='cd /srv/apps/dutyfreeprofessor && claude'
```

A session opened in the old WordPress folder writes no diary; that folder's `CLAUDE.md`
redirects here.

## The dev loop
- Python lives in `.venv-dev/` at the workspace root; the frontend in `main/web/` (Node is
  build-time only in production).
- **Dev database:** the `dfp-devdb` container on `127.0.0.1:5433`. Point `DATABASE_URL` at it
  for any local run and for every migration rehearsal. **Never run a migration against the
  live database** (`dutyfreeprofessor-db`, reachable only inside the compose network).
- Local API: from `main/`, `../.venv-dev/bin/uvicorn app.main:app --reload` with
  `DATABASE_URL` set to the dev database; the SPA dev server is `npm run dev` in `main/web/`.
- **Before every commit and every deploy:** `main/check.sh`. It runs the pure-logic tests,
  regenerates the typed API client from the backend's OpenAPI schema, typechecks the SPA
  against it, and runs the doc gates (`main/docs-check.sh`). Seconds, not the minutes a
  Docker build takes to find the same error.
- Tests are the bugs that shipped: `main/tests/README.md`.

## Migrations
`alembic heads` must show one head before you write one. Migrations are schema-only
(create tables, add nullable columns or columns with a `server_default`); every backfill is
an idempotent `python -m app.cli` command run after the deploy, never Python inside Alembic.
The container runs `alembic upgrade head` on start (see the Dockerfile `CMD`), so a deploy
applies whatever is at head. Ownership order for the current wave: build plan §2.

After a deploy carrying migration #1 (`0a1b2c3d4e5f`), run the backfills it expects, in this
order, each safe to repeat: `backfill accounts`, `vertical`, `stuck_runs`, `shops`,
`orphan_tiles`, `image_provenance`. Each prints one line saying what it changed.

After a deploy carrying migration #3 (`f3a4b5c6d7e8`), in this order, each safe to repeat and
each rehearsed twice on a copy of the dump: `backfill implausible_sizes`, `backfill brands`,
`backfill sizes`, `backfill categories`, `rederive`, `backfill merges`. Run them with no
collection in progress: `rederive` re-keys every product variant and `merges` re-points listings.

After 0.34.0: `backfill perfume` (renames the stored category Fragrance to Perfume; it has run
on the live database, and a second run reports zero).

After a deploy carrying the gone-page classification (no migration): `backfill false_refusals
--check`, then `backfill false_refusals` (a run recorded `blocked` with "refused with HTTP 404"
or "410" becomes `error`, which un-refuses its source on the collectors page and in the sweep
plan; the recorded text is kept and says why it moved, a real refusal is untouched, and a
second run reports zero).

After a deploy carrying the picture migration (`aw3b1c2d3e4f`, Stream AW3: the seven picture
columns on brands and product lines, the four provenance columns on product variants; rehearsed
up, down, up on a copy of the 19 Sep nightly): `backfill image_sources --check`, then `backfill
image_sources` (the two old provenance strings move to the vocabulary and every existing
picture is stamped `variant`; a second run reports zero), then the picture commands in their
order under Pictures.

After a deploy carrying migration #4 (`b5c6d7e8f9a0`, the account system), in this order, each
safe to repeat and each rehearsed twice on a copy of the dump: `backfill accounts` (the seeded
owner row gains its username), `backfill levels` (the two launch levels, insert if absent),
then the account commands under Accounts below, then `backfill authors --map Adam=adam --map
Mark=mark --map rian=rian` (links typed comment and to-do names to the three accounts; add
`--include-defaulted` only to link the two tables where the page defaulted an anonymous actor
to a name), and `backfill overrides` (reports what exists; writes nothing). Every backfill takes
`--check` to print what a run would change and write nothing.

After a deploy carrying migration #6 (`c6d7e8f9a0b1`, lines, attributes and aliases at every
level; Stream M), in this order, each safe to repeat and each rehearsed on a copy of the
dump: `backfill lines`, `backfill attributes`, then `rederive` and `backfill merges` (the
key changed shape, so every product variant is re-keyed and whatever now agrees on all four fields
folds), then `suggest` (fills the merge queue at brand, line and product level). Run them
with no collection in progress.

After a deploy carrying migration `e9f0a1b2c3d4` (the side panel: read state, the follow-up
mark, the move provenance; Stream R2b), in this order, each safe to repeat: `backfill threads`
(a second run reports zero threads and zero comments; it ends by ringing the legacy mentions,
the standalone form being `backfill legacy_mentions`: a mention row for every old comment
naming a current account that never rang, under the write path's own dedupe key, so the row
emitted by hand on staging on 13 Sep is not doubled), `discussion rethread --check` (the 13 Sep map, `import/rethread-2026-09-13.json`, found at `/srv/import/` inside the container: read the lines;
rian may edit the file first), then `discussion rethread --apply` once (a second `--apply`
reports every entry as already done). Nothing is deleted but the one test comment, and that is
the soft delete.

After a deploy carrying migration `f0a1b2c3d4e5` (the discussion workflow: outcome, the closing
record, archive, asks, acknowledgements), once: `backfill thread_reads_seed` (everything before
that day is read for everyone; a second run inserts nothing).

After a deploy carrying migration `b3c4d5e6f7a8` (the quantity model, the listed columns and the
decided layer; Stream L), in this order, each safe to repeat and each rehearsed twice on a copy of
the dump, with no collection in progress (`ls -lt .logs/runs/ | head && pgrep -af app.cli` first,
and `pg_dump` before the deploy as always):

```
docker exec dutyfreeprofessor-app alembic current        # must print b3c4d5e6f7a8 (head); stop if not
docker exec dutyfreeprofessor-app python -m app.cli backfill quantities --check
docker exec dutyfreeprofessor-app python -m app.cli backfill quantities
docker exec dutyfreeprofessor-app python -m app.cli backfill attributes
docker exec dutyfreeprofessor-app python -m app.cli backfill lines
docker exec dutyfreeprofessor-app python -m app.cli rederive
docker exec dutyfreeprofessor-app python -m app.cli backfill merges --check      # read the groups first
docker exec dutyfreeprofessor-app python -m app.cli backfill merges
docker exec dutyfreeprofessor-app python -m app.cli backfill listed --check
docker exec dutyfreeprofessor-app python -m app.cli backfill listed
docker exec dutyfreeprofessor-app python -m app.cli suggest
```

**Staging only**, never on production (the files carry staging's row ids; see the warning under
the identity rules v5 block):

```
docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-brands.json --check
docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-brands.json
docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-lines.json
```

`quantities` runs before `rederive` because the v4 key reads the quantity columns; `lines`
before `rederive` because a gram pack now leaves the line key; `merges --check` prints every
group so the fold is read before it runs; `listed` after `merges` so the listed columns land on
the surviving listings. Then open `/collectors`, Listings, turn every column on; then Merge, Desk.

After a deploy carrying **Stream K1's rename** (migration `c1d2e3f4a5b6`, one term per concept:
`product_variants`, `shops`, `suggestions`, `merges`, `attribute_aliases`, `variant_id`, `shop_id`,
`product_line_id`, `alias_of_id`, `quantity_ml`; no value changes), the two data moves the migration
left to backfills, before the identity rules v5 block below, each safe to repeat (the second run
reports zero; rehearsed 16 Sep on two fresh copies of the nightly dump, where the K1 `rederive`
after them changed 0 keys, 0 lines and 0 attribute sets):

```
docker exec dutyfreeprofessor-app python -m app.cli backfill suggestion_decisions   # merged -> same, kept_apart -> separate
docker exec dutyfreeprofessor-app python -m app.cli backfill attribute_keys         # attributes.variation -> attribute, variation_kind -> attribute_kind
```

After a deploy carrying **Stream K2's ledger** (migration `d2e3f4a5b6c7`: the append-only
`decisions` table, `decision_batches`, `proposal_passes`, `proposals` and `redirects`; `places`
and `shop_places`; `uid` on brands, product lines, product variants, suggestions and places;
`hidden`/`indexed` on brands, product lines and places; `overrides` dropped after asserting it
held 0 rows on every host), in this order, each safe to repeat:

```
docker exec dutyfreeprofessor-app python -m app.cli backfill places    # one airport place per IATA code; every shop's primary place
docker exec dutyfreeprofessor-app python -m app.cli decisions verify   # read-only: every effective decision against its column; non-zero on drift
```

Order is not load-bearing between the two; `backfill places` first only because a fresh
airport page's counts are worth reading before trusting the drift report. Nothing else moves:
K2 changes no identity (`rederive --check` prints zero) and carried no data out of `overrides`
(0 rows on every host, so there was nothing to migrate).

After a deploy carrying **identity rules v6** (Stream K3; no migration; the certain key, options
as fields, the word lists as proposal generators; `docs/REVIEW-PROCESS.md` section 1 is the
boundary and its numbers), AFTER the K1 and K2 blocks above and INSTEAD of the v5 block below
(v6 contains it), in this order, each safe to repeat, with no collection in progress:

```
docker exec dutyfreeprofessor-app python -m app.cli backfill options --check      # read the count first
docker exec dutyfreeprofessor-app python -m app.cli backfill options              # Shopify variants named by their product title; the shop's option fields written as option:<name>
docker exec dutyfreeprofessor-app python -m app.cli backfill attributes           # the tail-read shade comes off the variants the options now cover
docker exec dutyfreeprofessor-app python -m app.cli backfill lines                # the product lines of the residual names, with their display names
docker exec dutyfreeprofessor-app python -m app.cli rederive                      # every row, forwarding rows included, on its v6 key and line
docker exec dutyfreeprofessor-app python -m app.cli rederive                      # must print zero everywhere
docker exec dutyfreeprofessor-app python -m app.cli backfill merges --check       # READ every MERGE line for a wrong join before the next command
docker exec dutyfreeprofessor-app python -m app.cli backfill merges
docker exec dutyfreeprofessor-app python -m app.cli suggest                       # pairs on emptied lines close as superseded
docker exec dutyfreeprofessor-app python -m app.cli backfill prune_lines --check
docker exec dutyfreeprofessor-app python -m app.cli backfill prune_lines          # the v5 line rows nothing references
docker exec dutyfreeprofessor-app python -m app.cli backfill rule_proposals --check
docker exec dutyfreeprofessor-app python -m app.cli backfill rule_proposals       # what each word list would have grouped, as proposals; applies nothing
docker exec dutyfreeprofessor-app python -m app.cli decisions verify
```

`options` first because every later step reads the un-glued name and the option fields; `lines`
before `rederive` for the display names; `merges --check` is read by a person (the rehearsal's
groups were all right joins; a wrong one is a reason to stop and raise it, never to edit a list);
`rule_proposals` last, because it reads the lines `rederive` and `prune_lines` leave. A list that
proves bad is withdrawn alone: `app.cli proposals withdraw --pass rule:<list>:6`. **Rollback:**
`IDENTITY_RULES_VERSION` back to "5" with the v5 code and `rederive` restores the v5 keys; the
un-glued names stay (harmless: v5 reads a name without a tail as having none), and the proposals
are withdrawn per pass. To measure another boundary on a COPY: `DFP_REHEARSE_BOUNDARY=b` (or `c`)
in the environment of `backfill lines` and `rederive`; never on a deployed app.

After a deploy carrying **publish** (Stream K6; no migration; `services/publish.py` is the one
reader of `indexed` and `hidden`), AFTER the K3 block above, with no collection in progress:

```
docker exec dutyfreeprofessor-app python -m app.cli index suggest --check         # read the counts first
docker exec dutyfreeprofessor-app python -m app.cli index suggest                 # the pages over their rule, as suggestions; indexes nothing
docker exec dutyfreeprofessor-app python -m app.cli index list --limit 20
```

From that deploy every brand, product line, airport and category-at-airport page says
`noindex, follow`, and the sitemap and IndexNow carry none of them, until a person approves
pages on `/review` (Index candidates) or with `app.cli index approve <kind>:<slug> --by <user>`
(`brand:`, `line:`, `place:`, `airport:<IATA>`; `--all-suggested` takes every open suggestion).
Each approval is one batch: `app.cli decisions undo-batch <uid>` reverses it. `index suggest`
is safe to repeat and belongs after each collection; it never indexes and never removes, and a
page under its rule is only suggested for removal. An approval pings IndexNow once where the
host has `INDEXNOW_KEY`; staging has none and stays silent. Hide a wrong page with
`app.cli review brand|line <id> hidden`; it then answers 302. **Rollback:** the previous image; `indexed`
decisions and `redirects` rows are additive and read by nothing older.

After a deploy carrying **identity rules v5** (no migration; the catalogue decisions of 15 Sep,
`.logs/planning/catalogue-model-decisions-2026-09-15.md`: the shade and the marked flavour leave
the line key, a pack word in a drinks name is the expression, the flavour reader fires only on a
marked tail), in this order, each safe to repeat, rehearsed on a copy of staging taken the same day
(section 5 of the decisions has the numbers that run produced), with no collection in progress:

```
docker exec dutyfreeprofessor-app python -m app.cli backfill attributes       # the flavour kinds the old reader guessed come off
docker exec dutyfreeprofessor-app python -m app.cli backfill lines            # the shadeless lines are created with their display names; product variants and tombstones pointed; line decisions follow their product variants
docker exec dutyfreeprofessor-app python -m app.cli rederive                  # every row on its v5 line and key; nothing folds, the shade stays in the attribute slot
docker exec dutyfreeprofessor-app python -m app.cli backfill merges --check   # read it: the rehearsal showed 0 mergeable groups
docker exec dutyfreeprofessor-app python -m app.cli backfill merges
docker exec dutyfreeprofessor-app python -m app.cli suggest                   # pairs on emptied lines close as superseded
docker exec dutyfreeprofessor-app python -m app.cli backfill prune_lines --check   # the emptied line rows nothing references
docker exec dutyfreeprofessor-app python -m app.cli backfill prune_lines
```

`attributes` first because `lines` reads the kind; `lines` before `rederive` so the new rows take
the most common spelling as their name (`rederive` re-derives every row's line itself, so the
order is never load-bearing for the key); `suggest` before `prune_lines` so a pair the rules no
longer offer is withdrawn and stops being a reference; `prune_lines` last because a tombstone's
line and a person's decision are references it never touches. Every one of these refuses while
a collection is running (`--force` for a run known to be stuck). Then open the Desk: the emptied
lines are gone from it, and every confirm from now on writes its row in the `decisions` ledger
(Stream K2; `overrides` before it) as well as the column.

**Never run `propose --file` on production with a file written from staging.** The proposal files
under `import/proposals/` carry staging's row ids; on another database the same ids are other
brands and lines, and confirming them on the desk would alias unrelated rows. Write a new file from
the database it is for, or skip the step.

**The review gate** (decisions §2.8) is a CLI until the desk grows its column:
`docker exec dutyfreeprofessor-app python -m app.cli review brand|line <id> checked|hidden|clear
--reason "<why>" [--by <username>]`. `hidden` takes an eligible brand page off the site at once,
everywhere the one definition of a brand page is read (the index, the page, the sitemap, the link
on every product line and variant), and is a `hidden` decision in the `decisions` ledger since
Stream K2.
`checked` is retired (plan W18): an approved sheet is the quality fact now, and the command
refuses it with `VALUE_INVALID` rather than writing anything; the `checked` choice stays on the
CLI's argument list only until it is dropped alongside the desk's own column.

**Recrawl after the Stream L chain.** Extime Paris is the one stale shop that can be recrawled
(its 4 Sep crawl predates `raw_records`, so its listed columns are NULL until the fragments
exist): rian runs it on bwlive inside an announced window
(`.logs/runs/window-2026-09-15-extime-fragments.md` has the commands: collect, verify, then the
staging refresh, `backfill listed` and `suggest`). The four Heinemann-family shops refused by
their robots.txt and Dubai, which refuses our declared identity, are never contacted again by a
collector (`COLLECTORS.md` Posture per source); their prices stay frozen at their last date and
the collectors page says "refused since <date>".

The merge session is `/collectors`, the Merge view: one pair at a time, by level (brand, line,
product) and score, each side beside the other with its listings, prices, barcodes and images.
On the keyboard: `y` or Enter confirms (the chosen spelling becomes the brand's, the line's or
the product variant's name), `n` rejects (the pair is never shown again), `s` or the right arrow skips,
`1` and `2` pick a spelling, `b`, `l` and `p` switch level. Every decision is recorded against
the signed-in account (`suggestions.decided_by`, `merges.merged_by`, the
`decided_by` on `brands`, `product_lines` and `attribute_aliases`) and in the audit log. A
brand or line confirm re-runs the suggestions, so the pairs it puts within reach follow at
once. Refresh (the button, or `app.cli suggest`) after a collection.

The merge desk (Stream L) is the Merge view's other face, "Desk": a brands table and a lines
table, one row per standard brand or line with its best suggested pair, the reason, the score and a
readable why; tick the rows you agree with ("Tick all at or above" a score, "Tick all
proposed"), then Confirm ticked or Keep separate ticked. The batch (`POST
/api/collectors/merge/batch`, up to 200) applies each pair as the signed-in person's decision
in id order, reports a refused pair (a barcode veto without a note, two brands' lines, an
already decided pair) and applies the rest, regenerates the suggestions once at the end and
commits once; decided rows show who and when. A proposals file is a session's own reading of
the data, filed with `python -m app.cli propose --file /srv/import/proposals/<date>-<level>.json
[--check]` (`import/` is mounted read-only at `/srv/import`): a new pair is inserted with
reason `proposed`, which the rules never withdraw or rescore; a pair the rules already hold
undecided is endorsed (its why joins the row and "Tick all proposed" finds it); nothing is
merged by filing. The files for the 15 Sep deploy are `import/proposals/2026-09-15-brands.json`
and `2026-09-15-lines.json`.

**The decisions ledger's commands** (`app.cli decisions ...`, Stream K2). `list [--entity <type>
<id> | --batch <uid> | --pass <name>] [--since <ISO>]` is the tracked-changes view, newest
first. `undo-batch <uid> --reason <text> --by <username> [--force]` reverses a whole batch in
one transaction (its own tail folds first, then its rows in descending id order), skipping and
naming any row a later batch has since superseded. `verify` is read-only: every effective
decision compared to its materialised column, alias-followed; it prints the drift and exits
non-zero when there is any (the after-deploy block above runs it). `export [--since <ISO> |
--batch <uid> | --pass <name> | --all] [--out <file>]` writes the ledger as JSON lines by
natural key, no local id anywhere, in `(decided_at, id)` order. `replay --file <jsonl> [--check]
[--label <host>] [--force]` applies an export on this host: every row resolved by natural key,
matched by uid, an older row over a newer target row inserted but never materialised; what it
cannot resolve goes to `<file>.remainder.jsonl`, which the next replay of the same file retries
first. `--check` (on `replay`, and on `proposals load`, K4) runs the whole thing in one
transaction rolled back at the end and prints the counts; nothing is written. `undo-batch` and
`replay` refuse `COLLECTION_RUNNING` (`--force` to override); `list`, `verify` and `export` only
read and never gate.

Other commands, all `python -m app.cli ...` inside the container: `awards --rebuild [--file]`
(re-match competition medals from `import/winners.json`; ids survive a rebuild), `images fetch`
(product imagery from the openly licensed source; the picture commands are under Pictures below), `discussion import` (the decision cards on
`/discuss` from `import/discussion.json`), `discussion inbox --for claude [--since <date>]` (what rian
said to the next session on `/issues`; `ACCOUNTS.md` Notifications), `sources` (list, and the per-source kill switch),
`shops --show/--hide` (site visibility), `indexnow --since <ISO> | --hours N [--dry-run]`
(after a collection, once `INDEXNOW_KEY` is set and the host is not gated), `audit`, `verify`,
`verify-status`, `verify-clear` (`QUALITY.md`), `articles`, `subscribers` (Content, below). The
generated reference at the end of this file is the authority for flags.

Environment (`app/config.py`, read from `.app.env`): `APP_ENV` (production unless set),
`DATABASE_URL`, `PUBLIC_BASE_URL` (the origin every absolute URL is built from; required in
production), `INDEXNOW_KEY`, the feature switches `FEATURE_MY_AIRPORTS`, `FEATURE_FEATURED_HOME`,
`FEATURE_RICH_CARDS`, `FEATURE_TEASERS`, `FEATURE_CLIENT_TODOS`, the social profiles
`SOCIAL_INSTAGRAM_URL` and `SOCIAL_YOUTUBE_URL`, `BROWSER_URL` (the render sidecar; empty means
no browser), `FETCH_SERVICE_KEY` (absent by design), `ACCOUNT_OWNER` and `SITE_ACCESS` (the
account system, under Accounts below), and `MAIL_PROVIDER`, `MAIL_FROM`, `RESEND_API_KEY` (mail
for invites and resets; `none` until a provider is staged). A `REPLACE_WITH_` placeholder reads
as absent everywhere.

Host-side scripts in `main/scripts/` (run with the workspace Python, from the workspace root):
`items.py` (the running list), `plan-set.py` (task status on `/plan`), `docmap.py` (the generated
doc blocks), `seed-todos.py` (the client to-do rows, applied inside the container),
`review-checklist.py` (the thirty-page human check), `beauty-candidates.py` (the targeted list),
`export_competition_winners.py` (the winners file from the competition network),
`render-probe.py` (one rendered page through the sidecar). Each carries its usage in its
docstring.

## The three tokens
Rian grants each, one holder at a time (build plan §2):
- **Deploy.** Sessions build and test locally and never deploy. Say "green and ready".
- **Migration.** One head, serialized, schema-only, as above.
- **Crawl.** Long runs announce a window in `.logs/runs/`; no deploy inside it.

## Deploy (rian)
A deploy recreates the app container and **kills any running collection**.

1. Freeze starts, then check nothing is running: `printf 'deploy %s by %s\n' "$(date -u +%FT%TZ)"
   "$USER" > uploads/.collect-freeze` (while the marker exists the page answers `COLLECT_FROZEN`
   and `app.cli collect` exits 3, so nothing new begins under you; the away run's deploys stamp
   `D3` in place of `deploy`), then `ls -lt .logs/runs/ | head`, `pgrep -af app.cli`, and
   `/collectors#live` with nothing in its live group.
2. If the deploy carries a migration, dump first:
   `docker exec dutyfreeprofessor-db pg_dump -U dfp -Fc dfp > backups/dfp-$(date +%F)-<reason>.dump`
3. Bump `main/app/version.py`, and turn the `## Unreleased` heading in `main/CHANGELOG.md`
   into that version with today's date. `docs-check.sh` fails if the two disagree, or if the
   NOW block in `brief.md` still names the old version.
4. `main/check.sh`, then `srv-gw deploy --project dutyfreeprofessor --build`.
5. Confirm: `curl -s https://dutyfreeprofessor.demoing.info/api/health` shows the new version;
   `srv-gw security-audit` after any change to compose or ports. Then `rm -f
   uploads/.collect-freeze`: a marker left behind keeps refusing starts, and after six hours the
   page's banner calls it stale.

## Collecting
**From the page.** `/collectors#live` is the control room: one line per collector with its state
word and the actions that state allows, so a run is started, held and ended there rather than
by `docker exec`. **Start ▾** picks the mode, `discover` (look for new: the listing pages, as a
shell run) or `recheck` (read the held listings back one by one through `read_one`; not on a
rendered source), and an optional limit; **Pause** and **Resume** hold and release the run at
its next request boundary; **Stop** ends it there as `stopped` with the batch committed; **Stop
now** (SIGTERM, never SIGKILL) is offered only a minute after a Stop that went unanswered;
**Mark as ended** tidies a row the rules call dead. Every action is a POST recorded against the
person in the audit log; a refusal is one sentence on the row. `SOURCE_REFUSED` is final: a host
that refused us is started by nobody until rian records a decision. The page replaces the
command for a run, not the collectors; the mechanism is `docs/COLLECTORS.md` (How collection
works).

**The pace** is per source, seconds between requests, set on the row: never under the host's
robots.txt crawl delay (the loop reads it on every run and the row says "host asks N"), never
under our own 1 s, never under 10 s on a rendered source, never over 600 s. A running loop
re-reads the pace, the control and the kill switch every two seconds between requests, so a
change lands within two seconds plus the request in flight.

**From the shell** (a window's scripts, a sweep): `docker exec dutyfreeprofessor-app python -m
app.cli collect [--source <slug>] [--mode discover|recheck] [--limit N] [--delay S] [--by <who>]
[--ignore-freeze]`. One process per host so each host's crawl delay applies independently; a
second process on the same source finds the collector lock held and exits 3 with no run row.
Every run re-reads robots.txt; a refusal ends the run (`main/docs/COLLECTORS.md`). Log long runs
to `.logs/runs/`; a window entry names the page actions it expects pressed and the `psql` check
after each, as `.logs/runs/window-2026-09-19-w2-live-page.md` does.

`python -m app.cli status` prints each source's state word, the same the page shows: `running`,
`pausing`, `paused`, `stopping`, `stalled`, `dead`, `idle`, `refused`, or `off` for the kill switch.

The per-source kill switch is the `enabled` column on `sources` (`/api/sources`, the sources
page); a running collector reads it between requests and ends `stopped`, so a collector can be
stopped without a deploy, which is a legal control, not a convenience.

**The deploy freeze.** A deploy kills every collector in the container, so the Deploy step
writes `uploads/.collect-freeze` (one stamp line, `<deploy> <time> by <user>`) before its gate
check and removes it after health. While it exists nothing starts: the page refuses with
`COLLECT_FROZEN` and shows the stamp in a banner, `app.cli collect` exits 3 (`--ignore-freeze`
overrides). A marker older than six hours is a deploy that never removed it; the banner says so.

**Opening hours** (Stream G, `docs/COLLECTORS.md` Opening hours): `python -m app.cli hours collect
[--airport IATA] [--check]` reads the airports whose operator platform has a collector
(`services/hours/registry.py`), under the robots policy, and appends one collected row each;
announce it in `.logs/runs/` like any collection. An airport the run leaves to hand population
gets `python -m app.cli hours set <IATA> --file <one line of text> [--by <username>]`, a dated
hand row that outranks every collected one after it. `hours show [IATA]` prints the current line
and its provenance per airport. Every command takes `--check`.

### A sweep before a review

A sweep is every reachable source collected once, so the review passes work on fresh listed
words. `python -m app.cli sweep plan` prints, from the sources and their last run, the order:

1. `backfill stuck_runs`: a run that died stays `running` and the approval gate would count it.
   The gate ignores a run older than the dead age, and any run that began before the app container
   did (a reboot or a deploy killed it, whatever the timer says), but the row is tidied here.
2. `backfill prune_lines`: the empty product lines nothing references. A line held only by merge
   tombstones is a record and stays (the running list carries the reader-side fix); an undone fold
   prunes what it minted on its own.
3. One `collect --source <slug>` process per reachable source, in its own log under `.logs/runs/`.

A source whose last run was a refusal (robots.txt, or an edge that answers our declared identity
with a refusal) is **not** in the list and is never contacted by a sweep. Its listings keep their
last-seen prices and their collected names, which the review cites as such
(`docs/REVIEW-PROCESS.md` section 3). Permission changes are recorded on the source by a person,
never inferred by retrying.

After the sweep, passes: `python -m app.cli pass packet --brand <slug>` is what a pass is given
(`docs/REVIEW-PASS.md`), `pass status` says which brands are open, parked or settled and prints the
one done line, `pass parked` lists the questions parked to the running list with the command that
unparks each, and `precedents list | export` shows the register. Passes run until `pass status`
prints `done`.

## Splitting a brand the fold joined wrongly

`normalize.brand_key` drops a trailing listed word before a listing's brand text is resolved to a
row, so "Tanqueray Gin" and "Tanqueray" land on one row. The list is per vertical
(REVIEW-PROCESS.md section 1 item 1a) and every fold it makes today is right, but the fold is a
rule, not a decision: nothing records it, so there is nothing to undo. Splitting the row is what
there is to undo.

```bash
app.cli brands spellings --brand <slug>          # every listed spelling on the row, with counts
app.cli brands split --brand <slug> \
    --spelling "<one>" --spelling "<another>" \
    --name "<the brand they move to>" --by rian
app.cli decisions undo-batch <batch uid> --reason "<why>" --by rian
```

The split moves every variant carrying the named spellings (tombstones included) onto a brand row
of its own, re-derives their keys and product lines, prunes the lines it empties, and records ONE
decision that the undo reads back. It refuses a spelling no live variant carries
(`SPELLING_NOT_ON_ROW`), a split that would leave nothing on the source (`SPLIT_EMPTIES_SOURCE`, a
rename rather than a split), an alias row (`ENTITY_ALIASED`) and a running collection.

**On the sheet:** rejecting one of the `rule:brand_trailers` proposals offers the same split under
the rejected row -- tick the spellings, name the brand, press. A rejection by itself moves nothing.

**The spellings a split names become a CLAIM** the resolver reads before the fold, so the next
collection does not put them back (`brand_key("Appleton Rum")` is "appleton", the row it was split
away from). The claim lives in the decision, so undoing the split withdraws it too.

**Every word list**, its words, the vertical it applies to and what it folded is on `/review`.

## Content (articles in, subscribers out)
Adam's text arrives on `/todo` as Word, Markdown or plain-text uploads, stored under
`uploads/<todo_id>/` (mounted read-write at `/srv/uploads`). Inside the container:

```bash
docker exec dutyfreeprofessor-app python -m app.cli articles import --todo <todo_id> --author rian
docker exec dutyfreeprofessor-app python -m app.cli articles import --upload <upload_id> --kind airport_writeup --airport LHR
docker exec dutyfreeprofessor-app python -m app.cli articles list --all      # drafts too
docker exec dutyfreeprofessor-app python -m app.cli articles publish <slug> [<slug> ...]
docker exec dutyfreeprofessor-app python -m app.cli articles list --samples  # the Professor's own
docker exec dutyfreeprofessor-app python -m app.cli articles unpublish --samples
```

**The Professor's own pieces are imported with `--sample`** and come back to draft in one
command, `articles unpublish --samples`, which reads the column rather than a list of slugs:
that is the step before production's database is replaced, and the ten that exist, with their
exact import lines, are in `import/articles/samples/README.md`. `--sample` is the one field a
re-import always overwrites, so nothing can sit outside the step because somebody once imported
it plainly. `--tag <word>` is repeatable (lowercase, at most eight) and `--hero-credit` prints
the attribution as a caption under the hero: a supplied picture is licensed, not ours.

An import may print `note: <seq> was behind the rows that exist`. That is the repair, not a
warning: a refresh replays the rows people wrote with their own ids and leaves the table's id
sequence where it stood, so the next insert would collide. The import lifts it once and says so.

A file that is not an upload goes into `import/editorial/` on the host (create the folder on
first use; read-only in the container at `/srv/import/editorial/`) and is imported by path. Imports are drafts and
idempotent on slug: fix the text, run the same command, nothing is republished or
unpublished. Word images are counted and left out; put them under `public/` with versioned
names and reference them from the text. `--airport` and `--category` give the keyed kinds
one slug per airport or category, so a second hand-in replaces the first.

The subscriber list is personal data and leaves the database only through the owner's
shell, to a file only rian can read; never into `backups/` or `uploads/` (group-readable,
served). The procedure, in this order:

```bash
docker exec dutyfreeprofessor-app python -m app.cli subscribers stats
(umask 077; docker exec dutyfreeprofessor-app python -m app.cli subscribers export > ~/subscribers-$(date +%F).csv)
wc -l ~/subscribers-$(date +%F).csv            # one header line plus the `active` count above
docker exec dutyfreeprofessor-app python -m app.cli subscribers unsubscribe <address>
```

`stats` comes first because it prints `total`, `active`, `unsubscribed` and a count per form,
and `active` is how many rows the export should hold. **`umask 077` around the redirect, not
`chmod` after it**: the shell creates the file before the first byte arrives and rian's umask is
002, so a `chmod` afterwards leaves the whole list group-readable for as long as the export
takes to run. **Never `--out` inside the container**: it writes mode 600 wherever it is pointed,
but the container's only writable paths are its own layer, where the list would sit unnoticed
until the next recreate, and the bind-mounted `uploads/`, which is group-readable on the host
and partly served. `--out` is for a host-side run against a copy of the dump. The default
export leaves withdrawn rows out; `--include-unsubscribed` puts them back with the withdrawal date in the last
column, which answers a subject access request and never feeds the mailing tool. `unsubscribe`
stamps the row and keeps it, so someone who signs up again is the same person rather than a new
one, and the form tells nobody which of the two happened.

The file goes to whoever runs the mailing, through a channel that already carries the client's
material, and never pasted into a message, a session or a to-do. Delete it once it has landed:
`shred -u ~/subscribers-<date>.csv`. Capture itself follows `SITE_ACCESS`: while the site is
members-only a signed-out visitor's subscription is refused and nothing is stored, so the list
starts filling at the go-live flip (`ACCOUNTS.md`, Members-only mode).

### The privacy policy and the terms
The copy is `main/docs/legal/privacy.md` and `main/docs/legal/terms.md`, read per request
(`process_doc.legal_page`) and served at `/privacy` and `/terms`. An edit is a deploy and nothing
else: change the file, bump its `**Version N, YYYY-MM-DD**` line (the page prints that date as
"Last updated"), run `main/check.sh`, deploy with `--build` (the image copies `docs/legal/`).
Both pages name one contact address; change it in both files together.

## Pictures
Pictures live at three levels, brand, product line and product variant, each with a recorded
source and licence (`DATA-MODEL.md`; the posture is `COLLECTORS.md` L2: brand-supplied or openly
licensed, never a retailer's photograph). The `images` group holds every command; `images fetch`
is the Open Food Facts lookup by barcode and strict name (the former bare `images`).

### Manifest
A supplier's folder and a brand-owner's product export become a reviewable JSON before a byte
is copied. Nothing is read but names, and nothing touches the network:

```bash
DATABASE_URL=postgresql+psycopg://dfp:<password>@127.0.0.1:5433/<copy> ../.venv-dev/bin/python -m app.cli images manifest \
  --folder "/path/to/the supplier's folder" \
  --csv "/path/to/products.csv" \
  --out ../import/images/<supplier>-<date>.json
```

It runs on the host from `main/`, against `dfp-devdb` holding a fresh restore of the newest dump
(Backups and restore), because the folder is not mounted in the container. What it does
(`services/image_manifest.py`, pure): a file is classified from its path and name alone (the
supplier from the top folder, the product line from the folder it sits in, the view words, the
size, a barcode in the name; `BACK`, `CLOSE_UP`, `ICONS`, `LAY BACK`, `GROUP`, `BWC` and a box,
tube, pack or carton without the bottle are unusable, with the reason); an export row gives the
front bottle image as the variant's picture and the brand logo as the brand's, with the marketing
brand, the title before its first comma, the net content, the UPC and a leading-zero GTIN. Each
picture is then matched precision first: a barcode equal to a live variant's wins; else the brand
from the leading words, the line by its key or by the one live line whose key differs only in
category and format words (an age must agree on both sides), the variant by size. Two candidates
at any step, or a brand the catalogue does not hold, leave the target empty with the reason.

One picture per target: the logo once per brand; per variant the best shot; per line the
best-quality front bottle shot (a cut-out first, then a bottle with its box or tube, then a plain
shop-style shot), preferring the representative variant's size, else 700 or 750 ml; the same
shot in two formats is one candidate, the PNG. A tie at the top is nobody's choice: every tied
candidate is written unchosen with the tie as its reason. Byte duplicates are found by content;
twins whose names disagree are refused.

The output is tracked (`import/images/`) and small: one entry per picture and level with
`source_path` or `source_url`, `level`, `target` (`brand:<slug>`, `line:<slug>`, `variant:<id>`),
`chosen`, `reason`, `image_source` (`admin:<supplier-slug>`), `licence` and `attribution`, and a
`summary` per level. The command prints the summary: entries, with a target, chosen, targets,
unassigned, and the top three reasons. A person reviews the file before anything is staged:
flip `chosen`, or type a `target` the matcher refused (a slug or a variant id) and set `chosen`
on it; re-running the command regenerates the file from the names, so hand edits go in after the
last run. `images stage` and `images import` then read it.

### Stage
The chosen entries' pictures into the uploads home, as the derivatives a page serves. Runs on
the host from `main/` like `manifest`, because the folder is not in the container; no database:

```bash
../.venv-dev/bin/python -m app.cli images stage \
  --manifest ../import/images/<supplier>-<date>.json \
  --folder "/path/to/the supplier's folder" \
  --uploads /srv/apps/dutyfreeprofessor/uploads [--check]
```

`--uploads` is the folder the container mounts at `/srv/uploads` (the default is the container's
own path). For each chosen entry with a target it reads the file from the folder, or downloads the
address (a brand-owner export names addresses on its own CDN: one request per second, the bot user
agent, refused when the answer's content type is not an image, or when no picture library can read
the bytes), then `imagery.store` keeps the original under `originals/<level>/` (never served) and
writes `images/<level>/<id>-<sha8>.webp` and `-thumb.webp`, the long side 1200 and 400, alpha kept,
EXIF dropped, CMYK converted; `<id>` is the target's slug or variant id and `<sha8>` the content
hash, so the address is immutable and `/uploads/images/` carries the one-year cache header. The
entry gains `url` and `thumb_url`; a refused or unreadable source gains `stage_error` and no
address. Idempotent: an entry whose derivative exists is skipped without a read or a request, and
the same file or address chosen at two levels is read once. It prints per level `stored`,
`existing`, `failed` and the number of downloads, and writes the manifest back in place; the
manifest is committed with its addresses (the site's hero addresses are read from it). Every
download is announced in a `.logs/runs/window-<date>-images.md` entry before the first request,
with the hosts and expected counts, and closed with the counts. `--check` reads and requests
nothing and reports what a run would read, download and store.

### Import
The staged manifest into the database, in the container, after a deploy that carries the image
migration:

```bash
docker exec dutyfreeprofessor-app python -m app.cli images import --manifest /srv/import/images/<supplier>-<date>.json --check
docker exec dutyfreeprofessor-app python -m app.cli images import --manifest /srv/import/images/<supplier>-<date>.json
```

For each chosen entry with a `url` it verifies the derivative exists under `/srv/uploads`
(`--uploads` elsewhere), resolves the target (a brand or line by slug, an alias followed to the
row a picture is set on; a variant by id, a merged-away id followed to its survivor, else by the
barcode the manifest matched it on) and writes the picture through the one writer,
`imagery.set_image`, with the source, level, licence, attribution and the time. An entry no row
answers to is counted `unresolved` and skipped; one whose derivative is not on disk is
`missing_derivative`; one without an address is `not_staged`. An entry already applied is
`unchanged`, so a re-run is a no-op, and a fetched picture never replaces a supplied one: the
refusal is counted as `image_admin_kept` per level and in total. `--check` resolves, counts and
rolls back. After a deploy that carries the image migration: `backfill image_sources --check`,
then `images stage`, then `images import --check`, then `images import`; `images coverage`
before and after, so the handoff carries both.

Pictures reach every page through one cascade (`catalog_queries.pictures_of`, one query for the
lines and one for the brands per page): a product variant shows its own picture, else its
product line's, else its brand's, and each card, page and served body carries `image_level` and
`image_credit` with it. A brand mark shown in a bottle's place is classed `thumb--brand` and never
emitted as the product's `image` in structured data or as its Open Graph picture (`SEO.md`).

### Logos
A brand mark from Wikidata and Wikimedia Commons for each brand without a picture
(`services/images_public.py`), in the container after the same deploy:

```bash
docker exec dutyfreeprofessor-app python -m app.cli images logos --limit 3 --check
docker exec dutyfreeprofessor-app python -m app.cli images logos [--limit N] [--brand <slug>]
```

The brands visited are canonical, not hidden and empty at brand level, the ones with the most
product variants first. For each, precision first: `wbsearchentities` on the exact name; only an
entity whose English label equals the brand after NFKD case folding; only one whose `P31` is a
kind a brand can be (`BRAND_KINDS`: brand, trademark, business, enterprise, company, distillery,
winery, manufacturer); two such entities is a stop; its `P154` logo, else `P18`; the file's
`imageinfo` with `extmetadata`, and only a free licence (public domain, CC0, CC BY, CC BY-SA at
any version; never NC, ND, fair use or a copyrighted logo). Anything else is refused with the
reason, printed per brand and counted (`no_entity`, `label_mismatch`, `kind_outside_set`,
`ambiguous`, `no_logo`, `no_file_info`, `licence_not_free`, `no_answer`), and the row stays
empty. An accepted logo is downloaded (the Commons render at 1200 px, so an SVG arrives as a
PNG; never hotlinked), stored through `imagery.store` under `brand/<slug>-<sha8>` and written
through `imagery.set_image` as `public:wikimedia-commons` with the licence short name and the
attribution "Wikimedia Commons: <File> by <Artist>, <description URL>"; a brand holding a
supplied picture is `image_admin_kept` without a download. Every request, on both hosts, waits
one second after the last (`--delay`), with the bot user agent; `--check` looks up and prints
what it would apply, downloads nothing and writes nothing. Three or four requests per brand, so
a full run over every brand is hours and is rian's to start, inside an announced
`.logs/runs/window-<date>-images.md` entry; a `--remove` for a takedown is written when first
asked for.

**Robots first.** Before the first request to each host (`www.wikidata.org`,
`commons.wikimedia.org`, `upload.wikimedia.org`) the command reads that host's robots.txt through
the collectors' one matcher and policy (`images_public.RobotsGuard`); a matching Disallow, a host
that answers robots.txt with 401 or 403, or a robots.txt that cannot be read stops the whole run
at that brand, counted `stopped:robots_disallow` or `stopped:robots_unavailable`, nothing
written, exit 2 (`REFUSED`). No workaround, no second identity. Both Wikimedia hosts publish
`Disallow: /w/` for every agent, and `/w/api.php` is where the API lives, so **`images logos`
exits refused until rian rules otherwise** on the running list (the decision names the
alternative: Wikimedia's API etiquette welcomes a bot with a contact user agent, but under this
project's rule a Disallow is a no). The 19 Sep probe ran before the guard existed and asked those
paths; the window entry records it.

### Lines
A product line's picture from its own representative variant, no network, after `images fetch`
has run or after the import:

```bash
docker exec dutyfreeprofessor-app python -m app.cli images lines --check
docker exec dutyfreeprofessor-app python -m app.cli images lines [--limit N]
```

Each product line with no picture whose representative variant (the line page's rule: the most
comparison units among its priced variants, then the lowest id) carries an Open Food Facts
photo takes that photo, same address, level `line`, licence and attribution copied, through the
one writer. A line with no priced variant (`no_representative`), one whose representative has
no picture (`representative_without_picture`), one whose only photos sit on other sizes
(`other_size_only`: empty beats guessed, the card names the representative's size) and one
whose representative's picture was supplied rather than fetched (`representative_not_off`: the
manifest assigns line pictures itself) each stay empty and are counted. A second run is a no-op.

### Coverage
How many pictures the catalogue holds, per level, no network, nothing written:

```bash
docker exec dutyfreeprofessor-app python -m app.cli images coverage
```

One line per level over the canonical rows (a brand or product line that is no alias, a product
variant not merged away; hidden ones included, since coverage is of the catalogue, not the site):
`total`, `admin` (supplied), `public` (fetched), `without` a picture of its own, and for those
what the cascade shows instead, a product variant's `from_line` and `from_brand`, a product line's
`from_brand`, and `none` at any level. The same count (`services/image_ask.coverage`) is the
per-level stat cards on `/images`, whose rows say what each card shows today (`Adam's` or `public`,
and `line` or `brand mark` when the picture is not the row's own) and whose "Wanted first" lists
name, per level, the rows that show nothing anywhere in their cascade, by airports carried
(`/api/images/wanted`). Run it before and after an import and put both in the handoff.

## CLI reference
Generated from `python -m app.cli --help` by `main/scripts/docmap.py`; do not edit by hand.

<!-- docmap:cli:start -->
```text
usage: app.cli [-h]
               {collect,awards,discussion,backfill,rederive,review,suggest,propose,sweep,status,shops,verify,verify-status,verify-clear,audit,indexnow,articles,subscribers,accounts,sessions,audit-log,notify,staging-refresh,hours,images,featured,places,decisions,proposals,precedents,pass,index,brands}
               ...

positional arguments:
  {collect,awards,discussion,backfill,rederive,review,suggest,propose,sweep,status,shops,verify,verify-status,verify-clear,audit,indexnow,articles,subscribers,accounts,sessions,audit-log,notify,staging-refresh,hours,images,featured,places,decisions,proposals,precedents,pass,index,brands}
    collect             run collectors into the database
    awards              import competition medals onto product variants
    discussion          the decision cards import, and the inbox a session reads
    backfill            run one idempotent data move after a deploy
    rederive            recompute match keys under the current identity rules
    review              hide or clear the human publish gate on a brand or product line page
                        (checked is retired: an approved sheet is the quality fact)
    suggest             fill the merge queue at brand, line and product level from the rules
    propose             file a proposals JSON as merge candidates a person approves on the
                        desk
    sweep               a collection sweep: what it reaches and what comes first
    status              show catalog and collector status
    shops               list shops and toggle site visibility (collectors keep running)
    verify              re-read a sample of published listings, one host at a time
    verify-status       per-source rolling pass rate, blocks, spot-checked date
    verify-clear        a human clears failing checks (who and when are recorded)
    audit               no-network data audit: metrics against thresholds, review lists
    indexnow            tell the IndexNow engines which pages changed
    articles            editorial text: import hand-ins, list, publish
    subscribers         email capture: export the list (owner only), counts
    accounts            accounts: list, owner-check, bench, and the credential commands
    sessions            sessions: revoke one account's or everyone's, prune old rows
    audit-log           the audit log: prune the noisy actions to a bounded history
    notify              the notification digest mail (host cron, every thirty minutes)
    staging-refresh     the client-written rows out before a restore, and back in after
    hours               opening hours: collect from operator sites, set by hand, show
    images              pictures at three levels: fetch (Open Food Facts), manifest, stage,
                        import, logos, lines, coverage
    featured            featured selection v3: the admin's pin (pin, unpin, list) and the
                        evidence a rail is measured by
    places              places: the guide document in and out, and what each place holds
    decisions           the ledger: list, undo a batch, verify the columns, export and replay
    proposals           the review: load a proposals file, read a sheet, approve, withdraw a
                        pass
    precedents          the register of kinds of judgement a person has answered (K12)
    pass                a review pass's packet, the settled state per brand, the one done line
                        (K12)
    index               which pages search engines are shown: suggest, list, approve, remove
    brands              the brand rows: their listed spellings, and splitting one the fold
                        joined wrongly

options:
  -h, --help            show this help message and exit

usage: app.cli collect [-h] [--source SOURCE] [--limit LIMIT] [--delay DELAY]
                       [--mode {discover,recheck}] [--by BY] [--ignore-freeze]

options:
  -h, --help            show this help message and exit
  --source SOURCE       collector slug; default is every enabled collector
  --limit LIMIT         max listings per source
  --delay DELAY         politeness delay in seconds
  --mode {discover,recheck}
                        discover reads the listing pages; recheck reads back the held listings
  --by BY               who started it, recorded on the run (the page passes the user)
  --ignore-freeze       run even while the deploy freeze marker exists

usage: app.cli awards [-h] [--file FILE] [--rebuild]

options:
  -h, --help   show this help message and exit
  --file FILE  winners JSON (default /srv/import/winners.json)
  --rebuild    drop competition-network medals and re-match from scratch

usage: app.cli discussion [-h] {import,inbox,rethread} ...

positional arguments:
  {import,inbox,rethread}
    import              import the decisions/realities list onto /discuss
    inbox               what rian said to @claude on the running list, oldest first
    rethread            apply a re-threading map once: topics, moves, soft deletes (T18)

options:
  -h, --help            show this help message and exit

usage: app.cli backfill [-h] [--check] [--force] [--map TYPED=USERNAME] [--rename]
                        [--include-defaulted]
                        {accounts,attribute_keys,attributes,authors,brands,categories,false_refusals,image_provenance,image_sources,implausible_sizes,legacy_mentions,level_permissions,levels,lines,listed,merges,options,orphan_tiles,overrides,perfume,places,prune_lines,quantities,rule_proposals,shops,sizes,stuck_runs,suggestion_decisions,thread_reads_seed,threads,vertical}

positional arguments:
  {accounts,attribute_keys,attributes,authors,brands,categories,false_refusals,image_provenance,image_sources,implausible_sizes,legacy_mentions,level_permissions,levels,lines,listed,merges,options,orphan_tiles,overrides,perfume,places,prune_lines,quantities,rule_proposals,shops,sizes,stuck_runs,suggestion_decisions,thread_reads_seed,threads,vertical}

options:
  -h, --help            show this help message and exit
  --check               print what would change; write nothing
  --force               run a data move even while a collection is running
  --map TYPED=USERNAME  authors only: link this typed name to this account (repeatable)
  --rename              authors only: also rewrite each mapped spelling to the account's
                        display name
  --include-defaulted   authors only: also link feature_priorities and quote_selections, where
                        the page defaulted an anonymous actor to a name

usage: app.cli rederive [-h] [--force]

options:
  -h, --help  show this help message and exit
  --force     run even while a collection is running

usage: app.cli review [-h] [--reason REASON] [--by BY] {brand,line} id {hidden,clear}

positional arguments:
  {brand,line}
  id               the brand or line id
  {hidden,clear}

options:
  -h, --help       show this help message and exit
  --reason REASON  why, kept with the decision
  --by BY          the account username the decision is recorded against (default rian)

usage: app.cli suggest [-h] [--check]

options:
  -h, --help  show this help message and exit
  --check     print what would change; write nothing

usage: app.cli propose [-h] --file FILE [--by BY] [--check]

options:
  -h, --help   show this help message and exit
  --file FILE  the proposals file (import/proposals/<date>-<level>.json)
  --by BY      who proposed (default: the file's proposed_by)
  --check      report the counts; write nothing

usage: app.cli sweep [-h] {plan} ...

positional arguments:
  {plan}
    plan      the sources a sweep reaches, the refused ones it never contacts, and the two
              backfills that come first

options:
  -h, --help  show this help message and exit

usage: app.cli status [-h]

options:
  -h, --help  show this help message and exit

usage: app.cli shops [-h] [--show CODE [CODE ...]] [--hide CODE [CODE ...]]
                     [--only CODE [CODE ...]]

options:
  -h, --help            show this help message and exit
  --show CODE [CODE ...]
                        make these visible
  --hide CODE [CODE ...]
                        hide these from the site
  --only CODE [CODE ...]
                        make exactly these visible and hide everything else

usage: app.cli verify [-h] [--n N] [--source SOURCE] [--seed SEED]
                      [--mode {on_demand,after_collection,weekly}]

options:
  -h, --help            show this help message and exit
  --n N                 listings per source (20 after a collection, 200 weekly)
  --source SOURCE       collector slug; repeatable; default every eligible source
  --seed SEED           sampling seed (stored; random by default)
  --mode {on_demand,after_collection,weekly}

usage: app.cli verify-status [-h]

options:
  -h, --help  show this help message and exit

usage: app.cli verify-clear [-h] --check CHECK [CHECK ...] --note NOTE [--by BY]

options:
  -h, --help            show this help message and exit
  --check CHECK [CHECK ...]
                        check ids from verify-status
  --note NOTE           what was looked at and why it is fine
  --by BY               account display name (default rian)

usage: app.cli audit [-h] [--out OUT] [--json] [--seed SEED] [--no-store] [--fail-on-over]

options:
  -h, --help      show this help message and exit
  --out OUT       directory for the JSON snapshot (default: the workspace's .logs/verification
                  when present)
  --json          print the whole snapshot instead of the summary
  --seed SEED     sampling seed, so a review list is reproducible
  --no-store      do not write audit_snapshots
  --fail-on-over  exit 1 when any metric is over its threshold

usage: app.cli indexnow [-h] [--since SINCE] [--hours HOURS] [--dry-run] [--show SHOW]

options:
  -h, --help     show this help message and exit
  --since SINCE  ISO timestamp; pages with observations after it are submitted
  --hours HOURS  or: pages changed in the last N hours (default 24)
  --dry-run      list the URLs, submit nothing
  --show SHOW    how many URLs a dry run prints

usage: app.cli articles [-h] {import,list,publish,unpublish} ...

positional arguments:
  {import,list,publish,unpublish}
    import              a Markdown, text or Word file into the articles table (draft)
    list                published articles (--all includes drafts)
    publish             make articles public
    unpublish           take articles back to draft

options:
  -h, --help            show this help message and exit

usage: app.cli subscribers [-h] {export,stats,unsubscribe} ...

positional arguments:
  {export,stats,unsubscribe}
    export              CSV of active subscribers to stdout or --out FILE (mode 600)
    stats               counts by source; no addresses
    unsubscribe         record a withdrawal for one address

options:
  -h, --help            show this help message and exit

usage: app.cli accounts [-h]
                        {create,set-password,invite-link,reset-link,disable,enable,unlock,list,owner-check,bench}
                        ...

positional arguments:
  {create,set-password,invite-link,reset-link,disable,enable,unlock,list,owner-check,bench}
    create              an account row and its level; no password yet
    set-password        set a password (prompted); clears the lock; signs out everywhere
    invite-link         print a one-time welcome link (48 h)
    reset-link          print a one-time reset link (48 h)
    disable             disable an account and revoke its sessions
    enable              re-enable a disabled account
    unlock              clear a lockout
    list                every account with status, level and whether a password is set
    owner-check         is ACCOUNT_OWNER set, present and able to sign in?
    bench               the argon2 cost on this host

options:
  -h, --help            show this help message and exit

usage: app.cli sessions [-h] {revoke,prune} ...

positional arguments:
  {revoke,prune}
    revoke        sign one account out everywhere, or everyone (--all, the break-glass)
    prune         delete expired sessions, old revoked ones, used and old tokens

options:
  -h, --help      show this help message and exit

usage: app.cli audit-log [-h] {prune} ...

positional arguments:
  {prune}
    prune     delete login and logout rows older than --keep-days

options:
  -h, --help  show this help message and exit

usage: app.cli notify [-h] {digest} ...

positional arguments:
  {digest}
    digest    one mail per person for the notifications not yet delivered

options:
  -h, --help  show this help message and exit

usage: app.cli staging-refresh [-h] {export,apply} ...

positional arguments:
  {export,apply}
    export        every client-written row as one JSON document on stdout
    apply         put an exported document's rows back (by natural key, never losing one)

options:
  -h, --help      show this help message and exit

usage: app.cli hours [-h] {collect,set,show} ...

positional arguments:
  {collect,set,show}
    collect           read the airports that have a collector, under robots
    set               write an airport's hours by hand, dated and named to who entered them
    show              the current hours and their provenance per airport

options:
  -h, --help          show this help message and exit

usage: app.cli images [-h] {fetch,manifest,stage,import,logos,lines,coverage} ...

positional arguments:
  {fetch,manifest,stage,import,logos,lines,coverage}
    fetch               attach openly licensed product imagery from Open Food Facts
    manifest            classify a picture folder and a brand-owner export into a reviewable
                        manifest (no network)
    stage               copy each chosen manifest entry into the uploads home as derivatives
                        and write its address back (files, and one download per address)
    import              set each staged manifest entry's picture on its brand, product line or
                        product variant (database only)
    logos               a brand mark from Wikidata and Wikimedia Commons for each brand
                        without a picture (exact label, a kind a brand can be, a free licence,
                        else refused)
    lines               promote the representative variant's Open Food Facts photo to each
                        product line without a picture (no network)
    coverage            per level, how many brands, product lines and product variants hold a
                        picture (supplied and fetched apart), how many none, and what the
                        cascade shows (no network, nothing written)

options:
  -h, --help            show this help message and exit

usage: app.cli featured [-h] {pin,unpin,list,evidence} ...

positional arguments:
  {pin,unpin,list,evidence}
    pin                 pin product lines or product variants: they lead every featured list
                        they are a comparison in
    unpin               release pins (a ledger decision; undo with decisions undo-batch)
    list                every pin, newest first, marked inert (leads nothing) or drift (column
                        and ledger disagree)
    evidence            measure each page's first N against its pool; read-only, nothing
                        written

options:
  -h, --help            show this help message and exit

usage: app.cli places [-h] {guide,list} ...

positional arguments:
  {guide,list}
    guide       a place's written guide, stored on the place
    list        every place, its identifiers, and whether it has a guide and hours

options:
  -h, --help    show this help message and exit

usage: app.cli decisions [-h] {list,undo-batch,verify,export,replay} ...

positional arguments:
  {list,undo-batch,verify,export,replay}
    list                the tracked-changes view: newest first
    undo-batch          reverse a whole batch in one transaction, by uid
    verify              read-only: every effective decision against its column; non-zero on
                        drift
    export              JSON lines by natural key, no local id, in (decided_at, id) order
    replay              apply an export on this host by natural key; parked rows go to a
                        remainder file

options:
  -h, --help            show this help message and exit

usage: app.cli proposals [-h] {load,withdraw,sheet,hints,notes,approve} ...

positional arguments:
  {load,withdraw,sheet,hints,notes,approve}
    load                load one brand's proposals file under its pass (idempotent by content)
    withdraw            withdraw a pass: waiting rows withdrawn, approved batches listed with
                        their undo
    sheet               read-only: the sheets in value order, or one brand's sheet
    hints               read-only: what the word lists would have suggested for a brand (the
                        pass's input)
    notes               read-only: the notes rian left on deferred and rejected questions (a
                        pass reads these first)
    approve             approve a brand's proposals as one batch; reject rows (a pair's reject
                        is Keep separate)

options:
  -h, --help            show this help message and exit

usage: app.cli precedents [-h] {list,show,overturn,export} ...

positional arguments:
  {list,show,overturn,export}
    list                every precedent with its counts
    show                one precedent and the decisions made under it
    overturn            overturn a precedent: one undoable decision; regroups nothing
    export              write docs/PRECEDENTS.md (generated; never edited by hand)

options:
  -h, --help            show this help message and exit

usage: app.cli pass [-h] {packet,status,fingerprint,parked,unpark} ...

positional arguments:
  {packet,status,fingerprint,parked,unpark}
    packet              everything a pass may know about one brand, as one JSON document
    status              per brand: parked | open | deferred | settled | unsettled | never; and
                        the done line (exit 3 until done)
    fingerprint         the brand family's listed-words fingerprint a settled pass stamps
    parked              questions parked to rian, with the running-list command for each
    unpark              rian answered a parked question: it becomes a deferral carrying his
                        answer

options:
  -h, --help            show this help message and exit

usage: app.cli index [-h] {suggest,list,approve,remove} ...

positional arguments:
  {suggest,list,approve,remove}
    suggest             write the pages that meet their rule as suggestions to index
                        (idempotent; applies nothing)
    list                read-only: the suggestions waiting, with each page's facts
    approve             approve pages for indexing (one batch, one undo)
    remove              remove pages from the index (one batch, one undo)

options:
  -h, --help            show this help message and exit

usage: app.cli brands [-h] {spellings,split} ...

positional arguments:
  {spellings,split}
    spellings        every listed spelling a brand row holds, with how many product variants
                     carry it
    split            move listed spellings off a brand row onto one of their own, as one
                     recorded decision

options:
  -h, --help         show this help message and exit
```
<!-- docmap:cli:end -->

## Production (bwlive)

The live host is a DigitalOcean droplet in Toronto, `bwlive` on the tailnet (see
`deploy/` for every script named here). It mirrors the dev server's shape: the same compose
file plus `docker-compose.production.yml`, Caddy alone on 80 and 443, the app on the
bridge port, Postgres on the internal network with no published port. The app gates
itself (`SITE_ACCESS`); Caddy carries no gate.

- **Reach it:** `ssh deploy@bwlive` (keys only, from the tailnet or the dev server; root is off).
  Break-glass if the tailnet is gone: DigitalOcean's recovery console after a root password reset.
- **Deploy:** `deploy/production.sh` from this repository on the dev server. It dumps the
  production database first, rsyncs `main/`, `import/`, `public/`, `uploads/` and the compose
  files over Tailscale, builds there, recreates, and waits for `/api/health`. Add `--caddy`
  when `deploy/caddy/` changed. It never copies `.app.env` or `data/`. A deploy kills a
  running collection there too: check `.logs/runs/` first.
- **The launch: production becomes staging** (plan W17; the Launch checklist below): `deploy/production.sh --replace-db
  backups/launch/staging-<stamp>.dump`, from a dump `deploy/launch-dump.sh` wrote with its
  `.counts`, `.head` and `.sources` sidecars. Every step is fatal: the dump, its sidecar and this
  code must name one alembic head; production is dumped first to
  `backups/production/pre-replace-<stamp>.dump`; the app stops; the `public` schema is dropped and
  the dump restored with `--exit-on-error`; the restored head and every table's row count must equal
  the sidecar's; every session is revoked and every unused welcome or reset link expired; the sources
  are printed beside staging's for a typed `yes`; the app starts and health prints the version. A
  refusal leaves the app stopped and prints the rollback. `--dry-run` prints every command;
  `--local --db-container dfp-devdb --db <copy> --app-exec "<app.cli on that copy>"` rehearses the
  database half on the dev database. Follow it from the Launch checklist below, never on its own.
  (`--seed-db`, which swallowed restore errors, is gone.)
- **The launch runs once.** A finished `--replace-db` writes `backups/production/LAUNCHED` here
  (`launched <stamp> from <dump>`) and a copy in the droplet's `backups/`; while that file exists
  `--replace-db` refuses, before step 1 and before any ssh, naming the file and
  `--force-replace-db`. The rule it holds: a push to live never syncs collection data again. From
  the launch on, live is where collection runs and where the catalogue is decided, so a second
  replace discards all of it; force it only with the rollback dump in hand. A `--local` rehearsal
  replaces a scratch database on the dev server, not live, and is never blocked.
- **Roll back:** check out the previous commit and run `deploy/production.sh` again; a
  migration that rode along is reversed by restoring the pre-deploy dump it took
  (`backups/dfp-<stamp>-pre-deploy.dump` on the droplet).
- **The migration chain from 0.35.0** (production's schema `b5c6d7e8f9a0`, migration #4). At the
  launch the restore carries staging's data and schema, so nothing below runs. It is the path for
  the one case where production is NOT a copy of staging (the rollback, or a code deploy without the
  replace): the container's `alembic upgrade head` takes the ten migrations in one pass
  (`c7d8e9f0a1b2` #6 threads, `c6d7e8f9a0b1` #6 lines, `d8e9f0a1b2c3` hours, `e9f0a1b2c3d4` the side
  panel, `f0a1b2c3d4e5` the workflow, `f1a2b3c4d5e6` flags, `a2b3c4d5e6f7` ask kinds, `b3c4d5e6f7a8`
  quantities and the listed layer, `c1d2e3f4a5b6` K1's names, `d2e3f4a5b6c7` K2's ledger), then these,
  in this order, each safe to repeat, with no collection running. Every backfill runs on the head code,
  so the per-migration blocks above collapse: one `rederive` and one fold serve migration #6, Stream L,
  rules v5 and rules v6 alike.

  ```
  docker exec dutyfreeprofessor-app alembic current                                # must print d2e3f4a5b6c7 (head)
  docker exec dutyfreeprofessor-app python -m app.cli backfill threads             # the side panel's threads
  docker exec dutyfreeprofessor-app python -m app.cli backfill thread_reads_seed   # everything before is read
  docker exec dutyfreeprofessor-app python -m app.cli backfill quantities --check
  docker exec dutyfreeprofessor-app python -m app.cli backfill quantities          # before rederive: the key reads them
  docker exec dutyfreeprofessor-app python -m app.cli backfill suggestion_decisions
  docker exec dutyfreeprofessor-app python -m app.cli backfill attribute_keys
  docker exec dutyfreeprofessor-app python -m app.cli backfill places
  docker exec dutyfreeprofessor-app python -m app.cli backfill options --check
  docker exec dutyfreeprofessor-app python -m app.cli backfill options
  docker exec dutyfreeprofessor-app python -m app.cli backfill attributes
  docker exec dutyfreeprofessor-app python -m app.cli backfill lines
  docker exec dutyfreeprofessor-app python -m app.cli rederive
  docker exec dutyfreeprofessor-app python -m app.cli rederive                     # must print zero everywhere
  docker exec dutyfreeprofessor-app python -m app.cli backfill merges --check      # READ every group first
  docker exec dutyfreeprofessor-app python -m app.cli backfill merges
  docker exec dutyfreeprofessor-app python -m app.cli backfill listed --check
  docker exec dutyfreeprofessor-app python -m app.cli backfill listed              # after merges: onto the survivors
  docker exec dutyfreeprofessor-app python -m app.cli suggest
  docker exec dutyfreeprofessor-app python -m app.cli backfill prune_lines --check
  docker exec dutyfreeprofessor-app python -m app.cli backfill prune_lines
  docker exec dutyfreeprofessor-app python -m app.cli backfill rule_proposals      # proposals only; applies nothing
  docker exec dutyfreeprofessor-app python -m app.cli decisions verify             # 0 drift
  ```

  **Absent on purpose:** the three `propose --file /srv/import/proposals/2026-09-15-*.json` lines
  of the Stream L block. Those files carry staging's row ids and would write decisions onto the
  wrong rows here; a review reaches production by `decisions export` and `decisions replay`, or
  by the launch's replace. Also absent: `discussion rethread`, whose map names staging's comments of
  13 Sep. Rehearsed 17 Sep on a copy of production's nightly
  of 16 Sep (`b5c6d7e8f9a0`) on `dfp-devdb`, every line exit 0, 292 s in all: `threads` 64 threads and 146
  comments; `quantities` 17,340 written; `places` 21; `options` 3,363 named by title and 915 given option
  fields; `lines` 13,890 created; `rederive` 17,389 keys, then 0 everywhere; `merges` 191 groups folded
  (201 rows forward; production never ran the Stream L fold, so this number is large by design: read the
  `--check` groups), 201 left for a person; `listed` 11,115 listings; `suggest` 3,059 pairs; `prune_lines` 8;
  `rule_proposals` 1,249, none applied; `decisions verify` 0 drift; `alembic_version` `d2e3f4a5b6c7`.
- **Restore from dump** (rehearse into a scratch database first, as on dev):
  `docker exec -i dutyfreeprofessor-db pg_restore -U dfp -d dfp --clean --if-exists --no-owner < backups/<file>.dump`
- **Backups:** the deploy user's crontab dumps nightly at 03:15 UTC into `backups/`, 14 days;
  the dev server pulls that directory into `backups/production/` at 04:15 local
  (`deploy/install-backup-cron.sh` installs both). DigitalOcean snapshots, if enabled, are
  the whole-box tool; the dumps are the everyday one.
- **Secrets:** `.app.env` on the droplet (0600, deploy user). Machine secrets were generated
  there; the owner's values are filled in his own shell, never in chat. Caddy's Cloudflare
  DNS token lives in `/srv/caddy/cloudflare.env`, scoped to the one zone. Rotate by editing
  the file and recreating the container.
- **Certificates:** a Cloudflare Origin CA certificate (15 years) in `/srv/caddy/origin/` on the
  droplet, trusted only by Cloudflare's edge, so the zone must stay proxied and Full (strict).
  Renew by creating a new one in the dashboard and pasting it over the two files, then
  `docker compose up -d` in `/srv/caddy`. The DNS-challenge module stays configured for any
  un-proxied host (`deploy/caddy/`).
- **Egress:** `deploy/egress-test.py` re-runs the retailer check from any machine; compare
  against a same-day run from the dev server before trusting a new address.
- **The sidecar fence:** `deploy/sidecar-firewall.sh`, owner-applied, keeps the browser
  sidecar off private ranges and the cloud metadata service.

## Launch checklist (staging to production)

The soft launch's one page (plan W17; Stream K7). rian runs it top to bottom from the workspace root on
the dev server, on the merged branch; each line is a command or a check with what it must print. Stop
at the first line that does not print what it says. The rehearsal that proved the replace is
`.logs/runs/rehearsal-2026-09-17-k7-replace.md`.

**Before**

1. Freeze collections on both machines. `pgrep -af 'app.cli (collect|verify|hours)'` here and
   `ssh deploy@bwlive "pgrep -af 'app.cli (collect|verify|hours)'"` print nothing; open
   `.logs/runs/window-<date>-launch.md` saying collections are frozen until step 21.
2. The last staging collection finished and verified: `docker exec dutyfreeprofessor-app python -m app.cli
   verify-status` prints no uncleared MISMATCH or PARSE_FAIL (a blocked source hides its shops on the
   site, which is the tripwire working, not a reason to launch around it).
3. The review is complete for the brands in scope: `/review` lists no sheet with waiting rows you meant
   to approve (`docker exec dutyfreeprofessor-app python -m app.cli proposals sheet` prints the rest), and
   `docker exec dutyfreeprofessor-app python -m app.cli decisions verify` prints `0 drift`.
4. Indexing decided (K6): the product line, brand and airport pages you approved show `indexed`;
   everything else stays `noindex, follow` by default.
5. The branch is merged and green: `git status` clean on `master` with the refactor merged;
   `main/check.sh` prints `checks passed`; `main/app/version.py` and `main/CHANGELOG.md` name the launch
   version (`bash main/docs-check.sh` prints `0 fail`).

**Staging on the launch code**

6. `docker exec dutyfreeprofessor-db pg_dump -U dfp -Fc dfp > backups/dfp-$(date +%F)-pre-launch.dump`
   (staging's own rollback).
7. `srv-gw deploy --project dutyfreeprofessor --build`; then `curl -s
   https://dutyfreeprofessor.demoing.info/api/health` prints the launch version.
8. The after-deploy blocks the branch carries, in order, on staging (`RUNBOOK.md` Migrations): K1's
   (`backfill suggestion_decisions`, `backfill attribute_keys`), K2's (`backfill places`, `decisions
   verify`), K3's rules v6 block (ending `decisions verify` at `0 drift`), and any K4 to K6 line their
   handoffs name. `docker exec dutyfreeprofessor-app alembic current` prints the code's head.

**The replace**

9. The staging dump with its sidecars: `deploy/launch-dump.sh` prints the dump, its tables and rows,
   the head (equal to step 8's) and the sources.
10. The dry run: `deploy/production.sh --replace-db backups/launch/staging-<stamp>.dump --dry-run`
    prints the three heads equal and every command, and ends `dry run: nothing ran`.
11. The replace: `deploy/production.sh --replace-db backups/launch/staging-<stamp>.dump`. It prints, in
    order: the three heads equal; the rollback dump `backups/production/pre-replace-<stamp>.dump`
    arrived; build; `stop app`; the restore; `alembic_version <head>`.
12. The counts: `<N> tables, <M> rows: equal`, the same N and M step 9 printed.
13. Sessions: `revoked <n> session(s): everyone is signed out` and `expired <k> unused welcome or reset
    link(s)`; nobody is signed in on production, including you.
14. Sources: the two columns match line for line (slug, enabled, delay, identity mode); type `yes`.
15. Health: the script prints `{"status":"ok","version":"<launch version>","database":"ok","role":"staging"}`
    and `done: production holds staging's database`. `staging` is the correct answer here, because
    `SITE_ROLE` is not set until step 17. Then `srv-gw security-audit` shows nothing new.

**Go live**

16. Sign in on production as rian (`https://dutyfreeprofessor.com/login`); a product line page, an
    airport page and `/review` open.
17. The two environment lines: on the droplet, in your own shell, `SITE_ROLE=live` and then
    `SITE_ACCESS=public` in `.app.env` (both or neither: without the first the development
    surfaces stay on the public site); then
    `ssh deploy@bwlive "cd /srv/apps/dutyfreeprofessor && docker compose -f docker-compose.yml -f
    docker-compose.production.yml up -d app"`; `curl -s https://dutyfreeprofessor.com/api/health`
    prints ok with `"role":"live"`.
18. The unauthenticated sweep, signed out:
    `main/scripts/route-sweep.py --open --live https://dutyfreeprofessor.com` exits 0. It fetches
    every route the policy names with no cookie and prints one line each; a member or owner route
    answering 200, or redirecting anywhere but the sign-in page, exits 2 and names the route
    (`--open` is `SITE_ACCESS=public`, `--live` is `SITE_ROLE=live`, both set in step 17). For a
    spot check by hand:
    `curl -s -o /dev/null -w '%{http_code}\n' https://dutyfreeprofessor.com/api/review/sheets` prints `401`,
    and `/api/collectors`, `/api/plan`, `/api/items` print `401` or `403`.
19. The sitemap lists only approved pages (K6): `curl -s https://dutyfreeprofessor.com/sitemap.xml | grep
    -c '/products/'` equals `ssh deploy@bwlive "docker exec dutyfreeprofessor-db psql -U dfp -d dfp -tAc
    'select count(*) from product_lines where indexed and not hidden'"`; a `noindex` page is never in it.
20. IndexNow: `curl -s -o /dev/null -w '%{http_code}\n' https://dutyfreeprofessor.com/<key>.txt` prints
    `200` for the key in production's `.app.env`.
21. Collections resume on production only (production collects from now on; staging refreshes from it):
    close the window file from step 1 with the time.
22. The second pass: read lines 11 to 20 again against what the terminal printed, from a phone on
    mobile data, signed out, on a product line page, an airport page and the home page.
23. Tell Adam: the site is live at `https://dutyfreeprofessor.com`, what is in scope, that every page is
    `noindex` until approved and why, and where to comment.

**Rollback** (any line from 11 on): the replace prints the command when it refuses. By hand, on the
droplet in `/srv/apps/dutyfreeprofessor`: `docker exec -i dutyfreeprofessor-db psql -U dfp -d dfp -v ON_ERROR_STOP=1 -c 'drop schema public
cascade; create schema public;'` then `docker exec -i dutyfreeprofessor-db pg_restore -U dfp -d dfp
--no-owner --no-privileges --exit-on-error < backups/pre-replace-<stamp>.dump` (the copy the script left on
the droplet; `backups/production/` holds the same file here), then check out the previous tag and
`deploy/production.sh` (`SITE_ACCESS=members` again if step 17 ran; leave `SITE_ROLE=live`
unless you are also redeploying pre-launch code and want the development surfaces back to
diagnose). Production's schema is then
0.35.0's again, and the Production section's chain is the way forward from it.

## Backups and restore
- Nightly `pg_dump` from rian's crontab at 03:15 into `backups/dfp-nightly-<date>.dump`,
  kept for 14 days; failures append to `.logs/runs/backup.log`. Manual dumps before risky
  work follow the same shape with a reason in the name.
- Restore (rehearse on `dfp-devdb` first):
  `docker exec -i dutyfreeprofessor-db pg_restore -U dfp -d dfp --clean --if-exists < backups/<file>.dump`
- `data/postgres/` is the live cluster and is not the backup.

## Staging refresh
**A refresh no longer discards a catalogue decision** (Stream K2; before K2 every alias, merge,
pin, ignore and review state was lost, the catalogue decisions of 15 Sep §6). Every one of those
is a `decisions` row now, and the ledger travels by natural key like the client-written rows do.
The guard now counts `decisions` and `proposals` (alongside the legacy alias, pin and pair-decision
columns, kept as a second, informational check) with `-v ON_ERROR_STOP=1`, and fails hard when it
cannot count (a guard that silently stopped firing once, on a dropped table, is the failure this
refuses to repeat). When the count is non-zero, the ledger is exported by natural key before the
schema is dropped and replayed after `alembic upgrade head`; `--discard-decisions` skips both the
export and the replay, for the rare refresh meant to start the ledger over, and the script says so
plainly rather than silently losing the rows. After a refresh from a production that is behind
staging, the after-deploy blocks the restored schema has not seen must run again.

Adam comments on staging, so a refresh from production must keep every table people write on
the site (`CLIENT_WRITTEN_TABLES` in `app/services/discussion.py`: the decision cards and
their comments, the feature ranking, the quote selections and requests, the to-dos and their
uploads, rian's running-list states; the list grows with the schema and the refresh reads it,
never a copy). One host command, from the workspace root:

```
python3 main/scripts/staging-refresh.py --from backups/production/<file>.dump --check
python3 main/scripts/staging-refresh.py --from backups/production/<file>.dump
```

`--check` exports every client-written row to `backups/refresh/<stamp>-client-rows.json`, prints
the counts and the commands the real run would issue, and restores nothing. The real run
exports again, drops and recreates the `public` schema in the database container, restores the
dump, runs `alembic upgrade head` in the app container (production may be behind staging),
`backfill accounts` and `backfill levels`, then puts the exported rows back through
`app.cli staging-refresh apply` and revokes every session (production's session rows are not
staging's; sign in again). The report says, per table, what was restored, what was already
there, what was updated or kept on a natural-key conflict (the newer row wins, a tie keeps
production's) and what was renumbered (a comment whose id production has since used comes back
under a fresh id; the deep link to the old id then lands on production's comment). Who-columns
are remapped by username, so account ids need not agree between hosts; an author whose account
does not exist on the restored side keeps the typed name with an empty link, and `backfill
authors --map` relinks once the account exists. A second run of `apply` on the same file
changes nothing. The export file is kept: whatever fails after it, the rows are on disk, and
`app.cli staging-refresh apply --from - < <file>` inside the container puts them back. The
seeded to-do text is production's after a refresh; re-run `seed-todos.py` on staging if it
should differ. The app container is not stopped; a request during the restore fails once.
Rehearse on `dfp-devdb` first: `--db dfp_r --db-container dfp-devdb --app-exec
"../.venv-dev/bin/python -m app.cli" --alembic-exec "../.venv-dev/bin/alembic"` with
`DATABASE_URL` pointed at `dfp_r`. Rehearsed 11 Sep with Adam's three comments of that day as
the rows that had to survive (`tests/test_staging_refresh.py` pins the policy on SQLite).

## Secrets
`.app.env` (mode 660) holds the database password and any service keys; the compose file
reads it for both containers. Never print it, never commit it, never paste a value into chat.

### Accounts
The site's own login (`ACCOUNTS.md` is the mechanism; this is the procedure). Two lines in
`.app.env`, staged as placeholders in `.app.env.example` at the workspace root:

```
ACCOUNT_OWNER=REPLACE_WITH_USERNAME   # the super admin; the owner is code, not a row
SITE_ACCESS=members                   # sign in to see anything; `public` is the go-live flip
SITE_ROLE=staging                     # `live` on the production host only; see below
```

`SITE_ROLE` is the environment line (`ACCOUNTS.md`, the route policy): `live` hides the
surfaces built for the build, anything else including an absent line reads as staging and
shows them. It is not the public gate, and the two lines are independent: a host may be
`live` and still members-only. Because unset shows more, never assume it from the file:
`curl -s <host>/api/health` carries `"role":"live"` or `"role":"staging"`.

Set them in your own shell; delete `OWNER_PASSWORD` and `SESSION_SECRET` if present (ignored
either way). Tighten the file so only you and the gateway read it: `setfacl -b .app.env && chmod
600 .app.env && setfacl -m u:srv-gateway:r .app.env` (a bare `chmod 600` sets the ACL mask to
nothing and `srv-gw deploy` fails, because the gateway's compose reads `env_file` as
`srv-gateway`); check with `sudo -u srv-gateway test -r .app.env && echo readable` (reads
nothing); `srv-gw fix-permissions` widens it back to 660 plus the group ACL, so re-run the
`setfacl` line after any permissions repair; `grep -c REPLACE_WITH_ .app.env` prints 0. Then
recreate the container. With the owner unset the startup log says CRITICAL, nobody is super
admin and the site stays up; `python -m app.cli accounts owner-check` says the same.

**Seeding a host**, inside the container after the deploy that carries migration #4, in order:

```
python -m app.cli backfill accounts
python -m app.cli backfill levels
python -m app.cli accounts create --username adam --email <his> --display-name Adam --level admin
python -m app.cli accounts create --username mark --email <his> --display-name Mark --level admin
python -m app.cli accounts set-password --username rian        # prompted twice, never argv
python -m app.cli accounts invite-link --username adam          # printed once; send it yourself
python -m app.cli accounts invite-link --username mark
python -m app.cli backfill authors --map Adam=adam --map Mark=mark --map rian=rian
```

A welcome link is `https://<host>/welcome#<token>`: the token is the fragment, so it never
reaches a request line. The container's stdout is an access log of every request line, so
nothing secret may ever sit in a URL path or query; the CLI prints a link once to your
terminal and to nothing else. Links work for 48 hours and once. `accounts reset-link` is the
same for a forgotten password until a mail provider exists (`MAIL_PROVIDER=resend` with
`RESEND_API_KEY` and `MAIL_FROM` in an app-only env file, never the shared one); then
`/forgot` on the site and "Send reset link" in the People panel work too. Other commands:
`accounts list | disable | enable | unlock`, `sessions revoke --username <u> | --all` (the
break-glass: everyone out at once), `sessions prune`, `audit-log prune`, `accounts bench`
(the argon2 cost on this host). The generated reference at the end of this file has the flags.

**Rollout on the dev host** (the staging order, in full, in the accounts plan §7): dump first
because a migration rides; the two lines above; the tightening; bump the version and deploy;
seed as above and `awards --rebuild` if it is still pending; the curl list from the host
against `http://172.17.0.1:<port>` (every POST with `-H 'Origin: https://<host>'`); the
browser round trip; `srv-gw security-audit`; then, owner only, `srv-gw id-gate --site <host>
--mode public`, after which the app's login is the only barrier (rollback: `srv-gw id-gate
--site <host>`); the curls again through the public host; `srv-gw security-audit` again.
Production repeats the seeding with its own passwords and links minted on its own
`PUBLIC_BASE_URL`; the curl list runs through the domain credential before it comes off.

**Transfer of ownership:** set `ACCOUNT_OWNER` to the new owner's username, create that
account (`accounts create`, `set-password`), recreate the container; the previous owner's row
is an ordinary account from then on, and nothing in the codebase points at this server.

**One worker.** The per-address login throttle and the argon2 semaphore live in the process:
one uvicorn worker is a launch assumption, and a `login_attempts` table comes before a second.

**Members-only and the flip.** `SITE_ACCESS=members` answers every anonymous page with a 302
to `/login`, serves `Disallow: /` and no sitemap, and stamps `X-Robots-Tag: noindex, nofollow`
on every response. The flip to `public` is the go-live checklist's step (with the IndexNow
key and the uptime checks), one env line and a container recreate; it is never set earlier.

### Mail
The provider is Resend behind three lines, `MAIL_PROVIDER=resend`, `MAIL_FROM=<a verified
sender>` and `RESEND_API_KEY`, and they belong in an **app-only env file**, never `.app.env`,
which the database container reads too (`docker-compose.yml` gives both containers the same
`env_file`; the split is Stream W's W3 and its compose change gets a security review and
`srv-gw security-audit` first). Until those lines exist every mail path is inert: invites and
resets go by CLI link, the People panel says so, and `notify digest` reports "no mail provider"
and stamps nothing. When they do: every send counts against a daily cap in `email_sends` (6 auth
mails per account: welcome, reset, access notice; 25 invites per host; 40 notification mails per
account), a spent cap refuses before the provider is called (`MAIL_CAP` in the People panel), and
the notification digest goes out from host cron on the machine that runs the app:

```
*/30 * * * * docker exec dutyfreeprofessor-app python -m app.cli notify digest >> /srv/apps/dutyfreeprofessor/.logs/runs/notify-digest.log 2>&1
```

One mail per person per run for the rows not yet delivered (a row already read in the app is
stamped delivered and not mailed; a person who turned mail off under Account, or has no email,
is skipped and their rows wait); `notify digest --check` prints what would go out and writes
nothing. Nothing about a message is logged beyond the provider's status.

## What lives where
Workspace root: docs, `.logs/`, compose, `import/` (read-only into the container, so the
`/plan` page and the awards and discussion imports update without a deploy), `public/`
(explainer pages served at `/<page>.html`), `data/`, `backups/`. Code: `main/`.