# Architecture Sources of truth: `main/app/`, `main/app/routers/`, `main/app/services/`, `main/web/src/`, `main/Dockerfile`, `docker-compose.yml`. One Python process serves both the API and the built React SPA. Postgres sits beside it on a private compose network. Collectors are plain Python modules run from the same image via `python -m app.cli`. There is no queue, no cache tier, no worker fleet, and no step that uses an LLM. ## The request path 1. Caddy terminates TLS and proxies the host to the container's published port on the Docker bridge (managed by the gateway; the port is in `.env`). 2. Every request, reads and pages included, passes two middlewares before routing: the session loader (`app/services/sessions.py`, the account cookie to `request.state.session`) and the access policy (`app/services/access.py`), which puts every route, page, mount and SPA path in exactly one class: public always, public when `SITE_ACCESS=public` and otherwise any signed-in account, any member, or one permission. A route in no class is refused before its handler runs; `tests/test_route_inventory.py` fails on one. Every write needs a matching Origin. While the site is members-only an anonymous page request is one 302 to `/login`. The whole account system: `ACCOUNTS.md`; the procedure for a host: `RUNBOOK.md` (Accounts); which surfaces the client sees: `CLIENT-SURFACES.md`. 3. FastAPI (`app/main.py`) routes `/api/*` to the routers; `/assets`, `/fonts`, `/medals` and `/flags` (the airport cards' country flags) are real static mounts so HEAD and range requests work for crawlers. 4. Any other path returns the built SPA shell, **stamped server-side** by `app/services/seo.py`: title, description, canonical, social tags and JSON-LD per page, and for the product, airport, category-at-airport, brand, article and data pages the body itself, so a client that runs no JavaScript reads the same page the SPA draws. The same process serves the machine surfaces (`/robots.txt` with per-bot policy, `/sitemap.xml`, `/feed.xml`, `/llms.txt`, the IndexNow key file); each mechanism and how to verify it is in `SEO.md`. Feature flags ride in the shell (`window.__DFP_FLAGS__`, from `app/config.py`). 5. Explainer pages in `public/` are served at `/.html` by rule; an absent page is a 404, never the SPA shell (`agents.md`, the catch-all rule). 6. The SPA (`web/src/App.tsx`) owns the routes; its typed client (`web/src/api/schema.ts`) is generated from the backend's OpenAPI schema at build time and by `check.sh`. 7. The site chrome is the SPA's (`web/src/components/SiteHeader.tsx`, `SiteFooter.tsx`), and the server-rendered pages carry a copy of the header in its closed state (`seo.py header_html()`) so nothing moves when React mounts; a change to the header's shape changes the mirror in the same deploy (`SEO.md`). The top row is the logo, the search (`SiteSearch.tsx`: suggestions as one types, airports and brands matched in the browser from the lists it holds, product variants from `/api/search/suggest`, a match-and-rank query without the price aggregation that makes `/api/products` too slow per keystroke; a code or a city typed whole opens that airport, `lib/search.ts`), the airports button and the profile circle (`AccountMenu.tsx`: the account links and the build-discussion pages this account may open, from `lib/clientPages.ts`). The nav leads with four departments (Airports, Drinks, Beauty, Articles; `MegaMenu.tsx` DEPARTMENTS, the families being taxonomy's), each a link to its own page and the toggle for its panel, and ends with My savings. The panel reads `/api/stats` (shelves by family), `/api/brands` (each brand's `family`), `/api/airports` (and warms its flags), the featured savings and the latest articles once the page is idle, or sooner on hover, focus or the drawer, so the first hover finds them in hand; a list the API returns empty is simply absent. "All drinks" is `/products?family=liquor`, the products API's `family` filter over the family's shown categories. Social profile links (`SocialLinks.tsx`) render only for URLs the shell injects as `window.__DFP_SOCIAL__`, the same mechanism as the feature flags. 8. The SPA draws its controls by capability: one `GET /api/bw/me` (the vendored pack's provider in `web/src/lib/auth.tsx`, refetched on focus and after writes, never on a timer) carries the kit's booleans and DFP's `capabilities` block, and `/plan`, `/sources`, curator mode, the client pages and the account menu draw from it; the server refuses the same routes whatever the SPA draws (step 2). The sign-in pages are DFP's own (`web/src/pages/LoginPage.tsx`, `ForgotPage`, `SetPasswordPage`, `AccountPage`); the People, Levels and Access panels on `/admin` are the pack's. Every string a shopper or the client reads is checked for house style by `tests/test_house_style.py`. 9. Conversations follow the Interaction Standard (`CLIENT-SURFACES.md` Comments, `ACCOUNTS.md` Notifications): one thread per subject, one `GET /api/discussion/threads` read per page that the vendored `caddie-ui` pack draws through `SubjectThread.tsx`, every write through `app/services/discussion.py` (the author from the session, mentions parsed from the stored body, the fan-out scoped by the subject's permission, every emit deduped), the bell polling `GET /api/notifications` every thirty seconds while the tab is visible, attachments through `app/routers/attachments.py` over the uploads seam, and the digest mail from host cron (`app/cli_notify.py`). The client's material is one page in tabs (`/discuss?tab=`), and the moved addresses answer a 301 that keeps the fragment. The one side panel (`web/src/components/DiscussionPanel.tsx`, the pack's `SidePanel` over the same threads read; state in `web/src/lib/discussionPanel.tsx` above the header, so the control beside the bell, a bell row and the account menu all open it) lists every thread the account may see with what each is to the viewer, computed by the server from `thread_reads` and the viewer's own notification rows, so the bell's count and the panel's Needs-you tab cannot disagree; opening a thread anywhere moves the stamp and clears the bell in one write. ## The pipeline `fetch.py → .collect() → RawListing → ingest.run_collector → normalize / taxonomy / fx → ProductVariant / Listing / PriceObservation`. Every collector yields the same `RawListing` shape; ingest owns identity resolution (GTIN first, the computed key as fallback: `services/keying.py` resolves the standard brand through the brand alias and the attribute through the alias table, then `normalize.match_key` composes standard brand, line, attribute, quantity and form from the pure rules in `services/product_lines.py` and `services/quantity.py`, the decided layer read first from `overrides` through `services/overrides.py`), the plausibility guards, and the race-tolerant row creation. The words every one of these is talked about in, each with its test: `VOCABULARY.md`. Right after the raw record is written, `services/collected.py` fills the listing's listed columns from that fragment. Details and per-platform recipes: `COLLECTORS.md`. The schema and its rationale: `DATA-MODEL.md`. What merges and what waits for a person, the merge session and the desk behind `/collectors`: `services/merges.py`, `services/suggest.py`, `services/merge_session.py`, `services/merge_desk.py` (their docstrings), and `RUNBOOK.md`; the giant Listings table is `services/listings_table.py` with `web/src/components/collectors/ListingsTable.tsx`, the desk `MergeDesk.tsx`. Medals join onto product variants after the fact (`app/services/awards_import.py`, one row per product variant, competition, year and medal, reconciled in place); which medal a card or page shows is `app/services/award_picker.py`. ## Build and run Three Docker stages (`main/Dockerfile`): Python deps and an OpenAPI dump; the typed client and SPA build against that dump; a runtime image that is Python only and runs `alembic upgrade head` then uvicorn. Compose (`docker-compose.yml`): `app` and `db`, the database with no published port, `import/` and `public/` mounted read-only. ## Module map Generated from each module's first docstring line by `main/scripts/docmap.py`; a module without a docstring shows as such, which is the signal to write one. Do not edit by hand. - `app/cli.py`: Operational commands. - `app/cli_accounts.py`: The account commands: `accounts`, `sessions` and `audit-log` (docs/ACCOUNTS.md). - `app/cli_brands.py`: The brand commands (Stream K9): `brands spellings` and `brands split`. - `app/cli_decisions.py`: The ledger's commands (Stream K2): `decisions list`, `decisions undo-batch`, `decisions verify`; - `app/cli_editorial.py`: The editorial commands: articles in, subscribers out. - `app/cli_hours.py`: The opening hours commands: `hours collect`, `hours set` and `hours show` (docs/COLLECTORS.md, Opening hours). - `app/cli_index.py`: The indexing commands (Stream K6; plan W18): `index suggest`, `index list`, `index approve`, - `app/cli_notify.py`: The notification digest: one mail per person for what arrived since the last run. - `app/cli_pages.py`: CLI subcommands for the crawl surface (Stream B): `indexnow`. - `app/cli_pass.py`: `app.cli pass packet | status | fingerprint | parked | unpark`: what a review pass is given, and - `app/cli_precedents.py`: `app.cli precedents list | show | overturn | export`: the register a pass reads and cannot edit - `app/cli_proposals.py`: The review's commands (Stream K4): `proposals load`, `proposals withdraw`, `proposals sheet`, - `app/cli_quality.py`: The quality commands: `audit` (no network) and `verify` (re-reads, per host). - `app/cli_refresh.py`: The staging refresh: production's dump replaces staging's data and the client's rows survive. - `app/cli_replay.py`: `decisions export` and `decisions replay` (Stream K2; spec §8), registered under the - `app/config.py`: Application settings. Values come from the environment (see .app.env). - `app/db.py`: Database engine and session handling. - `app/main.py`: Duty Free Professor - application entrypoint. - `app/models/__init__.py`: (no docstring) - `app/models/accounts.py`: Accounts: credentials, levels, members, grants, sessions, tokens, and the audit log. - `app/models/base.py`: (no docstring) - `app/models/catalog.py`: Catalog schema. - `app/models/client.py`: The client's to-do list and the files they hand us. - `app/models/decisions.py`: The decisions ledger and the proposals store (Stream K2; plan W13, W15, W16; the escalation-2 - `app/models/discussion.py`: The discussion tables: decision cards, threads, the comments on every subject, the feature - `app/models/editorial.py`: Editorial storage: articles as rows, and the people who asked to hear from us. - `app/models/hours.py`: Opening hours as observations: one row per reading, collected or entered by hand, nothing deleted. - `app/models/hubs.py`: API response shapes for the hub pages (airports now; brands and categories follow). - `app/models/owner.py`: Rian's running list: the state he sets on items the sessions raise. - `app/models/places.py`: Places and shop places (Stream K2; plan W19). A place is where a shop is, of any kind: an - `app/models/quality.py`: What verification and the audit record: runs, checks, snapshots, refusals. - `app/models/schemas.py`: API response shapes. The frontend's TypeScript client is generated from these. - `app/routers/__init__.py`: (no docstring) - `app/routers/airports.py`: The airport reads beside the detail: `GET /api/airports/{iata}/hours`, the current line and its provenance. - `app/routers/articles.py`: Public reads of editorial text: the article list, one article, and the two keyed blocks. - `app/routers/attachments.py`: Attachments in comments: a picture or a PDF handed over mid-conversation. - `app/routers/auth.py`: The app's own sign-in: `/api/auth/*` (login, logout, forgot, token-state, welcome, reset, - `app/routers/catalog.py`: (no docstring) - `app/routers/collectors.py`: The collector and catalogue review area: /collectors, and its merge session. - `app/routers/discussion.py`: The decision cards, and every comment thread on the site. - `app/routers/health.py`: (no docstring) - `app/routers/image_ask.py`: The image sourcing list: /images, for the client to choose which official images to chase. - `app/routers/items.py`: The running list on /issues: decisions for rian, things he must do, issues to address. - `app/routers/notifications.py`: The one inbox: what the bell polls, and marking rows read. - `app/routers/ops.py`: The owner's operational reads under `/api/ops/*` (R1: active impersonations only). - `app/routers/plan.py`: The build plan as data, for the /plan page. - `app/routers/publish.py`: Index candidates (Stream K6; plan W18): `/api/review/index`, the panel on `/review` where a - `app/routers/quote.py`: The client-facing quote builder. - `app/routers/review.py`: The review (Stream K4): `/api/review`, the sheet a person reads and the approval, reject and - `app/routers/sources.py`: Collection health, and the per-source kill switch. - `app/routers/subscribers.py`: The one public write for email capture: POST /api/subscribers. - `app/routers/todos.py`: The client's to-do page: items, completion, and file hand-in. - `app/routers/trip.py`: (no docstring) - `app/services/access.py`: The route policy: every route in exactly one class, the pure `decide()`, the middleware. - `app/services/accounts.py`: The account system's front door to the vendored BW kit: the mapped store, the permissions, - `app/services/airport_featured.py`: Featured at this airport, by family: the best-value comparables here, one row per family. - `app/services/airport_guides.py`: The written half of an airport page: where the duty free actually is. - `app/services/attention.py`: How much of a person's judgement a suggestion needs, which is not how sure the machine is. - `app/services/attributes.py`: The attribute registry and the one accessor every reader uses (Stream K2; plan W7, W8). - `app/services/audit.py`: Is this data true? The no-network audit: metrics with thresholds, and the human review's lists. - `app/services/audit_log.py`: The audit sink every account mutation calls, and the kit's hooks into it. - `app/services/award_picker.py`: Which medal a product shows when it holds several. - `app/services/awards_import.py`: Attaching competition medals to duty-free product_variants. - `app/services/brands.py`: Splitting a brand row the fold joined wrongly, as one recorded decision with an undo. - `app/services/catalog_queries.py`: Read queries behind the catalog API. - `app/services/collected.py`: The listed layer: what the shop showed, read back from the fragment the collector kept - `app/services/collection_rules.py`: What is cleaning the data right now, read from the code that does it. - `app/services/collector_view.py`: The collectors and catalogue, as the review page reads them. - `app/services/collectors/ari.py`: ARI's "The Loop" — Dublin and Cork airport duty-free. - `app/services/collectors/avolta.py`: Avolta storefronts — World Duty Free and Shop Duty Free. - `app/services/collectors/base.py`: Shared shape for every collector. - `app/services/collectors/changi.py`: iShopChangi, the online duty-free marketplace of Singapore Changi (Changi Airport Group). - `app/services/collectors/dubai.py`: Dubai Duty Free. - `app/services/collectors/extime.py`: Extime, the Paris airports storefront. - `app/services/collectors/fetch.py`: The fetch port. - `app/services/collectors/heinemann.py`: Gebr. Heinemann -- global online catalogue only. - `app/services/collectors/heinemann_platform.py`: Shops running Gebr. Heinemann's storefront platform. - `app/services/collectors/registry.py`: Every collector the app knows about, in one place. - `app/services/collectors/robots.py`: The one robots.txt policy every collector applies. - `app/services/collectors/shilla.py`: The Shilla Duty Free, Seoul Incheon's online store, read through the browser sidecar. - `app/services/collectors/shopify.py`: Duty-free shops running on Shopify. - `app/services/collectors/targets.py`: The targeted beauty set: which lines a collector reads beauty pages for (Decision 4). - `app/services/coverage.py`: What we cover, what we do not, and how good each source's data is. - `app/services/decisions/__init__.py`: The decisions ledger's service package (Stream K2): natural keys, the effective read, the - `app/services/decisions/appliers.py`: The appliers (spec §4 step f, §5): today's code moved behind one registry, each split in - `app/services/decisions/effective.py`: The effective read (spec §3.3): the newest decision per `(entity_type, entity_id, field)` - `app/services/decisions/natural_keys.py`: Natural keys: the portable address of every row a decision can name (spec §2). One grammar, - `app/services/decisions/replay.py`: Export and replay by natural key (spec §8): the review on staging reaches production, and - `app/services/decisions/undo.py`: Undo and batch undo (spec §7): a reversing row, never a delete; consequences first; a - `app/services/decisions/writer.py`: The one writer (spec §4): nothing else inserts a decision. A batch is the unit of approval - `app/services/directory.py`: The local directory: what the vendored kit calls `bw_auth`, plus the token mint and the - `app/services/discussion.py`: Discussions on the Interaction Standard: threads, comments, mentions, the inbox. - `app/services/discussion_import.py`: Load the decisions-and-realities list from a JSON file (idempotent, upsert by title). - `app/services/editorial.py`: Articles: reading a hand-in into a row, and the published reads every page uses. - `app/services/featured.py`: Which savings the home page leads with. - `app/services/feeds.py`: The plain-text and syndication surfaces: the RSS feed and llms.txt. - `app/services/folding.py`: Everything waiting to be joined or kept apart, at every level (Stream K10.3). - `app/services/fx.py`: Currency conversion, so prices from different airports are comparable. - `app/services/guidelines.py`: The guidelines a review pass follows, read from `docs/AI-REVIEW-GUIDELINES.md`. - `app/services/hours/__init__.py`: Opening hours collectors: airport operator sites, one module per platform (docs/COLLECTORS.md, Opening hours). - `app/services/hours/base.py`: The shared shape of an opening hours collector, the one `collect()` they all run through, and the line. - `app/services/hours/dublin.py`: Dublin Airport's own site (www.dublinairport.com): the shops in the terminals page. - `app/services/hours/heathrow.py`: Heathrow Airport's own site (www.heathrow.com): the World Duty Free page under Shops A-Z. - `app/services/hours/registry.py`: Every opening hours collector the app knows about, in one place, and which airport each covers. - `app/services/hours/store.py`: The airport_hours rows: writing a reading, and which row an airport shows. - `app/services/hours/toronto.py`: Toronto Pearson's own site (www.torontopearson.com): the duty free store pages. - `app/services/identity.py`: Who the caller is: the two request-scoped resolvers the whole app runs on. - `app/services/image_ask.py`: The image sourcing list behind /images: which images would cover the most comparisons. - `app/services/images.py`: Product imagery, from an openly licensed source. - `app/services/indexnow.py`: IndexNow: tell the search engines which pages changed, instead of waiting. - `app/services/ingest.py`: Turning collected listings into catalog rows. - `app/services/items.py`: The running list: merge the items file with rian's recorded state. - `app/services/keying.py`: The keying call: what ingest and rederive ask before a product is keyed (Stream M). - `app/services/listings_table.py`: The giant Listings table on /collectors (Stream L, LT6): every listing with its listed, - `app/services/mail.py`: The mail seam: a `Mailer` sends one plain-text message; `none` is the launch provider. - `app/services/mail_resend.py`: The Resend provider behind `MAIL_PROVIDER=resend`: one HTTPS POST per message, stdlib only. - `app/services/markdown.py`: Markdown to HTML for editorial text, one renderer for the API and the server-rendered page. - `app/services/merge_desk.py`: The merge desk (Stream L, LT7): the brands table and the lines table a person confirms - `app/services/merge_session.py`: The merge session: the queue a person works through, and what confirming or rejecting a - `app/services/merges.py`: Recorded merges: two product rows that are one bottle become one, with a record; and - `app/services/normalize.py`: Turning messy retailer strings into comparable facts. - `app/services/overrides.py`: The decided layer's shim (Stream K2): the public names every older reader calls, delegating - `app/services/pass_packet.py`: What a review pass is given, and how the review knows it is done (Stream K12.5). - `app/services/passwords.py`: Passwords: argon2id hashing and verification, the rules, the generated-password alphabet. - `app/services/places.py`: Places: where a shop is, of any kind, and the comparison unit defined once (Stream K2; plan - `app/services/precedents.py`: The precedent register: every kind of judgement a person has answered, and what was answered. - `app/services/process_doc.py`: The review process document, read at runtime so the page cannot drift from it. - `app/services/product_lines.py`: The line above the product, the attribute vocabulary, and the form: the pure rules. - `app/services/proposal_rules.py`: The word lists as proposal generators: what each list WOULD have grouped or read, written as - `app/services/proposals.py`: The review (Stream K4; plan W10, W13, W15, W16; the escalation-2 spec §6 and §7): a proposals - `app/services/proposals_store.py`: The proposals store's thin writer (Stream K2; spec §1.3, §1.4, §6.2's shape): a pass by name, - `app/services/publish.py`: Publish (Stream K6; plan W18): the one reader of a page's `indexed` and `hidden`, the - `app/services/quantity.py`: The quantity a product comes in, read from text: identity rules v4 (Stream L, 15 Sep). - `app/services/review_detail.py`: One suggestion, opened up: everything a person needs to answer it without leaving the page. - `app/services/robots_policy.py`: Our own robots.txt: what we tell other readers, by name. - `app/services/seo.py`: Server-rendered SEO layer. - `app/services/sessions.py`: Sessions: the store, the `SessionRow` mapping the kit reads, the cookie, the loader middleware. - `app/services/subscribers.py`: Email capture: taking a subscription and exporting the list, with the personal-data rules. - `app/services/suggest.py`: Merge suggestions at every level, rules first: what the queue offers a person (Stream M). - `app/services/taxonomy.py`: Consumer-facing product categories. - `app/services/throttle.py`: A small in-process rate limiter for the few public routes that write. - `app/services/trip.py`: Savings across the shopper's airports. - `app/services/uploads.py`: Validation and storage for files the client hands us. - `app/services/urls.py`: The site's URL shapes, in one place. - `app/services/verify.py`: Re-read a sample of published listings and say what moved, what broke, what refused. - `app/services/view_as.py`: View As: DFP's policy over the vendored `bw_view_as` mechanism, and its reporting. - `app/vendor/__init__.py`: The BW application kit, vendored byte-identical (plan: accounts-2026-09 §4.1). - `app/vendor/bw_accounts.py`: bw_accounts.py — the standard BW-Auth app admin framework (P4, external-users plan). - `app/vendor/bw_admin_api.py`: bw_admin_api.py — the standard BW-Auth admin API as a mountable FastAPI router. - `app/vendor/bw_store_sqlalchemy.py`: bw_store_sqlalchemy.py — a ready SQLAlchemy store for the BW-Auth accounts kit. - `app/vendor/bw_view_as.py`: bw_view_as.py — standard "View As" (impersonation) for BW-Auth (Pattern B) apps. - `app/version.py`: Single source of truth for the app version (surfaced by /api/health and the UI).