# SEO and AEO

Sources of truth: `main/app/services/seo.py`, `main/app/services/urls.py`, `main/app/main.py`,
`main/app/models/hubs.py`, `main/app/services/robots_policy.py`, `main/app/services/feeds.py`,
`main/app/services/indexnow.py`, `main/app/cli_pages.py`, `main/web/src/lib/urls.ts`,
`main/web/src/lib/structure.ts`.

The machine-readable layer (title, description, canonical, social tags, JSON-LD) is stamped
server-side into the SPA shell for every page. Product pages also get a server-rendered body
that matches the SPA's default state exactly; anything else is cloaking (`seo.py` docstring).
Client surfaces are noindexed, and their writes are the only ones open past the shared
password: every other write route is closed by the default-deny middleware
(`CLIENT-SURFACES.md`, `app/services/mutations.py`). Page-type decisions and their evidence
are on `/structure`; the plan is build plan §6.

## Which shops the site shows
Every catalogue query, `/api/stats`, the sitemap and the IndexNow list scope their shops through
`catalog_queries.publishable(db)`: visible locations (Decision 1) minus any shop whose source
verify has blocked (Decision 10, `catalog_queries.blocked_location_ids`, fed by
`verify.blocked_sources`). A blocked shop is hidden everywhere at once until a human clears
the check (`app.cli verify-clear`): its prices leave every card and price table, its airport
page is a 404 like a hidden airport's, its products leave the sitemap. Nothing else hides a
row; a refusal or a moved price keeps its date. The `/trip` comparison and the coverage page
apply the same rule (`tests/test_trip_publication_block.py`; the aliased pair query takes
`blocked_location_ids` by hand). Tests: `tests/test_publication_block.py`; rehearsal 2026-09-05 on a
scratch copy: one MISMATCH_SIZE on a JFK listing removed JFK from stats, browse, a product's
price table, the airports index and page, the dataset facts and the sitemap; clearing it
restored every figure exactly.

## URL scheme and redirects
- Every URL shape lives in `app/services/urls.py`, mirrored by `lib/urls.ts`; the two must
  agree or server canonicals fight client links (`tests/test_seo.py::TestSlugs`,
  `tests/test_seo_airport.py::TestAirportUrls`). `seo.py` re-exports the helpers.
- Product: `/products/<slug>-<id>`. The trailing number is the identity; the words are for
  humans.
- Category at airport: `/airports/<airport>/<category-slug>` (`airport_category_path`): the
  airport's own address, then the category's word from the declared table `CATEGORY_SLUGS` in
  `urls.py`, mirrored exactly by `lib/urls.ts` (`tests/test_airport_category.py` pins the two
  equal). A page exists only where the pairing is at or over `coverage.CATEGORY_AT_AIRPORT_MIN_PRODUCTS`
  published products, read at request time; under it the address is a real 404 and absent from
  the sitemap. The airport part canonicalises like the airport page (a stale slug 301s, keeping
  the query); the category word must be exact. The shopping-feature flags, the sort and the page
  are views for people; every one names the clean pair address as its canonical.
- The airport page's shopping-feature chips (`?multi_only|awarded_only|exclusives_only=true`, the
  names `/api/products` takes) and the comparison preset (`/savings?from=<IATA>`) are views too:
  the clean airport address stays the canonical of every filtered view.
- A bare id (`/products/60`) or a stale slug 301s to the current canonical path.
- Airport: `/airports/<common-name>-<iata>-<city>` (`airport_path`), the review's shape:
  the airport's common name leads because it is what people search ("heathrow", not "lhr"),
  then the code, then the city unless the name already says it (`/airports/heathrow-lhr-london`,
  `/airports/hong-kong-hkg`, `/airports/paris-cdg` for CDG and Orly together). JFK is the one
  airport known by its code (`/airports/jfk-new-york`); any other exception is Adam's to name.
  The name is declared data: one table keyed by IATA, `AIRPORT_SLUGS` in `urls.py`, mirrored
  exactly by `lib/urls.ts` (`tests/test_seo_airport.py::TestAirportSlugTable` reads the TS
  file and pins the two equal). The IATA code stays the identity; the words never come from
  a request. An airport with no entry keeps the earlier `<iata>-<city>` shape from its
  `locations` row so nothing 404s. `parse_airport_slug` resolves, in order: a declared slug
  exactly; a declared code leading the slug (the bare code, the earlier `lhr-london`); a
  declared code in any later segment (a stale new-shape slug); an undeclared code leading
  it. A three-letter word is never read as a code unless it leads or is declared
  (`san-salvador-sal` is SAL, never San Diego). Every shape that resolves and is not the
  canonical path 301s to it, keeping any query string; an airport we do not show (hidden,
  catalogue-only, no stock) is a 404. The API sends every airport link ready-made
  (`AirportSummary.path`, `PriceOut.location_path`, `CoverageLocation.path`), so the SPA
  rebuilds no address; the TS table serves `airportIataFromSlug` on the airport page. Filtered
  and paged views (`?category=`, `?sort=`, `?page=`) render server-side too but canonicalise
  to the clean page: they are for people. `page` and `sort` are read leniently (garbage means
  page one, featured), never a 422 where a page should be.
- Brand: `/brands/<slug>` (`brand_page_path`), the slug being `brands.slug` from migration #3,
  never a string folded from a name. An alias row (`canonical_id` set) and a differently cased
  slug 301 to the house's path, keeping the query; an unknown slug, or a brand under the floor,
  is a 404 and has no sitemap entry. The floor (review, 9 Sep) is `BRAND_PAGE_MIN_PRODUCTS`
  (three) published products priced, between them, at `BRAND_PAGE_MIN_AIRPORTS` (two) distinct
  airports by IATA; read that way because the page is a comparison, and stated on the running
  list for Adam to correct in one line. `catalog_queries._house_counts` is the one definition,
  shared by the index, the page, the sitemap and the product page's brand link. Filters and paging as on airport
  pages. `brand_path(name, slug)` / `brandPath(name, slug)` link to the page when the API handed
  over the slug and fall back to a catalogue search otherwise. `ProductSummary.brand_slug`
  (`catalog_queries.brand_page_slugs`) is that slug, and only for a house `list_brands` would
  list, so the product page's brand name (body and SPA) reaches the page and can never 404.
- Article: `/articles/<slug>` (`editorial.article_path`, the row's slug verbatim; Stream D
  owns the slugging) with `/articles` the index. A draft or unknown slug is a real 404 exactly
  like a slug that never existed, so nothing about unpublished work is inferable from outside.
- Category links come from `category_path` / `categoryPath` (a catalogue filter until B6).
- Every merged product 301s to its survivor through `catalog_queries.resolve_product_id()`,
  which the page route and both product API routes already call (the API keeps the query
  string), following `products.merged_into_id` (migration #3).

## What renders server-side, and how to verify it
- Every route: head only (title, description, canonical, OG, JSON-LD, noindex where due).
  Titles and every served string follow the house style, no em dash among them
  (`DEFAULT_TITLE`, `tests/test_house_style.py`, `tests/test_seo_body.py`).
- `/products/<slug>-<id>`: the whole body. `product_body()` mirrors `ProductPage.tsx` with
  no airports chosen: header nav, hero, meta line, medals, verdict, price table. Each helper
  names the component it mirrors in a comment block in `seo.py`; change one side, carry the
  other. The detail object is seeded as `window.__DFP_PRODUCT__` so React's first commit is
  the finished page (no skeleton, no product request). The seed is the no-airport medal
  order; a shopper with saved airports gets one request with `at=` for their order
  (`useProduct`), the seed standing in as placeholder meanwhile, so the served body is
  never swapped for the skeleton.
- Omitted on purpose: the "On your trip" block (the shopper's airports live in their
  browser), the related-products rail (a second request after mount), the footer.
- The header (`header_html`) is `SiteHeader.tsx` closed: announcement bar, brand, the no-JS
  search form (`action=/products`, the one deliberate difference from the SPA's; the input is
  SiteSearch's combobox, closed, with its empty hidden suggestion list), the airports
  button and the profile circle drawn signed out (the SPA swaps in the initials at the same
  size), the nav with its four departments (each a `site-nav__group`: the department's link
  and its panel toggle), the rest of the row after a divider and My savings at its end, and
  the panel's empty hidden shell (`#site-mega-menu`), exactly what the SPA renders before a
  panel opens; its links are never pre-filled server-side, because a visitor does not get
  them until they open it. The active item follows the SPA's: Airports on an airport page,
  Articles on an article, none on a product page. A header change in the SPA is a mirror change here the same
  day (`tests/test_seo_body.py::TestHeaderMirror`).
- `/airports/<common-name>-<iata>-<city>`: the whole body. `airport_body()` mirrors `AirportPage.tsx` with
  no airports chosen: breadcrumb, hero with the lede sentence, the facts panel (summary, the
  standing facts, the four counts), the terminal rows where a guide is written, the write-up,
  then the one shelf of products the URL asks for — the savings (the default), the travel
  exclusives, or the full list with its category chips, sort control and pager
  (`product_card_html()` mirrors `ProductCard.tsx`). `?tab=` names the shelf and a filter, a
  sort or a page implies the full list (`urls.airport_tab`, mirrored in `lib/urls.ts`); every
  shelf carries the clean page as its canonical, so they never compete with it. Every figure comes from `catalog_queries.airport_detail()`, the same object
  `GET /api/airports/{iata}` returns and the SPA is seeded with (`window.__DFP_AIRPORT__`,
  used only when the seeded view matches the URL's filters). Verified by diffing the served
  root against headless Chrome's DOM of the same URL: identical apart from the header's
  no-JS search form attributes and react-router's `active` class.
- Verify without JavaScript: `curl -s https://<host>/products/<slug>-<id> | grep '<h1'` must
  show the name, and `price-table__usd` rows must carry prices. The pure renderer is pinned by
  `tests/test_seo_body.py`; the routes by `tests/test_site_routes.py` (no database).
- Performance is measured, not assumed: Lighthouse mobile, local, gzip on both sides.
  2026-09-04: CLS 0.003 to 0, LCP 3.2s to 3.0s, performance 93 both. 2026-09-10, home page
  after the airport picker drew loading placeholders at the rows' final height: CLS 0.017 to
  0.001, LCP 2.4s, performance 97. Re-measure after any change to the header mirror or the
  hero; a header of a different height shifts the hero on mount. A local server caches the
  shell at startup, so restart it after a rebuild or Lighthouse reports no paint at all.

## Structured data per page type
- Home: `Organization` (`@id /#organization`, name, url, logo; `sameAs` once the client
  confirms handles) and `WebSite` (`@id /#website`, publisher). No `SearchAction`: Google
  retired the sitelinks search box in November 2024.
- Product: `Product` with `@id`, `url`, `description` (the same factual sentence as the meta
  description), `brand`, the barcode as `gtin8/12/13/14` by width, `image`, `category`,
  `award`, and `offers` as an `AggregateOffer` (`offerCount`, `url`) holding one `Offer` per
  shop in the shop's own currency with `seller`, `availableAtOrFrom` and the shop's URL. The
  price of record (review, 9 Sep): the shop's own price in its own currency is the price on
  the page and the only price in the markup; the combined offer states `priceCurrency`,
  `lowPrice` and `highPrice` only when every shop quoted one currency, from those quotes, and
  carries no range otherwise. Our USD conversion never enters the markup: on the page it sits
  second, labelled as our conversion at the rate held on the date checked. Airport and brand
  pages carry no offer markup at all (`tests/test_seo_body.py::TestOfferMarkup`). `availability` appears only when a stock flag was
  observed: any InStock wins, all OutOfStock says so, unknown is omitted, never defaulted.
  `aggregateRating`/`review` are deliberately absent: there are no reviews.
- Product: `BreadcrumbList` (Products, category, name) with absolute `item` URLs.
- Airport: `CollectionPage` (`@id <path>#page`, `url`, `name`, `description`, `isPartOf` the
  WebSite) whose `about` is an `Airport` (`name`, `iataCode`, and a `PostalAddress` holding
  only the locality and country the `locations` row has: no street, no coordinates, because
  we do not hold them) and whose `mainEntity` is an `ItemList` (`numberOfItems` = the
  airport's product total; one `ListItem` with `position`, `url`, `name` per card on the
  served page, positions continuing across pages). Plus `BreadcrumbList` (Airports we price,
  the airport).
- Brand and category hub pages (B5, B6): `ItemList` + `CollectionPage` the same way, with
  `Brand` on brand pages.
- Brand: `CollectionPage` (`@id <path>#page`, `isPartOf` the WebSite) whose `about` is a `Brand`
  (`@id <path>#brand`, `name`, `url`) and whose `mainEntity` is an `ItemList` as on airport
  pages; `BreadcrumbList` (Products, the brand). Body: `brand_body()` mirrors `BrandPage.tsx`:
  breadcrumb, hero with the lede (products, airports, comparable count and the airport most
  often cheapest, all counted), four counts, "Where to buy" (one row per airport that stocks
  the brand with its product count, how many comparable bottles it is cheapest for, checked
  date), category chips only for a house with several categories, then the shared list section
  (`_list_section`, also the airport page's). Data: `catalog_queries.brand_detail()`
  (`GET /api/brands/{slug}`, seeded as `window.__DFP_BRAND__`); `list_brands()` feeds the
  sitemap and `GET /api/brands`.
- Article: `Article` (`@id <path>#article`, `url`, `mainEntityOfPage` the WebPage, `headline`,
  `description`, `datePublished`, `dateModified` = the last edit, `articleSection` = the
  category when set, `image` = the hero when set, `publisher` the Organization, `isPartOf` the
  WebSite; no `author` until the rows carry a name) plus `BreadcrumbList` (Articles, the
  piece); `og:type` is `article` there and `website` everywhere else.
- `/data`: `Dataset` (`@id /data#dataset`, `creator`/`publisher` the Organization,
  `isAccessibleForFree`, `temporalCoverage` first to last observation, `dateModified`,
  `spatialCoverage` one `Airport` per airport page, `variableMeasured` from
  `DATASET_VARIABLES`, `measurementTechnique` from `DATASET_METHOD`, `size`). No `license` and
  no `distribution`: neither is decided (running list), and an invented one is a promise.
  Plus `BreadcrumbList`. The body (`dataset_body`) mirrors `DataPage.tsx`; every figure is
  `catalog_queries.dataset_facts()` (`GET /api/dataset`, seeded as `window.__DFP_DATASET__`).

- Organization (`organization_jsonld`): name, url, logo, and `sameAs` for the social profiles
  that exist. The profiles have one home, `SOCIAL_INSTAGRAM_URL` / `SOCIAL_YOUTUBE_URL` in
  `.app.env` (`settings.social_profiles`: https only, placeholders ignored): the shell's
  `window.__DFP_SOCIAL__` (the SPA's header and footer links), the header mirror and `sameAs`
  all read it, so nothing can show a profile the markup does not claim, or the reverse.

Every URL in JSON-LD is written site-relative and made absolute at apply time
(`absolutise`), so heads can be module constants and still carry the configured origin.

## Origin, caching, conditional requests
- `PUBLIC_BASE_URL` (in `.app.env`) is the origin for canonical, sitemap and JSON-LD URLs.
  Outside production an unset value falls back to the Host header; in production it does not,
  and the startup log warns. Set it before the first production deploy.
- HTML and JSON are `Cache-Control: no-store` so a CDN never serves an owner's edit late.
  `/assets` and `/fonts` are immutable for a year; `/medals` and `/flags` for 30 days.
- HTML, `sitemap.xml` and `robots.txt` carry a weak `ETag` and `Last-Modified` (a product's is
  its newest observation; the sitemap's is its newest lastmod) and answer `If-None-Match` /
  `If-Modified-Since` with 304. HEAD is supported on all of them.
- `/articles/<slug>`: the whole body. `article_body()` mirrors `ArticlePage.tsx`: eyebrow,
  title, standfirst, the published/updated line (`updated_after_published`: a later UTC day,
  as `formatDate` counts it), hero figure, the rendered Markdown in `<div class="prose">` (one
  renderer, `services/markdown.py`, so the API and the body cannot differ), the "All articles"
  link. The piece is seeded as `window.__DFP_ARTICLE__` (`api/editorial.ts` reads it) and
  `ArticlePage` ships in the main bundle so React's first render redraws the body rather than
  a Suspense blank. Omitted on purpose: the subscribe form (a JavaScript form whose ids React
  mints at mount) and the footer. `/articles` itself is a static head over the SPA's list.
- An airport's published write-up rides in `AirportDetail.writeup` (`editorial.airport_writeup`),
  so `airport_body()` and `AirportPage.tsx` both draw it from the same object, with
  `EditorialBlock.tsx`'s classes (`editorial_block_html()`), after the shops line and before
  the savings cards; no second request, no shift. Category intros wait for B6.
- The written guide to an airport's duty free (where the shops are by terminal, the specialty
  boutiques, a service, a map link) rides in `AirportDetail.guide` from
  `services/airport_guides.py`, in code because the data holds no terminals;
  `airport_facts_html()` and `airport_terminals_html()` mirror `AirportGuide.tsx`'s two
  components class for class. The demo's placeholder view, which marks the parts
  of the template nobody has filled (`web/src/lib/airportTemplate.ts`), is a browser setting and
  never reaches the server body: a crawler sees the page without the part, never a note about it
  (`tests/test_seo_airport.py::TestAirportGuide`).
- Unknown product ids and unknown routes are real 404s (the shell with a noindex head, so the
  SPA draws its not-found page). `is_known_route` mirrors `App.tsx`;
  `tests/test_seo.py::TestRouteInventory` reads `App.tsx` and fails on drift.

## The crawl surface (robots, IndexNow, feed, llms.txt)
- `/robots.txt` is `robots_policy.robots_txt()` once `SITE_ACCESS=public`; while the site is
  members-only (`ACCOUNTS.md`) it answers `User-agent: *` / `Disallow: /` with no Sitemap line,
  the machine files 404, and every response carries `X-Robots-Tag: noindex, nofollow`, so nothing
  under a product URL is indexed before launch. Public: a `*` group with the three Content Signals
  (`search=yes, ai-input=yes, ai-train=no`), `Allow: /`, `Disallow: /api/`; then one group per
  named bot whose crawl serves a signal (`NAMED_BOTS`), allowed iff any of its signals is on.
  The stance is the one dict `CONTENT_SIGNALS`; flipping `ai-train` flips GPTBot, ClaudeBot,
  CCBot, Bytespider, Applebot-Extended, cohere-ai with it. Google-Extended and
  meta-externalagent also govern grounding, so they list `ai-input` too and stay open. The
  file is written to the same standard the collectors' `robots.py` reads by (a bot addressed
  by name gets its own group); `tests/test_crawl_surface.py` parses our file with our matcher.
  Zone rule (Stream E's checklist): Cloudflare's managed robots.txt and AI-bot blocking stay
  off, or the edge overrides this file.
- IndexNow (`indexnow.py`, `cli_pages.py`): `python -m app.cli_pages indexnow --since <ISO>`
  (or `--hours N`, `--dry-run`) inside the container after a collection. Changed URLs come
  from the database, not from a list a collector keeps: products with an observation after
  `since`, the airport pages whose shops gained one, and the sitemap. One POST per 10,000 URLs
  to `api.indexnow.org`; the key is `INDEXNOW_KEY` in `.app.env` (any alphanumeric string we
  mint, not a secret) and is served at `/<key>.txt` only when set. Without `PUBLIC_BASE_URL`
  or the key the command lists and exits 2. The subcommand is registered in `app.cli`
  (`register_pages(sub)`), so `python -m app.cli indexnow` is the same command.
- `/feed.xml` (`feeds.rss_xml`): RSS 2.0, one channel newest first: the fifty products most
  recently added to the catalogue at a shop the site shows (`catalog_queries.newest_products`,
  `pubDate` = the product's `created_at`) and the twenty newest published articles
  (`editorial.feed_items`, `pubDate` = `published_at`, the description never the body), absolute
  links, `lastBuildDate` and `Last-Modified` = the newest item. Every head carries
  `<link rel="alternate" type="application/rss+xml">`.
- `/llms.txt` (`feeds.llms_txt`): markdown from `dataset_facts()`: what the site is, how to read
  a price, the live counts, every airport page with its count and checked date, the
  categories, the machine surfaces and the reader's identity page. `Last-Modified` = the
  newest observation. No number in it is typed (`TestLlmsTxt::test_no_typed_numbers`).
- All of these answer HEAD and conditional GET like the pages (`conditional()` in `main.py`).

## The structure proposal page
`/structure` (`lib/structure.ts`, `components/SiteMap.tsx`) is the page Mark comments on. Its
figures are measured at the sixteen airports collected today and say so; the launch set is
nineteen. The published-pages count and switched-on airport count are read live from
`/api/stats`, never typed. House style there: no em dashes, never "cheap" or "free".
Page rows carry one of three states (`PageStatus`): `live`, `built` (the code has landed and
waits for rian's deploy; drawn like a live node, tagged BUILT, "Built, waiting for the next deploy"
in the table) and `build`. Flip `built` to `live` at the deploy checkpoint, never before,
because Mark reads the page as fact.

## Sitemap
- In: `/`, `/products`, `/savings` (when the my-airports feature is on), `/exclusives`,
  `/awards`, `/airports`, `/articles`, `/data`, every airport page (`catalog_queries.list_airports`),
  every brand page (`list_brands`, houses that meet the floor), each with
  `lastmod` = its newest observation, every published article (`editorial.sitemap_rows`,
  `lastmod` = its last edit), and every product with at least one observation at a shop the
  site shows (`publishable`), with `lastmod` = its newest observation.
- Out: noindexed client surfaces (`/discuss` and its tabs, `/settings`, `/plan`, `/issues`,
  `/sources`, `/collectors`; the moved `/todo`, `/structure` and `/quote` answer a 301 before any
  head, `CLIENT-SURFACES.md`), `/trip`, `/feature/*`, hidden or blocked locations' products, thin
  brands, filtered or paged hub views. Category pages join when they ship (B6).

## Build plan §6 corrections
| Correction | Status |
|---|---|
| Server-rendered product body | done 2026-09-04 (B1) |
| `PUBLIC_BASE_URL`; absolute `url`, `offers.url`, breadcrumb `item`, `@id` | done (B2); value to be set in `.app.env` |
| gtin by width | done (B2) |
| availability omitted when unknown | done (B2) |
| per-shop `Offer` with `priceCurrency` | done (B2) |
| `Organization`; `WebSite` without `SearchAction` | done (B2); `sameAs` pending handles |
| real 404s, HEAD, ETag/Last-Modified, `no-store` | done (B2) |
| favicon and manifest | done (C, 11 Sep): `favicon.ico`, PNG icons, `site.webmanifest` in `web/public`, pinned by `test_tokens_contrast.py` |
| IndexNow, per-bot robots with Content-Signal, RSS, `llms.txt`, Dataset page | done (B7); `INDEXNOW_KEY` to be set in `.app.env` |
| airport pages with ItemList/CollectionPage/Airport | done (B4) |
| brand pages with ItemList/CollectionPage/Brand on the brands table | done (B5); product pages link the brand name to its page through `brand_slug` |
| category and category-at-airport pages | B6, after rian's page-types answer |
| merged products 301 | done: `resolve_product_id()` follows `merged_into_id` (A, migration #3) |

The owner's image sourcing list at `/images` is `noindex`, like every owner surface (`app/main.py` `HEADS`).