# Collectors

Sources of truth: `main/app/services/collectors/`, `main/app/cli.py`, `main/app/services/ingest.py`.

## How collection works
One shared pipeline: `fetch.py → collector.collect() → RawListing → ingest.run_collector →
normalize / taxonomy / fx → ProductVariant / Listing / PriceObservation`. **One collector per
commerce platform, airports as configuration rows**, plus bespoke modules for platforms seen
once. The registry (`collectors/registry.py`) is the list; `python -m app.cli status` is the
live source table. **No step uses an LLM or any AI.** Adding an airport is one configuration
line on an existing platform; a new JSON-API platform is a small module; server-rendered HTML
needs a bespoke parser; a client-rendered site needs the rendered fetch (Stream A2's section
below). Listing pages are harvested, not product pages; product pages are read one at a time
only to verify (`read_one`).

A run is steered between requests, never inside one (`collectors/control.py`). The fetch port
calls the control hook where it used to sleep; bound by `run_collector`, the hook commits the
open batch, heartbeats the run row (`heartbeat_at`, `requests_made`) and re-reads the source's
`control` (run, pause, stop), its pace and the kill switch every two seconds while it sleeps or
is paused, so a page action lands within two seconds plus the request in flight and a paused
run is never silent. The pace it sleeps is the slowest of what was asked, what the host's
robots.txt publishes (recorded on the source as `robots_crawl_delay`, a fact, not a control) and
the source's own `delay_seconds`. A stop is `RunStopped`, a `BaseException` no collector's broad
`except` can swallow; SIGTERM only raises a flag, honoured at the next poll, so a run the guard
sheds ends `stopped` with its batch committed instead of sitting `running` for a day. Two
processes on one source never both run: the child takes an advisory lock on its own connection
before it writes a run row. `recheck` mode walks the held listings through `read_one` under the
same hook, counting each as checked, changed or missing. The page that presses all this is
`/collectors#live` (`RUNBOOK.md`, Collecting).

A staging refresh (`RUNBOOK.md`, Staging refresh) replaces every collection run, listing and
observation on staging with production's and keeps only what people wrote on the site; never
run one while a collection is in progress, and never point it at production.

## Robots policy
Applied at the top of every `collect()`, `read_one()` and `render()` by `collectors/robots.py`,
on every run, because permission can be withdrawn between runs and once was:
- The group is matched on the bot's own name (`DutyFreeProfessorBot`, case-insensitive
  substring), then `*`; the crawl-delay comes from the same group and the run uses the slower
  of theirs and ours.
- Modern wildcard matching; **any matching Disallow is a no**, even when a longer Allow would
  technically win.
- robots.txt itself: 404 or 410 → unrestricted, logged; **401 or 403 → a refusal**, recorded,
  run ends; 5xx or timeout → stop this run and retry next run (RFC 9309 §2.3.1; build plan
  §10 #13 ratifies).
- A block on any page (challenge, WAF, empty body behind a 2xx) is a refusal too: recorded,
  never worked around (`fetch()` raises `SourceBlocked`).
- **A page that is not there is not a refusal.** 404 and 410 raise `PageGone` (a `FetchError`,
  never a `SourceBlocked`): a discovery walk logs the URL and reads the next page, a recheck
  counts one `existing_missing`, a verification check is `GONE`, and only a walk that could
  skip nothing ends the run `error`. The same status at robots.txt still means "no rules
  published" (above). A refusal is final for the whole source, so a delisted bottle must
  never be able to produce one.
- Identity mode is honest by default. The rule (build plan section 4 item 11): a source
  switches to a browser-like identity only with a `permission_record` naming who agreed, when
  and how. Today `sources.identity_mode` and `permission_record` exist as columns and nothing
  reads them: every collector sends the declared identity and no code path can switch, so the
  rule holds by absence; the refusal the rule describes is an open issue for Stream A
  (`issue-identity-mode-is-a-column-nothing-enforces`), to land before any switch is possible.

## Per-platform recipes

Recipes exist below for the two rendered sources; the six text-fetch platforms (Avolta,
Shopify, ARI, Extime, Dubai, Heinemann and its platform shops) are documented by their
module docstrings and the posture table until their recipes are written
(`issue-per-platform-recipes-for-the-six-text-fetch-collectors`, Docs). Operational
companions: the per-source kill switch is `sources.enabled` (`POST /api/sources/{slug}/enabled`,
owner only; `python -m app.cli sources`), the no-network audit and the re-read check are
`python -m app.cli audit` and `verify` (`QUALITY.md`), and the skip reasons a run counts are
the strings passed to `report_skip()` in each collector (`no_price`, `no_offer`,
`not_a_product_page`, `reread_failed`, `variant_page_unreadable`, `not_offered`,
`multi_variant_unpriced`).
Stream A fills this: one subsection per collector module with the discovery method, the
pagination rule, where the price and currency are read from, and the known fragilities.

### iShopChangi, Singapore (`changi.py`, rendered)
Discovery: `/en/sitemap-products.xml` (about thirty thousand product URLs; the trailing
`mp` code is the identity), filtered to `import/changi-targets.json` unless the collector is
built with `full_catalogue=True` (rian's crawl token). Per URL: one render with
`assets.adobedtm.com` declared as the only asset host (the storefront bundle crashes without
its tag manager; beacon hosts stay aborted), waiting for a price element; the page's own
`pdp/inventory.json` response is read, never requested directly. Price: the `DEPARTURE`
channel of the variant's selected offer, accepted only when its string form reads `S$`
(`price_type` "departure"; Arrival is the same figure, non-traveller delivery is about a
fifth higher and is never published). Size from the name, else the bare sale-measure
qualifier; ABV from the `alcohol_percentage` classification; stock from the offer; category
from the breadcrumb. **No barcode anywhere**, so SIN rows match other shops on brand, name
and size only. The seller (a marketplace offer names one) is kept in the raw record.
Fragilities: the tag-manager dependency; three channels; a variant per size under one
product code. Tests: `tests/test_collectors_changi.py` on the real Johnnie Walker Blue Label
record.

### The Shilla Duty Free, Seoul (`shilla.py`, rendered)
Discovery: the home page (the one page that serves to a text client) and each product
page's related-product rail, walked breadth-first under the run's render budget (200) from
the liquor, skin care, makeup and fragrance trees; the sitemap is empty and the category
grids are drawn by scripts robots.txt disallows, so they are unreadable to a compliant bot
and their loader is not read either. Per URL: one render at `domcontentloaded` with every
asset request refused (robots); the document is complete without them. Price: the hidden
`prdPriceDollar` field, equal to the visible "Discount price" (`price_type` "discount"),
the "Price" tier as the crossed-out figure, the online-member tier kept in raw only.
Barcode from "REF.NO" (a UPC); the neighbouring "SKU.NO" is not one. Stock: the sold-out
block is in every page's markup and means sold out only when it is shown. Size from the
"Weight,Volume" specification row (a weight yields no size). Category from the page's
three hidden depth fields. Fragilities: the Cloudflare judgement (a challenge to the
browser one day is a refusal, recorded, never solved); robots.txt now names many AI and
rendering crawlers by name, which is a signal about intent worth raising in the
partnership ask. Tests: `tests/test_collectors_shilla.py` on the real NARS record.

## Rendered fetch
`fetch.render(url)` is the second door for sources that draw their prices with JavaScript
(a text fetch of such a page sees a shell and no price). It keeps the `fetch()` contract:
the same honest identity, robots re-read per run through `robots.py` (or the run's own
`Robots` passed in), the host's crawl delay with a **ten-second floor** where none is
published, validation on content never status (`cf-mitigated: challenge`, a challenge
title, a shell that drew fewer than forty characters → `SourceBlocked`), and a per-run
render budget (`render_budget(cap)`; the count is logged at exit; the default cap outside
a declared budget is small on purpose). A render is roughly fifty times a text fetch: it is
a per-URL tool for the comparisons that matter, never a mode for a whole source.

**Pace per source.** `render_wait` takes the slowest of the host's Crawl-delay, the run's
delay and the source's own floor: a rendering collector declares `render_floor_seconds`
(None keeps the general floor) when a host has shown it wants a slower pace, so the pace is
the source's declaration rather than a flag someone remembers; one run is slowed with
`collect --delay N` (or the source's `delay_seconds`), and every `rendered` log line carries
the pace used as `wait=`, so a change of pace and what followed it read together in the run
log. The wait is measured from the start of the previous render, so a page that takes nine
seconds to draw under a ten-second wait leaves the host about one second idle.

**The sidecar** (`main/browser/`, service `browser` in compose) is Playwright's bundled
Chromium behind a small HTTP server (`GET /health`, `POST /render`). Isolation is
structural: its own compose network with only the app on it (the database is unreachable),
no `env_file`, no published port, the image's non-root user, `cap_drop: ALL`,
`no-new-privileges`, and Chromium's own renderer sandbox ON, made possible by
`browser/seccomp.json` (Docker's default profile plus the user-namespace and `chroot`
calls the sandbox needs; measured 2026-09-05, `chroot` was the one Docker's default blocks).
Every render gets a fresh context (no cookies, no storage) and an **egress allowlist**:
`route('**/*')` aborts any request to an IP literal, loopback, private or link-local
address, or a host outside the source's origin plus its named asset hosts; aborted hosts
are reported back and logged once per render. It also captures the JSON bodies of the API
calls the page makes for itself (`Rendered.api_responses`), so a collector can read what
the page fetched without calling the API itself.

**What it never does:** click, type, scroll, or dismiss a cookie, age, terms or
shopper-mode dialog. If a price appears only after one, that is a STOP and an item on the
running list (`items.py add --kind issue`). And a challenge page is a refusal like any other: recorded, the source disabled,
never solved by other means.

`render()` also takes `wait_for` (a CSS selector the caller expects once the page has drawn)
and `settle_ms`; the sidecar reports the page's console errors and uncaught exceptions
(`Rendered.console`), which is how a shell that drew nothing explains itself. A refusal on
content raises `RenderRefused` (a `SourceBlocked`) carrying the `Rendered` it judged, so a
probe can still read the headers and captured API bodies. The host's own Disallow rules are
sent along as regexes and bind the page's subresource requests too (`aborted_paths`).

Probe a new source with `scripts/render-probe.py <url>` (announce it in `.logs/runs/`
first). Tests: `tests/test_render.py`. Local run of the sidecar for a probe:
`docker run` the image with the same flags compose declares and a port on `127.0.0.1`.

## The quantity a collector states (identity rules v4, Stream L)

`RawListing.quantity_ml` stays the derived millilitre figure (None for a weight or a count).
Two fields beside it: `quantity` is the collector's OWN structured quantity as a
`services/quantity.Quantity` in a canonical unit (ml | g | pcs), None when the collector
only has the name; `quantity_text` is the shop's size wording verbatim. Ingest parses the
name with `quantity` as the hint that outranks the text within one dimension
(`collectors/base.quantity_of`; precedence in `services/quantity.py`). Where each is read:

| Collector | Structured quantity | Text |
|---|---|---|
| heinemann, heinemann_platform | `contentUnit.quantity` with `unit.code` through a named code table: MLT, ML, CLT, LTR, L are ml; GRM, G, KGM, KG are g; PCE, PC, ST, EA are pcs; any other code yields state `unparsed` with the code kept, never a silent None (the fixtures show only L; the codes come from the spec) | `"<quantity> <code>"` |
| extime | `capacity` with `capacity_unit`, through the same price-per-unit cross-check `quantity_ml` passes; `net_weight` with `net_weight_unit` ONLY when there is no capacity and neither the variation's `name` nor the product name states a quantity (on the shop a net weight is usually a packaged weight, 100 ml beside 86 g), stamped `quantity_source: extime.net_weight` on the raw record so a wrong gram is traceable | `variation.name`, else the capacity |
| changi | the `level1saleMeasure` qualifier: `saleMeasureType` Volume as ml, Weight as g | the qualifier's value and unit |
| shilla | the "Weight,Volume" (else "Volume") spec row parsed with the same rules ("10G" is 10 g) | the spec row |
| avolta, shopify, ari, dubai | none: the name is parsed at ingest (Shopify's `variant.grams` is a shipping weight and is never read) | none |

The CLI side of both layers: `python -m app.cli backfill quantities` (every live product's
standard quantity from its name and stored pair; before `rederive`), `backfill listed` (the
listed columns from each listing's newest fragment; after `merges`), and `propose --file` (merge
proposals for the desk); `RUNBOOK.md` has the after-deploy order.

## The listed columns: one reader over the fragment (Stream L)

`services/collected.listed_fields` reads the LISTED layer of a listing (`listed_brand`,
`listed_name`, `listed_variant`, `listed_quantity_text`, `listed_category`, `listed_gtin`,
the two folds and the parsed quantity) from the fragment in `raw_records`, never from
`RawListing.name` (already ours: Avolta and Shilla prepend the brand, Shopify appends the
variant, Extime, Avolta and Changi append a size label). The same function runs at ingest
(right after the raw record is written; a quarantined price still refreshes the words) and
in `backfill listed` (the newest fragment per listing); every column plus `listed_record_id`
is written as one unit, NULLs included, so a newer fragment that lacks a field nulls it.
Texts are `html.unescape`d as ingest does, never trimmed or cased.

| Platform | brand, name | variant, size words | category | barcode |
|---|---|---|---|---|
| avolta | `tile.brand`, `tile.name` | the size words inside the name | the path of `category_url` | the SKU only when it already is a GTIN width (`gtin_from_sku`) |
| shopify | `product.vendor`, `product.title` | `variant.title` unless "Default Title" | `product_type` | none: the public feed carries no barcode |
| ari | `jsonld.brand.name`, `jsonld.name` | the size words inside the name | `jsonld.category` | `sku`, then `mpn`, each only when it validates as a GTIN (the fragments carry only sku and mpn) |
| extime | `jsonld.brand.name` else `main_offer.brand_name`, `variation.product_name` | `variation.name` (else `capacity capacity_unit`) | `main_offer.categories_name` joined with " > " | `main_offer.gtin`: the family barcode on a `::size` row, stored as listed, never fed to `ProductVariant.gtin` or the veto |
| shilla | `brand`, `name` | `specs["Weight,Volume"]` | `categories` joined | `ref_no` only when it validates |
| changi | `brandName`, `name` | the `level1saleMeasure` qualifier | the breadcrumb, reversed | none |
| dubai | `brand`, `displayName` | the size words inside the name | NULL (the feed carries category ids only) | `x_gTIN` |
| heinemann | `brand` else `manufacturerName`, `name` | `contentUnit` quantity and code | `typeOfGoods` | `gtin`, then `ean` |

## Posture per source
Stream A fills this: robots read date, identity mode, permission record, and whether prices
are collected, frozen, or a partnership ask. Never a typed count; the numbers come from
`python -m app.cli status`.

| Source | Robots read | Identity | Access | Posture |
|---|---|---|---|---|
| Singapore, iShopChangi (SIN) | 2026-09-05: product paths permitted, no Crawl-delay (floor 10 s) | honest UA | client-rendered; the page's own inventory response read through the sidecar; tag-manager host declared | collector built (`changi.py`), targeted set seeded (`.logs/runs/changi-targets-2026-09-05.md`); first run after the sidecar deploy; no barcodes |
| Heinemann global catalogue, Iceland (KEF), Sydney (SYD), BorderShop Scandinavia (SCA): `heinemann-global`, `iceland-duty-free`, `sydney-duty-free`, `bordershop-scandinavia` | 2026-08-23: `Disallow: /en/global/search/results` on all four hosts | honest UA | refused before the first request: `check_allowed` raises `SourceBlocked` (runs 52 to 55, 23 Aug) | frozen at their 21 to 22 Aug prices; never contacted again; the partnership ask is A16's (running list, 15 Sep) |
| Dubai Duty Free (DXB): `dubai-duty-free` | product paths permitted | honest UA | the edge answers 403 to our declared identity (verification check 396, 11 Sep); no run since 22 Aug | frozen at 21 to 22 Aug prices; Decision 3: no bypass, the ask at Cannes; the collectors page reads "refused since <date>" |
| Extime Paris (CDG): `extime-paris` | product paths permitted, floor 1.5 s | honest UA | text fetch; the 4 Sep crawl (run 115) predates `raw_records`, so most CDG listings have no fragment | the one stale shop that can be recrawled: rian, on bwlive, in an announced window (`.logs/runs/window-2026-09-15-extime-fragments.md`) |
| Seoul, Shilla (ICN) | 2026-09-05: product paths permitted, Crawl-delay 5, assets under `/estore/_ui/`, `/medias/`, `/files/` disallowed | honest UA | text client is challenged by Cloudflare; the browser passes; document is server-rendered (read at `domcontentloaded`, assets never fetched) | confirmed possible (`.logs/runs/shilla-probe-2026-09-05.log`); collector built (`shilla.py`), discovery by home page + rails under a 200-render budget; first run after the sidecar deploy |
| Las Vegas, Avolta (LAS): `avolta-las` | 2026-09-19: `/en/`, category and product paths permitted, query strings too (paged), Crawl-delay 60; search, cart, customer, review and ajax paths disallowed | honest UA | text fetch; the grid declares its currency as `currency_code` (USD), not as the microdata `declared_currency()` reads, so the configured currency carries it; one tile in fifteen on the scouted grid types a Club Avolta price as its final price (running list, Stream A) | configured; prices collected from its first run; the shop is hidden until shown (`app.cli shops --show LAS`) |
| London Gatwick, Avolta (LGW): `avolta-lgw` | 2026-09-19: `/en/`, category and product paths permitted; `Disallow: /*?` (page-one-only, the held-URL re-read grows it from the second run), `/en/174/` and `/en/175/` closed; Crawl-delay 30; checkout, catalog, customer and review paths disallowed | honest UA | text fetch; the grid declares its currency as `currency_code` (GBP), so the configured currency carries it; one shop for the north and south terminals (the storefront is one catalogue with two sitemaps); the UK storefronts share one catalogue and most of Gatwick's grid prices equal the other UK airports' | configured; prices collected from its first run; the shop is hidden until shown (`app.cli shops --show LGW`) |
| Helsinki, Avolta (HEL): `avolta-hel` | 2026-09-19: no Disallow at all (every path permitted, query strings too, so paged), Crawl-delay 60, four sitemaps (two per language, en and fi) | honest UA | text fetch; the grid declares its currency as `currency_code` (EUR), so the configured currency carries it; the three multi-size tiles on the scouted grid carry a hidden Club Avolta block whose two figures are equal, so no member price reaches the parser (the Las Vegas item on the running list covers the visible case) | configured; prices collected from its first run; the shop is hidden until shown (`app.cli shops --show HEL`) |

## Opening hours (`services/hours/`, Stream G, 2026-09-11)

A source class of its own, decided 11 Sep: an airport's duty free hours are read from the
**airport operator's** own site where its robots allow, and entered by hand where they do not.
Never from a retailer's site, never guessed. Sources of truth: `services/hours/base.py` (the
shape, the one `collect()`, the line), `registry.py` (which platforms read), one module per
platform, `store.py` (the rows and the reader's rule), `models/hours.py`, `cli_hours.py`.

- **Robots per operator domain, before it is added.** Each domain is its own host with its own
  robots file, read fresh on every run through the collectors' one policy above; permission
  never transfers between domains. The verdict per domain and the outcome per airport of the
  first read are in `.logs/runs/hours-robots-<date>.md`. A refusal (401 or 403 at robots.txt or
  on a page, a challenge page behind a 200) is recorded by the run and that host is not asked
  again in the same run; the airport goes to hand population.
- **One collector per operator platform, never per airport.** The airports a platform serves
  are its configuration (`airports`); a platform declares its own pace (`delay_seconds`) where
  the host has shown it wants one, and the run uses the slowest of the host's Crawl-delay, the
  run's delay and that. The run's page cap is `MAX_PAGES`.
- **Validated on content.** A page that yields no store is `NothingParsed` (hand population),
  and a 2xx that is a challenge page is a refusal (`looks_challenged`).
- **Facts only.** What a parser keeps is the store's name as the operator lists it, its terminal
  and area, and its clock or the rule the operator gives instead ("open 3 hours before
  flights"); the operator's prose about the shops is never kept, and the fixtures are trimmed to
  the same.
- **The row.** `airport_hours(id, shop_id, source_kind collected|hand, entered_by_id,
  observed_at, source_url, text, detail, created_at)`: one row per reading, nothing deleted.
  `text` is the one line the page prints; `detail` the stores behind it and the pages read. The
  reader shows the newest **hand** row over any collected one (a human value is never
  overwritten by a machine; a later collected reading is stored and not shown), else the newest
  collected row, else nothing.
- **Provenance.** `GET /api/airports/{iata}/hours` answers `{kind: collected|hand|none, text,
  observed_at, entered_by_username, source_url}`; the airport detail carries the same as
  `guide.hours_provenance`. The public page prints the hours and the date they were collected or
  entered, never the word "verified", never a rate; the owner's views print the whole line
  (`components/AirportHours.tsx`).
- **Commands.** `hours collect [--airport IATA] [--check]` (announce it in `.logs/runs/`),
  `hours set <IATA> --file <one line> [--by <username>]` (a dated hand row named to an account),
  `hours show [IATA]`, `backfill hours_seed` (`RUNBOOK.md`).

## Legal guardrails (researched 2026-08-21)

**Not legal advice, and rian is not a lawyer.** These are conservative engineering
practices drawn from current US case law. **Action item: 30 minutes with an actual
lawyer before LAUNCH — not before the PoC.** A few dozen products, logged-out,
facts-only, at trivial volume is a negligible-risk proof of concept; 168 airports
refreshed daily as a commercial product is a different conversation.

**L1. NEVER LOG IN. Never create an account, never accept terms of service, never
authenticate to a retailer site.** This is the sharpest practical line in scraping law.
In *Meta v. Bright Data* (N.D. Cal. 2024) Bright Data won summary judgment on Meta's
breach-of-contract claim specifically because it scraped **only while logged out** — the
terms didn't reach that conduct. Conversely *hiQ v. LinkedIn* won on the CFAA and still
**lost on breach of contract**. Contract, not copyright, is the realistic exposure, and
logging in is what creates the contract. Happy convergence: this is also why Club Avolta
member prices are unreachable (F4) — the technical limit and the legal safe harbour
point the same way. Treat "log in to get better data" as permanently off the table.

**L2. Take facts, not expression.** Price, SKU, size, ABV, currency, stock status,
brand and airport are **facts** — uncopyrightable under *Feist*, and an actually-charged
retail price sits at the *NYMEX v. ICE* end of the spectrum (held not copyrightable)
rather than the *CDN v. Kapes* end (derived/estimated price guides, held copyrightable).
**But marketing copy and product photography are creative expression and are the real
copyright exposure in retail scraping** — see *FragranceNet.com v. FragranceX.com*, a
direct e-commerce competitor-scraping case where photographic originality survived a
motion to dismiss. Build rules: store structured attributes; **do not ingest or
republish retailer marketing prose**; **do not rehost retailer product photography** —
use brand-supplied or licensed imagery, or ship without photos, and revisit images as a
partnership deliverable. Always **link and attribute to the source retailer** (good
product design anyway, and non-attribution is an element of the "hot news" theory).
- *The imagery posture in practice:* every picture the site shows is brand-supplied (the
  client's folder, a brand owner's export) or openly licensed (Open Food Facts, a Wikimedia
  Commons mark under an open licence), never a retailer's photograph; every picture carries a
  recorded source, licence and level (brand, product line, product variant), a supplied picture is never
  replaced by a fetched one, and a takedown on complaint is one command away because the
  provenance says where each picture came from (`RUNBOOK.md` Pictures, `DATA-MODEL.md`).

**L3. Buying scraped data does not transfer the risk.** In *Reddit v. SerpApi*
(S.D.N.Y., 31 Jul 2026) the court held at the pleading stage that Perplexity — which
merely *purchased* SerpApi's output — could be a **direct** circumventer because it "set
the parameters for circumvention activity." Consequence for us: using Firecrawl or any
paid fetcher is a convenience and a capability, **not** a liability shield. Choose it on
engineering merit (D1), not in the belief that it moves risk off Adam.

**L4. The controlling question is genuinely unsettled, and it splits in our favour.**
Two federal courts reached **opposite results on the same anti-bot technology 11 days
apart**: *Google v. SerpApi* (N.D. Cal., 20 Jul 2026) dismissed DMCA §1201 claims
**without leave to amend** where the protected material wasn't copyrightable at all —
which is exactly the posture of bare price and SKU data; *Reddit v. SerpApi*
(S.D.N.Y., 31 Jul 2026) held defeating a bot wall by proxying, IP rotation and spoofed
user-agents **is** circumvention. **The best argument available to a price scraper is
therefore "the measure does not control access to a copyrighted work," not "our copying
was fair use."** L2 is what keeps that argument available — the moment we ingest
photography and prose, we hand the other side the copyrighted work it needs.

**L6. EUROPE INVERTS THE RANKING — the most counterintuitive finding, and the one most
likely to catch us out.** The controlling case is *Ryanair v PR Aviation* (CJEU
C-30/14, 2015) — **literally a flight price-comparison site scraping an airline**, i.e.
our exact product. Held: where a database is protected by **neither** copyright **nor**
the sui generis database right, the Database Directive's mandatory lawful-user rights
don't apply, so **the owner is free to impose contractual restrictions**. The paradox:
**an unprotected database gets MORE effective protection than a protected one.**
Concluding "prices are just facts, so no database right" does not clear us — it moves
us into the zone where the retailer's terms bite hardest. **In Europe, contract is the
primary exposure, not IP**, which makes L1 (never log in, never accept terms) the
single highest-value control in the whole build.

**L7. Architecture is legally constrained: crawl-and-index, NEVER live query
passthrough.** *Innoweb v Wegener* (CJEU C-202/12) held that a dedicated meta search
engine which **translates a user's query in real time into the target's own search
engine** and re-presents the results is unlawful re-utilisation — "very close to the
manufacture of a parasitic competing product." A periodic crawl feeding **our own
index** is materially different. So: never proxy a user's search into a retailer's
search. (*CV-Online Latvia* C-762/19 further requires an adverse effect on the owner's
investment, which gives a comparator that **drives referral traffic** a real defence —
another reason every price should link back to the retailer.)

**L8. Jurisdiction matters, and it is not uniform.** **Germany is favourable** — the
BGH held (Ryanair v Cheaptickets, 2014) that automated retrieval for display on a
comparison site is not unfair obstruction **even where terms prohibit scraping**,
because price transparency benefits consumers. **France is the highest-risk** —
*parasitisme* requires neither confusion nor lost turnover, and copying a competitor's
**product descriptions** has been held unfair on its own (note `aeliadutyfree.fr` is a
plausible target). **The UK has no commercial TDM exception at all** — s.29A CDPA is
non-commercial research only. And *Getty v Stability* [2025] EWHC 2863 turned partly on
Getty being unable to show the acts happened in the UK: **where our crawling
infrastructure physically sits is a legal fact, not just an ops detail.**

**L9. The closest analogues to this product are the airline cases, and they all turned
on the same thing.** In *American Airlines v Skiplagged* (N.D. Tex. 2024) the
**contract claim failed** and the **copyright claim succeeded** — via AA's **logo and
creative content**, not via fare scraping. *Southwest v Kiwi.com* likewise ran on
contract plus **trademark**. **Every plaintiff who won, won on pictures and branding,
not on prices.** This is the strongest possible confirmation of L2: scrape prices,
never imagery or marketing prose, and never reproduce a retailer's logo.

**L10. The retail baseline is genuinely favourable — and the scary airline cases don't
generalise to us.** There is essentially **no US case law of a general retailer suing a
price-comparison service over price data**. The retail baseline is instead:
- ***Rapaport USA v. Nivoda*** (S.D.N.Y., Rakoff, May 2025) — **the single best
  authority for this product.** A subscription diamond price-list publisher sued a
  retailer for displaying its prices alongside its own. **Held: prices are NOT
  copyrightable, under the merger doctrine** — "the only way to express the price of a
  particular diamond is with the specific number corresponding to that price."
  (Two Second Circuit appeals pending; watch them.)
- ***Meta v. Bright Data*** — terms construed **against the drafter** as an adhesion
  contract and held **not to reach logged-off scraping of public data**.
- ***X Corp v. Bright Data*** (Alsup) — a contract claim whose real object is
  controlling copying/resale of content is "camouflage" for a copyright claim and is
  **preempted**. A second line of defence behind L1.
- ***Ryanair v. Booking*** — a pure CFAA scraping case litigated **to a jury verdict**,
  where the airline recovered **nothing**: only $2,457.72 of loss survived *Van Buren*'s
  "technological harms" limit, below the $5,000 threshold. Appeal voluntarily dismissed
  Aug 2025, so it stands. At summary judgment: **public pages are not accessed "without
  authorization."**
- ***Intel v. Hamidi*** — trespass to chattels requires actual impairment.

**Four things make the airline wins work, and none applies to us:** (1) a *booking
relationship* — Kiwi bought 20,000+ tickets through clickwrap, Booking's vendor created
accounts and transacted through a password-gated page; (2) a *logo hook* — American's
entire $9.4M was its Flight Symbol, and its **contract claim actually lost** (time-barred);
(3) a *consumer-harm narrative* (inflated fares, hidden-city disruption); (4) *Contract
of Carriage* violations. A read-only comparison site that never transacts has none of
these. Note also the Skiplagged jury found **nominative fair use** for using the
airline's mark to identify whose fare it was — which is exactly our attribution case.
**Identify, don't decorate.**

**L11. Two operational controls with real legal weight.**
- **Rate-limit hard.** *QVC v. Resultly* is the only retail case in this set with real
  exposure on the facts, and it was **200–300 requests/minute spiking to 36,000**,
  crashing QVC's site. It was never about the price data. Politeness delays and low
  concurrency aren't manners here, they're the damages defence — civil CFAA needs
  ≥$5,000 loss and trespass needs actual impairment.
- **If anyone asks us to stop, stop that day.** A cease-and-desist is the moment
  unenforceable browsewrap becomes an enforceable contract (*Southwest v. BoardFirst* —
  actual knowledge plus continued use equals assent; it's how Southwest has won
  repeatedly). The *QVC* court pointedly noted there was **no allegation Resultly
  refused to stop when asked**. **Build rule: a per-source kill switch in the admin, so
  a collector can be disabled in seconds without a deploy**, and route any retailer
  contact to rian immediately.

**L5. Everything above is provisional.** Nearly every case expanding liability here is
a district-court motion-to-dismiss ruling, which decides only that allegations state a
claim. The only binding appellate decisions in the set — *Cox v. Sony* (SCOTUS, Mar
2026) and *Amazon v. Perplexity* (9th Cir., Aug 2026) — both **narrow** liability. Note
*Amazon v. Perplexity* does **not** help us: it turned on an agent running in the end
user's own browser under their own credentials. A collector hitting a retailer from our
server is *us* accessing their servers, squarely outside that holding.

## Robots policy (`collectors/robots.py`, Stream A, 2026-09-04)

One module, one identity (`BOT_NAME = "DutyFreeProfessorBot"`), one policy; every
`collect()` and `read_one()` opens with `check_allowed(base_url, paths)` and takes its
crawl delay from the returned `Robots.delay_for(requested)`.

- A group applies to us when its agent value is a case-insensitive substring of
  `BOT_NAME` or the reverse. Never the first User-Agent token (that misreading once let a
  run past a group addressed to us by name).
- Paths match RFC 9309 style (`*` spans anything, `$` anchors). **Any matching Disallow is
  a no**, in our group or `*`, and a longer Allow does not out-lawyer it.
- Crawl-delay: our group's, else `*`'s; the run waits the slower of that and its own delay.
- Unreadable robots.txt: 404/410 (and other plain 4xx) means no rules, proceed and log the
  date; **401/403 is a refusal** (`SourceBlocked`, run status `blocked`); 5xx, 429 or a
  timeout stops the run without calling it a block (`RobotsUnavailable`, run status
  `error`, retried next run).
- Per-URL checks inside a run use `robots.allows(url)` offline; no second read.
- Measured 2026-09-04: LHR asks 30 s, the other nine Avolta stores 60 s; LHR, ATH, MAD and
  BCN forbid `/*?`, so they are page-1-only. Tests: `tests/test_robots.py`.

## read_one (`Collector.read_one(listing)`, Stream A, 2026-09-04)

One published listing read back from its source, fresh, for verification. Takes a
database `Listing` or a `ListingRef(source_sku, url, shop_code)`; returns a
`RawListing`, or `None` when the source no longer offers it (404/410, the page now serves
another SKU, the size is no longer priced); raises `SourceBlocked` when the host refuses
and `FetchError` when it should be retried. Opens with the same robots check as `collect()`
(cached ten minutes per host) and waits the host's crawl delay before the request.

| Collector | Reads | Pure function tested offline |
|---|---|---|
| Avolta | the product page: offer microdata for a plain SKU, `variant_prices()` for a `parent::size` row | `AvoltaCollector.listing_from_product_page` |
| Shopify | `/products/<handle>.json`, variant picked by id | `ShopifyCollector.listing_from_product_json` |
| ARI | the product page's JSON-LD (query string dropped) | `ari.listing_from_page` |
| Extime | the family page, the row with the same `source_sku` | `Extime._listings` |
| Dubai, Heinemann family | nothing: raises `SourceBlocked` (no permitted read path) | |

## Audit defects fixed in the first pass (Stream A, 2026-09-04)

Build plan §3a; every fix is a test in `tests/test_audit_defects.py` with the record that
failed. What changed at the collection layer:

- **Ingest refuses a barcode/size contradiction** (`RejectedListing("gtin_size_veto")`) instead
  of re-routing to brand+name+size, which once created two 7,000 ml Glenfiddichs. Rejections
  go through `record_rejection()`: counted on the run and stored in `rejected_observations`
  (migration #2, `d1e2f3a4b5c6`; the insert is guarded, so a schema mismatch cannot end a run).
- **`observed_at` is stamped per listing**, and each observation carries `fx_rate`; each run
  carries `fx_source` (live | fallback) and `fx_fetched_at`.
- **A source's earlier run still "running" is marked error when a new run starts.**
- **A shop's currency follows its collector's `ShopSpec`** (`sync_shop_currency`).
- **Dubai `in_stock` is None** unless the API item carries a stock field (`active` is not one).
- **Extime cross-checks the declared capacity** against `price_per_quantity` / `net_weight` when
  present and returns an unknown size on disagreement. The live shape of those two fields is
  pinned from a raw page (`tests/fixtures/extime_cheirosa_76.html`, 5 Sep); the older readings
  stay for the shapes seen before the fixture existed.
- **Image provenance** is `Open Food Facts (barcode)` or `(name)`; a name match requires the
  number tokens (ages, editions) to agree.
- Every `backfill <name>` takes `--check` (print what a run would change, write nothing).
- **Orphan parent tiles** (`-P` with no `::size` sibling) are purged by `backfill orphan_tiles`;
  the next collection of that store reads the product page and creates the per-size rows.

## Beauty widening (Stream A, 2026-09-05)

Decision 4: beauty is collected at every airport, targeted pages first and full crawls
last. The targeted set is `import/beauty-targets.json` (twenty fragrance and twenty skincare
folds, written by `scripts/beauty-candidates.py` from rows already held plus feeds already
fetched; the list with its evidence is `.logs/runs/beauty-candidates-<date>.md`). It is
mounted read-only into the container like the other import files and read by
`collectors/targets.py`; **an absent or empty file means no beauty product pages, never all**.

| Platform | Drinks | Beauty | Product-page fetches |
|---|---|---|---|
| Avolta | its own slug list, paged until a page adds nothing | its own slug list, **page 1 only** | a multi-size family only when `targets.matches(name, brand)`; drinks and beauty have separate budgets (`MAX_VARIANT_LOOKUPS`, `MAX_BEAUTY_LOOKUPS`) |
| ARI Dublin | every `/alcohol/` URL in the product sitemap | `/beauty/` (and sibling prefixes) only when the URL path names a targeted line (`ari.wanted_urls`; the slug omits the brand, so line words plus size) | one per wanted URL, as before |
| Shopify | the shop's shelf names, English and Spanish (`shopify.shelf_vertical`: Licores, Vinos, Rones, Champanas, Cordiales, Icewine, Eau de Vie...) | every beauty shelf (Fragancias, Rostro, Cuerpo, Capilar, Women's Perfume, Men's Cologne...): the feed arrives whole either way, so keeping every row costs no request | none (the feed is the read) |

- **Two trees, two caps, at Avolta.** Each tree is capped at `MAX_CATEGORIES_PER_VERTICAL`
  and a cap that drops anything logs `categories_capped store=<code> dropped=[...]`. The
  old single alphabetical cap once dropped whisky and wine at Heathrow because "beauty"
  sorts first.
- **Page-1-only stores re-read what they hold.** Where robots forbids the paging query
  string (LHR, ATH, MAD, BCN carry `Disallow: /*?`), the collector detects it
  (`page_one_only`), walks page 1, then re-reads every product URL the database already
  holds for the store that the walk did not reach (`AvoltaCollector._reread_held`; ingest
  hands the URLs over through `wants_held_listings`), one fetch per URL, bounded by
  `MAX_HELD_REREADS` and logged as `held_reread store=... unseen_urls=N reading=M`. The
  shrink at such a store is `prices_written` of its run against the previous one, plus the
  `held_reread_capped` line if the bound bit.
- **Shop-hidden products are not on the shelf.** Shopify tags such as `auto-oculto` and
  `Apagar` skip the product (`not_offered`, counted as a collector skip).
- **Extime's capacity is cross-checked against the live shape**: each price tier carries
  `price_per_quantity` and `price_per_quantity_type` ("100ml"); a declared capacity that
  disagrees with the implied size beyond the 700-vs-750 tolerance becomes an unknown size.
  The page that failed (a mist declared at 7,624 ml) is `tests/fixtures/extime_cheirosa_76.html`.
  `backfill implausible_sizes` clears sizes already stored that no single item comes in
  (`normalize.size_is_implausible`).