# Stream A: collectors and data
> **Status 11 Sep:** A13 done (the nineteen are on, behind the sign-in). The away period adds A14 to A18 at the bottom (collector hardening, wave two, **never while Stream M runs**). Earlier: A0 to A10 landed (handoffs 5 to 9 Sep, `A:` commits); open on `/plan`: A13 (rian switches the nineteen on and deploys), A11, A12. The body below is the brief as asked on 4 Sep.
**Rewritten 2026-09-04 (Fri) against build plan v3.** Supersedes the 3 Sep brief. Runs Fri 4
night → Sat 6 (first pass), then Wave 2 Mon 7 – Fri 11, then Wave 3.

## Goal
Make the collection layer honest, re-doable and beauty-capable, then collect the targeted
beauty set and refresh the sixteen. You own the data; Stream Q verifies it; Stream A2 owns the
browser path and its two collectors (Changi, Shilla). **You never touch A2's files:** `fetch.py`,
`registry.py`, `fx.py`, `changi.py`, `shilla.py`, the `dfp-browser` compose service. If you need a
change there, write it under "Requests" in `.logs/decisions-for-rian.md` and continue.

## Tonight (Fri 4 → Sat 6), in this order; each is a task id on /plan
1. **A1 `collectors/robots.py`** (about an hour, first, because A2 will import it). One shared
   module: `BOT_NAME = "DutyFreeProfessorBot"`; a group matches when its agent value is a
   case-insensitive substring of `BOT_NAME` (or vice versa), **never the first UA token** (today
   `heinemann._robots_allows` keys on "mozilla" and so ignores a group addressed to us by name:
   reproduced 2026-09-04); Allow/Disallow with wildcards, any matching Disallow is a no;
   crawl-delay from the same group rule (bot group over `*`); **one policy for unreadable robots:
   404/410 → unrestricted (log the date), 401/403 → refusal (`SourceBlocked`), 5xx/timeout →
   stop the run and retry next run.** `check_allowed(base_url, paths)` called at the top of every
   `collect()` and `read_one()`. Retire the stdlib parser from `avolta.py`; give ARI, Dubai and
   Shopify their runtime check. Tests: the by-name refusal, the wildcard `/*?` case, the three
   unreadable outcomes, crawl-delay precedence. Leave `fetch.py`'s `USER_AGENT` alone (A2 owns
   the file); import `BOT_NAME` from `robots.py` there is A2's one-line job.
2. **A2 (task id) Migration #1**, schema-only, one head, **deployed by rian Sat evening** (you
   never deploy): `accounts(id, email NULL, display_name, created_at)` seeded with one row for
   rian via a backfill command; `raw_records(id, listing_id FK, run_id FK, payload JSONB,
   parser_version)`; `price_observations.source_kind String(16) NOT NULL server_default
   'collector'`; `sources.identity_mode` (default `'declared'`) and `sources.permission_record`
   (nullable text); `Location.visible` default False (model + server_default). Write the
   migration; run it against the dev database `dfp-devdb` (127.0.0.1:5433), never the live one.
3. **A3 `RawListing.raw` and `RawListing.vertical`** (dataclass fields, `raw: dict | None`,
   `vertical: str | None`). Every collector you own populates `raw` with the parsed fragment it
   derived the listing from (the tile, the JSON-LD offer, the API item), not the page. Extime
   sets `vertical` from `page_category`; the others from their category family. Ingest writes
   `raw_records` when the table exists and sets `products.vertical` from the listing (backfill
   command for existing rows: taxonomy family → vertical).
4. **A4 `read_one(listing) -> RawListing | None`** on the `Collector` protocol (`base.py`) and
   implemented for Avolta (product page → `variant_prices()`/tile parse), Shopify
   (`/products/<handle>.json`), ARI (JSON-LD), Extime (`_listings`). Dubai and the Heinemann
   family raise `SourceBlocked`. Q builds `verify` against this; keep the signature stable.
5. **A5 the audit defect list** (plan §3a): orphan `-P` tiles (25) purged or re-resolved;
   `gtin_size_veto` never creates a wrong-size product; Extime capacity cross-checked against
   `price_per_quantity` and `net_weight`; Dubai `in_stock=None` unless the API exposes stock;
   MEX location currency; FX rate and source persisted per run, SGD/HKD/MXN in fallbacks (FX
   lives in `fx.py` = A2's file: write the request, do the rest); `image_source` split into
   barcode vs name provenance with a number-token agreement check; `observed_at` per fetch; the
   8 stuck runs marked error. Each fix becomes a test with the real record that failed.
6. **Collector-layer skips reach `rejected_observations`** through a run-scoped hook (Q owns the
   table; you add the hook and have it no-op until the table exists).

## If time remains tonight
- The **targeted-40 discovery** at near-zero network cost (plan §7 method): Dublin beauty slugs
  from its product sitemap, Shopify beauty rows already fetched, Changi slugs from
  `sitemap-products.xml`, Paris GTIN rows, the 9 ATH×CDG hits. Produce
  `.logs/runs/beauty-candidates-2026-09-05.md` (brand + line + size + concentration, distinct
  store count, GTIN yes/no). **No Avolta page fetches tonight**: those wait for rian's crawl
  token and `robots.py` being deployed.
- The **brand fold key** (case, accents, punctuation, trailing category words) as a pure
  function with tests, ready for migration #3.

## Wave 2 (Mon 7 – Fri 11), in this order; task ids are on /plan
**A10** drinks refresh of the sixteen minus the refusing shops (overnight; network) → **A7** targeted-40 list (Tue 8) → **A6** beauty widening → **A8** migration #3 (after Q2 lands; one head) → **A9** brand fold, uncategorised backfill, 52 merges. A11/A12 are Wave 3.

Beauty widening: per-vertical category walks (drop the alphabetical `MAX_CATEGORIES` slice; log
any category a cap drops, naming the store), Dublin beauty path prefixes, Shopify Spanish and
beauty shelf names, **page-1-only at LHR/ATH/MAD/BCN plus a re-read of every held `listings.url`**
(those four forbid `/*?`; measure the shrink at LHR first and tell rian). Targeted 40 list by
Tue 8. Migration #3 (brands table + `products.brand_id`, `size_value/size_unit` with `size_ml`
derived, per-vertical attribute column that **vetoes and never keys**, `identity_rules_version`,
`merged_into_id`, `merge_candidates`, `product_merges`, `reverifications`). Brand fold backfill,
uncategorised backfill, the 52 launch-scale duplicate groups as recorded merges with
`resolve_product_id()` for B. `app.cli images import <dir>`. The drinks refresh of the sixteen
minus Dubai, overnight, after robots.py is deployed. `app.cli rederive`.

## Owns
`app/services/collectors/*` **except** A2's files, `normalize.py`, `ingest.py`, `taxonomy.py`,
`cli.py`, `models/catalog.py`, `models/base.py`, `alembic/versions/` (migrations #1 and #3),
`main/scripts/` backfills, their tests, `main/docs/COLLECTORS.md` sections on your collectors
(A2 creates the file if absent; add to it, do not restructure it).

## Must not touch
`web/`, `seo.py`, `main.py`, `routers/`, `check.sh` (Docs owns it tonight), A2's files.

## Rules
- Conduct rules are absolute and are a selling point: honest UA, robots including wildcards,
  crawl-delay, never log in, never accept terms, stop when refused, no stealth.
- **No network collection tonight** beyond a single robots.txt read per host to test
  `robots.py`. Announce any run to rian first; log it under `.logs/runs/`.
- You never deploy and never run a migration against the live database.
- Migrations are schema-only; backfills are idempotent `app.cli backfill <name>` commands.
- Keep `size_ml` working throughout; `match_key` semantics change only in migration #3 with
  a `rederive`.

## Away period, wave two: collector hardening (A14 to A18; `/stream-a`; never with Stream M)
(added 2026-09-11 by the away-plan finalising session, from `AWAY-PLAN.md` section 4)

**Starts only when Stream M's final handoff is at the top of `.logs/handoff.md`**: the two
lanes share the identity layer (`normalize.py`, `merges.py`, `models/catalog.py`), and M owns it
until then. Staging only; nothing on production; A11 and A12 (network recollections) are not
started in this period.

**Working assumptions (rian's to overturn; `AWAY-PLAN.md` §3):** the four standing rules
hold (a blocked shop stays public on its last-seen prices with the date; robots 404 is open,
401/403 is refused; host cron for audit and verify is not automated collection; the
pages-2-and-later Avolta observations are marked unpublished, not purged). Singapore is not
retried. No host that has refused us is contacted again.

1. **A14 Paco Rabanne to Rabanne.** First read `brands.canonical_id` on `dfp-devdb`: if Stream
   M's M4 recorded the alias as rian's confirmed decision, verify it (the 100 ml 1 Million EDT
   resolves to one product at five airports), resolve the running-list issue with M's commit,
   and mark A14 done. If not, do not guess it in: the alias is a recorded human decision, so
   raise it on the merge queue M5 built (`app.cli suggest` should already list it) and mark
   A14 blocked on rian's confirm.
2. **A15 The six Avolta stores showing our own breakage.** From the run history on
   `/collectors` (the story column marks a refusal apart from our own failure), diagnose each
   store from its stored run rows and `rejected_observations`; where a reproduction needs a
   page, **one announced `read_one` per store** (robots re-read first, `.logs/runs/` entry,
   crawl delay honoured) and nothing more. Each fix is a test with the real record that
   failed and what it cost. A store that turns out to be refusing is recorded as such, not
   retried.
3. **A16 The four Heinemann-family shops that worked and then refused.** Recorded as refusals
   with the date of the last clean run and the first refusal (`sources.identity_mode` stays
   `declared`; the run story says "worked before, refused now"); never retried; their listings
   stay public on their last-seen prices with the date shown. Draft the partnership ask for
   rian in `notes/` (what we read, when they said no, what we would like), one page, no em
   dashes.
4. **A17 One display rule for sizes.** 700ml against 70cl: one function (`normalize.size_display`
   mirrored in `lib/format.ts`) decides what the page shows for a `size_value` and `size_unit`,
   applied in the product card, the product page, the listings table and the server body; the
   shop's own wording stays in `raw_records` and in the product name as the record. Tests on
   the pairs Mark named. A decide item if the rule needs rian's taste (which unit at which
   threshold), with the assumption you proceeded under.
5. **A18 Shops and terminals within one airport.** A written data-shape proposal
   (`.logs/planning/shops-and-terminals-2026-09.md`): what a shop and a terminal are, which
   collectors can name one today, what changes on the location, listing and product pages,
   what the address of a terminal page would be, and what it costs; then a decide item for rian
   with the options. **No migration in this period**: the model change touches addresses.

**Owns for these tasks:** as the Owns column above, plus `notes/` for A16's draft and the
proposal file for A18. `lib/format.ts` for A17 only. **Must not touch:** as above, plus
Stream G's and Stream W's columns (their briefs list them) and, until M's final handoff,
everything M owns.