# Stream A: collectors and data
> **Status 10 Sep:** A0 to A10 landed (handoffs 5 to 9 Sep, `A:` commits); open on `/plan`: A13 (rian switches the nineteen on and deploys), A11, A12. The body below is the brief as asked on 4 Sep.
**Rewritten 2026-09-04 (Fri) against build plan v3.** Supersedes the 3 Sep brief. Runs Fri 4
night → Sat 6 (first pass), then Wave 2 Mon 7 – Fri 11, then Wave 3.

## Goal
Make the collection layer honest, re-doable and beauty-capable, then collect the targeted
beauty set and refresh the sixteen. You own the data; Stream Q verifies it; Stream A2 owns the
browser path and its two collectors (Changi, Shilla). **You never touch A2's files:** `fetch.py`,
`registry.py`, `fx.py`, `changi.py`, `shilla.py`, the `dfp-browser` compose service. If you need a
change there, write it under "Requests" in `.logs/decisions-for-rian.md` and continue.

## Tonight (Fri 4 → Sat 6), in this order; each is a task id on /plan
1. **A1 `collectors/robots.py`** (about an hour, first, because A2 will import it). One shared
   module: `BOT_NAME = "DutyFreeProfessorBot"`; a group matches when its agent value is a
   case-insensitive substring of `BOT_NAME` (or vice versa), **never the first UA token** (today
   `heinemann._robots_allows` keys on "mozilla" and so ignores a group addressed to us by name:
   reproduced 2026-09-04); Allow/Disallow with wildcards, any matching Disallow is a no;
   crawl-delay from the same group rule (bot group over `*`); **one policy for unreadable robots:
   404/410 → unrestricted (log the date), 401/403 → refusal (`SourceBlocked`), 5xx/timeout →
   stop the run and retry next run.** `check_allowed(base_url, paths)` called at the top of every
   `collect()` and `read_one()`. Retire the stdlib parser from `avolta.py`; give ARI, Dubai and
   Shopify their runtime check. Tests: the by-name refusal, the wildcard `/*?` case, the three
   unreadable outcomes, crawl-delay precedence. Leave `fetch.py`'s `USER_AGENT` alone (A2 owns
   the file); import `BOT_NAME` from `robots.py` there is A2's one-line job.
2. **A2 (task id) Migration #1**, schema-only, one head, **deployed by rian Sat evening** (you
   never deploy): `accounts(id, email NULL, display_name, created_at)` seeded with one row for
   rian via a backfill command; `raw_records(id, listing_id FK, run_id FK, payload JSONB,
   parser_version)`; `price_observations.source_kind String(16) NOT NULL server_default
   'collector'`; `sources.identity_mode` (default `'declared'`) and `sources.permission_record`
   (nullable text); `Location.visible` default False (model + server_default). Write the
   migration; run it against the dev database `dfp-devdb` (127.0.0.1:5433), never the live one.
3. **A3 `RawListing.raw` and `RawListing.vertical`** (dataclass fields, `raw: dict | None`,
   `vertical: str | None`). Every collector you own populates `raw` with the parsed fragment it
   derived the listing from (the tile, the JSON-LD offer, the API item), not the page. Extime
   sets `vertical` from `page_category`; the others from their category family. Ingest writes
   `raw_records` when the table exists and sets `products.vertical` from the listing (backfill
   command for existing rows: taxonomy family → vertical).
4. **A4 `read_one(listing) -> RawListing | None`** on the `Collector` protocol (`base.py`) and
   implemented for Avolta (product page → `variant_prices()`/tile parse), Shopify
   (`/products/<handle>.json`), ARI (JSON-LD), Extime (`_listings`). Dubai and the Heinemann
   family raise `SourceBlocked`. Q builds `verify` against this; keep the signature stable.
5. **A5 the audit defect list** (plan §3a): orphan `-P` tiles (25) purged or re-resolved;
   `gtin_size_veto` never creates a wrong-size product; Extime capacity cross-checked against
   `price_per_quantity` and `net_weight`; Dubai `in_stock=None` unless the API exposes stock;
   MEX location currency; FX rate and source persisted per run, SGD/HKD/MXN in fallbacks (FX
   lives in `fx.py` = A2's file: write the request, do the rest); `image_source` split into
   barcode vs name provenance with a number-token agreement check; `observed_at` per fetch; the
   8 stuck runs marked error. Each fix becomes a test with the real record that failed.
6. **Collector-layer skips reach `rejected_observations`** through a run-scoped hook (Q owns the
   table; you add the hook and have it no-op until the table exists).

## If time remains tonight
- The **targeted-40 discovery** at near-zero network cost (plan §7 method): Dublin beauty slugs
  from its product sitemap, Shopify beauty rows already fetched, Changi slugs from
  `sitemap-products.xml`, Paris GTIN rows, the 9 ATH×CDG hits. Produce
  `.logs/runs/beauty-candidates-2026-09-05.md` (brand + line + size + concentration, distinct
  store count, GTIN yes/no). **No Avolta page fetches tonight**: those wait for rian's crawl
  token and `robots.py` being deployed.
- The **brand fold key** (case, accents, punctuation, trailing category words) as a pure
  function with tests, ready for migration #3.

## Wave 2 (Mon 7 – Fri 11), in this order; task ids are on /plan
**A10** drinks refresh of the sixteen minus the refusing shops (overnight; network) → **A7** targeted-40 list (Tue 8) → **A6** beauty widening → **A8** migration #3 (after Q2 lands; one head) → **A9** brand fold, uncategorised backfill, 52 merges. A11/A12 are Wave 3.

Beauty widening: per-vertical category walks (drop the alphabetical `MAX_CATEGORIES` slice; log
any category a cap drops, naming the store), Dublin beauty path prefixes, Shopify Spanish and
beauty shelf names, **page-1-only at LHR/ATH/MAD/BCN plus a re-read of every held `listings.url`**
(those four forbid `/*?`; measure the shrink at LHR first and tell rian). Targeted 40 list by
Tue 8. Migration #3 (brands table + `products.brand_id`, `size_value/size_unit` with `size_ml`
derived, per-vertical attribute column that **vetoes and never keys**, `identity_rules_version`,
`merged_into_id`, `merge_candidates`, `product_merges`, `reverifications`). Brand fold backfill,
uncategorised backfill, the 52 launch-scale duplicate groups as recorded merges with
`resolve_product_id()` for B. `app.cli images import <dir>`. The drinks refresh of the sixteen
minus Dubai, overnight, after robots.py is deployed. `app.cli rederive`.

## Owns
`app/services/collectors/*` **except** A2's files, `normalize.py`, `ingest.py`, `taxonomy.py`,
`cli.py`, `models/catalog.py`, `models/base.py`, `alembic/versions/` (migrations #1 and #3),
`main/scripts/` backfills, their tests, `main/docs/COLLECTORS.md` sections on your collectors
(A2 creates the file if absent; add to it, do not restructure it).

## Must not touch
`web/`, `seo.py`, `main.py`, `routers/`, `check.sh` (Docs owns it tonight), A2's files.

## Rules
- Conduct rules are absolute and are a selling point: honest UA, robots including wildcards,
  crawl-delay, never log in, never accept terms, stop when refused, no stealth.
- **No network collection tonight** beyond a single robots.txt read per host to test
  `robots.py`. Announce any run to rian first; log it under `.logs/runs/`.
- You never deploy and never run a migration against the live database.
- Migrations are schema-only; backfills are idempotent `app.cli backfill <name>` commands.
- Keep `size_ml` working throughout; `match_key` semantics change only in migration #3 with
  a `rederive`.