# Stream AW1: three new collectors **Status 2026-09-19 19:10 UTC (updated at hand-back):** AW1.1 to AW1.5 done. The scout ran and its window closed; Las Vegas, London Gatwick and Helsinki are configured and tested (see Design decisions); window W1's file, scripts and commands are in `.logs/runs/` for the orchestrator. ## Goal Three new sources on platforms the app already parses, so that after deploy D1 and window W1 twenty-one sources work at launch (eighteen do on 19 Sep: ten Avolta, Dubai, Extime, Shilla, four Shopify, The Loop; five recorded refusals are never contacted again). Rian: "make them easy ones (more avolta ones or shopify or something)". A new source is configuration, a registry entry, tests on the real tiles the scout fetched, and rows the collector creates on its first run. Every new airport starts hidden. The orchestrator deploys D1 and launches W1 from AW1.5's commands. ## What already exists, so you build on it - `collectors/registry.py`: `COLLECTORS` from constructors (`avolta_collectors()`, `shopify_collectors()`, `TheLoop()`); `get_collector(slug)`. `collectors/avolta.py`: `AvoltaStore(code, iata, name, city, country, currency, base_url)`, the `STORES` tuple; `AvoltaCollector` (slug `avolta-`, `wants_held_listings=True`) opens with `check_allowed(base_url, ["/en/"])`, finds categories on `/en/` through `_LINK_RE`, derives `page_one_only` from `robots.allows(f"{first}?p=2")`, reads `declared_currency()` before any symbol, parses tiles with `parse_product_variants()`, defers `-P` tiles to `variant_prices()`, re-reads held URLs (`_reread_held`). A new store is one `AvoltaStore` line. - `collectors/shopify.py`: `ShopifyShop`, the `SHOPS` list of `(shop, retailer_slug, retailer_name, operator)`; `ShopifyCollector` (slug `shopify-`) reads `/products.json` under `check_allowed(base_url, [FEED_PATH])`; `shelf_vertical()` decides the family from the shop's shelf names; barcode `clean_gtin(variant.barcode) or gtin_from_sku(sku)`. The feed carries no currency: the shop's currency is configuration. A new shop is one `SHOPS` entry. - `collectors/ari.py`: `TheLoop`, one shop (`DUB`), `BASE` and `LOCATION_CODE` module constants, sitemap discovery, JSON-LD per product page. A second ARI host means parameterising it into a spec plus constructor; only if the scout finds one. - `collectors/robots.py` (`check_allowed`, `parse()`, `Robots.allows`, `delay_for`; tests inject a fetcher), `fetch.py` (`fetch()`, `SourceBlocked`, content never status), `targets.py` (the beauty targets apply to any Avolta store without configuration). - `services/ingest.py`: `run_collector` calls `ensure_source`, `ensure_retailer_and_shops` (`_get_or_create` in a savepoint, tolerating a concurrent creator), `sync_shop_currency`, `held_listing_refs`; `Shop.visible` defaults false (`models/catalog.py`, Decision 1), so a new shop is hidden until `app.cli shops --show `; `record_rejection` writes `rejected_observations(stage, reason, run_id)`. - `app/cli.py`: `collect --source [--limit] [--delay]`, `status`, `sweep plan`, `shops [--show|--hide]`, `backfill places` (one airport `Place` per `shops.iata`, idempotent, `--check`). `services/collector_view.py`: `PLATFORM_NAMES`, `platform_of()`, `_story()` (a refusal apart from our own failure); `tests/test_collector_view.py` fails on an unnamed platform. - `services/urls.py` `AIRPORT_SLUGS` (declared, mirrored exactly by `web/src/lib/urls.ts`, pinned by `tests/test_seo_airport.py::TestAirportSlugTable`; no entry falls back to `/airports/-`). `web/src/lib/countries.ts` `FLAG_OF_COUNTRY`, `REGION_OF_COUNTRY` and `web/public/flags/.svg`; an unknown country gets no flag, never a guess. - Tests to copy: `tests/test_collectors_avolta.py` (tiles as strings; fixtures `avolta_product_page.html`, `avolta_configurable_product.html`), `test_beauty_widening.py::TestShopifyShelves`, `test_read_one.py`, `test_raw_records.py` (a fragment carries facts, never markup), `test_parser_version.py` (`PINNED` per module: configuration changes no fingerprint; a change to the `raw=` expression bumps `PARSER_VERSION`). - Docs: `main/docs/COLLECTORS.md` (robots policy, listed columns, posture table), `RUNBOOK.md` Collecting, `main/CHANGELOG.md` Unreleased. Run records: `.logs/runs/window-2026-09-19b-sweep.md` (one process per host; the memory lesson), `sweep-2026-09-19b/{admit-queue,guard,finish}.sh`, `avolta-refresh-2026-09-05.log` (an Avolta airport takes 35 to 45 minutes at its delay). ## Tasks, in order (commit prefix `AW1:`) ### AW1.1 The scout: enumerate, read robots, one listing read each, rank, choose three (2 to 3 h) Write `main/scripts/scout-source.py --platform avolta|shopify [--fixture-dir]`, reusing `robots.check_allowed`, `fetch.fetch`, `avolta.parse_product_variants`, `avolta.declared_currency`, `avolta._LINK_RE`, `shopify.shelf_vertical`, `normalize.gtin_from_sku` and `clean_gtin`; never a second matcher. Per host at most three requests, each after the host's delay (`robots.delay_for(1.0)`): `robots.txt`; the discovery page (`/en/` for Avolta, how `collect()` starts; `/products.json?limit=250` for Shopify, which IS the listing); one drinks category page for Avolta (the first `_LINK_RE` finds). It prints the delay, whether `/en/` and `?p=2` (Avolta) or `/products.json` (Shopify) are allowed, the declared currency (Avolta; a Shopify feed carries none, so "unconfirmed" unless the feed page declares it), the tile count, and the barcode share (`gtin_from_sku` on Avolta SKUs; `barcode` or `gtin_from_sku(sku)` on in-scope Shopify variants). `SourceBlocked` or 401/403 at robots prints REFUSED and the host is never asked again; a hostname that does not resolve is "no such storefront", not a refusal. With `--fixture-dir` it saves the listing trimmed to facts (tiles only, `` and description blocks stripped; Shopify products through `facts_only`, at most ten) for the tasks below. Announce first: `.logs/runs/window-2026-09-19c-scout.md` (type run) names every host before the first request; close it with the per-host table. Candidates, none confirmed, all to test: the other UK airports on Heathrow's pattern (`london-gatwick`, `manchester`, `london-stansted`, `edinburgh`, `glasgow`, `birmingham`, `bristol` `.worlddutyfree.com`); the `*.shopdutyfree.com` airports the platform's own storefront list names (read `https://www.shopdutyfree.com/` once, robots first; Avolta's footprint suggests Vancouver, Las Vegas, Los Angeles, San Francisco, Melbourne, Helsinki, Lisbon, Milan); other Attenza storefronts on `.attenza.net` and any duty-free operator on Shopify the code or `.logs/` already names; a second ARI host only if one exists on the Salesforce shape. Also, on a restored copy of the newest dump (never staging), read `collector_view.collectors()['history']` and list any source whose story is our own breakage rather than a refusal (A15); record the diagnosis in the design section, not as one of the three. Rank by: allowed listing paths; barcode share before tile count (agents.md); a NEW airport before a second shop at an existing one (shops and terminals within one airport are not modelled); a country already in `FLAG_OF_COUNTRY`; a shorter published delay last. Choose three, write the ranking table with reasons into "Design decisions" below; commit the script, the window file and the brief. Telegram one line. ### AW1.2, AW1.3, AW1.4 One source each, in rank order (1 to 2 h each) For an **Avolta** store: one `AvoltaStore` line in `STORES` (code = IATA, currency = the declared one the scout read); an `AIRPORT_SLUGS` entry in the review's shape (name, code, city; JFK is the only code-first) in `urls.py` AND `lib/urls.ts`; a `FLAG_OF_COUNTRY` and `REGION_OF_COUNTRY` line plus the flag file only if the country is new. Tests in `tests/test_collectors_avolta.py` (a class per store, docstring naming the scout window): the slug is in `COLLECTORS` and `platform_of` answers `Avolta (Magento)`; `shops()` carries IATA, country, currency; `robots.parse()` allows `/en/` and says what `?p=2` is, so `page_one_only` is pinned to the truth; `parse_product_variants` on the fixture yields the scout's count with price, was-price and SKU from the typed blocks; `declared_currency(fixture)` equals the configured currency; the fixture's barcode share equals the scout's; a tile whose name omits the size gets it from the URL slug (`parse_quantity_ml_from_url`); `wants_held_listings` is true. Doc: a posture-table row in `COLLECTORS.md` (robots date, delay, page-one-only or paged, honest identity; no run, window or stream name: the first run is recorded in the window file and the CHANGELOG line) and one CHANGELOG line. For a **Shopify** shop: one `SHOPS` entry; the same slug, flag and platform steps. Tests in a new `tests/test_collectors_shopify.py` beside `test_beauty_widening.py::TestShopifyShelves`: `shelf_vertical` on the fixture's shelf names (an other-language shelf is named before it costs a shop half its drinks, as Panama's `Licores` did); `listing()` on a fixture product reads price, `compare_at_price` as was-price, options as fields, `barcode` cleaned; robots allows `FEED_PATH`; the barcode share; the configured currency equals the scout's reading. Posture row, CHANGELOG. For an **ARI** host (only if chosen): parameterise `TheLoop` into an `AriShop` spec plus constructor without changing the `raw=` expression (else bump `PARSER_VERSION` and re-pin); existing tests unchanged; the new host's tests mirror `test_read_one.py`'s ARI case. Every task: no row created by hand (the first run creates source, retailer and shop through `ensure_retailer_and_shops`; `backfill places` after W1); check.sh green; one commit. ### AW1.5 Window W1 for the orchestrator, and what to read afterwards (1 to 2 h) Write `.logs/runs/window-2026-09-19d-new-collectors.md` (type run): the three slugs, each host's published delay and expected duration (an Avolta host at 60 s is one to three hours; a Shopify feed is minutes), "no deploy while open", the closing section left for `finish.sh`. Copy `admit-queue.sh`, `guard.sh`, `finish.sh` from `sweep-2026-09-19b/` into `.logs/runs/new-collectors-2026-09-19d/` and adapt: the window path; the process-name `case` patterns in all three (`avolta*|extime*`) must match the new slugs exactly; `CEILING_MIB` was written for a 1 GB cap and the cap is 4 GB (state the figure and why); `finish.sh`'s summary limited to the three sources; Telegram `--from "AW1 W1"`. Then the exact commands after D1: the gate check (`app.cli status` shows no `running`; no `app.cli collect` process), `setsid nohup bash admit-queue.sh `, `guard.sh` and `finish.sh` in the background, `srv-gw notify` on open. How to read the outcome: `app.cli status` (three rows `ok (N prices)`), the run rows and `rejected_observations` by reason for those run ids (one `psql` line each, as `finish.sh` does), the barcode share per new shop (`listings` joined to `product_variants.gtin`), `app.cli shops` showing the new codes HIDDEN, then `backfill places --check` and `backfill places`. The airport page check is on a restored copy only: `shops --show ` there, render `/airports/` from the dev server, assert a string only that page contains (the airport's name in the H1), discard the copy. Handoff entry: the three sources, the W1 commands by reference, the after-run checks, the questions below with their assumptions. ## Design decisions (AW1.1 fills this; the orchestrator and rian read it) **Scout run 19 Sep 17:58 to 19:05 UTC**, window `.logs/runs/window-2026-09-19c-scout.md` (the per-host table, every request, and one deviation: a second pass of two requests per Avolta host because the first pass read the tree root, which is a landing page or a carousel, not the grid). **No host refused.** Every robots.txt answered 200. The registry at the hub names 171 storefronts; of the brief's eight `shopdutyfree.com` candidates only Helsinki, Vancouver, Las Vegas and Milan Malpensa exist (no Los Angeles, San Francisco, Melbourne or Lisbon storefront), all seven UK hosts exist, and of fourteen `.attenza.net` codes only `ec` resolves. **Ranking** (the brief's order: allowed listing paths; barcode share before tile count; a new airport; a country already in `FLAG_OF_COUNTRY`; the shorter published delay last). Every host below is a new airport. | rank | host | airport | platform | robots | delay | paging | currency | grid tiles | barcodes | flag | chosen | |---|---|---|---|---|---|---|---|---|---|---|---| | 1 | `lasvegas.shopdutyfree.com` | LAS, Las Vegas, United States | Avolta | 200 | 60 s | paged | USD | 15 (4 multi-size) | 33% (5) | yes | **AW1.2, `avolta-las`** | | 2 | `london-gatwick.worlddutyfree.com` | LGW, London, United Kingdom | Avolta | 200 | 30 s | page-one-only (`/*?`) | GBP | 9 (1) | 22% (2) | yes | **AW1.3, `avolta-lgw`** | | 3 | `helsinki.shopdutyfree.com` | HEL, Helsinki, Finland | Avolta | 200 | 60 s | paged | EUR | 10 (3) | 20% (2) | no: `fi.svg` and two table lines | **AW1.4, `avolta-hel`** | | 4 | `manchester.worlddutyfree.com` | MAN, United Kingdom | Avolta | 200 | 30 s | page-one-only | GBP | 9 (0) | 22% (2) | yes | no: 6 of its 9 grid prices equal Gatwick's | | 5 | `london-stansted.worlddutyfree.com` | STN | Avolta | 200 | 30 s | page-one-only | GBP | 9 (1) | 22% (2) | yes | no: 9 of 9 equal Gatwick's | | 6 | `edinburgh`, `birmingham`, `bristol` `.worlddutyfree.com` | EDI, BHX, BRS | Avolta | 200 | 30 s | page-one-only | GBP | 9 (1) | 22% (2) | yes | no: 8 of 9 equal Gatwick's | | 7 | `ec.attenza.net` | UIO, Quito, Ecuador | Shopify | 200 | 1 s (ours) | feed | USD (home page) | 134 variants on 102 of 250 products | 1% (1) | no | no: without barcodes its rows match only the three Attenza shops we have | | 8 | `milan.shopdutyfree.com` | MXP, Milan, Italy | Avolta | 200 | 60 s | page-one-only | EUR | 29 (4) | 3% (1) | no | no | | 9 | `vancouver.shopdutyfree.com` | YVR, Canada | Avolta | 200 | 60 s | paged | CAD | 1 (a one-item category) | 0% | yes | inconclusive: rescout on a bigger category before choosing | | 10 | `glasgow.worlddutyfree.com` | GLA | Avolta | 200 | 30 s | page-one-only | | the first child category (`2-for-20`) was a dead link, 404 | | yes | no grid read; nothing wrong with the host | **Why these three.** Las Vegas leads on every criterion that separates hosts (paged, the highest barcode share, a flagged country). Gatwick is the next barcode share, a flagged country and the shorter delay; it is page-one-only exactly as Heathrow is (`Disallow: /*?`), so its first run reads page one of each category and the held-URL re-read grows it from the second run. Helsinki is third over Manchester because the first criterion (allowed listing paths: paged against page-one-only) and a new country outrank a flag line and a second UK shop whose price list mostly repeats Gatwick's; the flag is one file from flag-icons 7.5.0 and two table lines. *To overturn: name Manchester; the change is one `STORES` line, two slug lines, no flag work.* **What the scout read that the tasks must know.** - The UK storefronts share one catalogue: the six grids read carry the same nine SKUs and 6 to 9 of the prices are identical to Gatwick's. They are still distinct shops at distinct airports (prices do differ), but a second UK host adds coverage more than price information. - Every Avolta grid declares its currency as `"currency_code": "XXX"` per tile, never as the `priceCurrency` microdata `declared_currency()` reads, so the collector falls back to the symbol map (which lacks `CA$`) and then the configured currency. The scout reads the `currency_code` form and the fixture head carries it as a `priceCurrency` line, so `declared_currency(fixture)` answers what the page declared (running list, P3, Stream A). - The tree root a collector reads first is a landing page at `shopdutyfree.com` and a carousel at `worlddutyfree.com` whose links are `href`-first (unparsed); the collector finds no tile there and walks on, so nothing is lost. The scout prefers a child category now. - Expected W1 durations at the published delays: Las Vegas and Helsinki one to three hours each (8 to 9 drinks categories paged, the beauty page-one, up to 40 multi-size product pages at 60 s); Gatwick thirty to sixty minutes (page one of each category at 30 s, nothing held yet). **A15 diagnosis (read on a restored copy of the 19 Sep nightly, `dfp_aw1`, then dropped).** No source's story is our own breakage dressed as a refusal. Every `blocked` run is a genuine refusal: the four Heinemann hosts by `Disallow: /en/global/search/results` (23 Aug), iShopChangi by a 403 and an empty render (5 and 11 Sep), Dubai by the edge's 403 to the declared identity (verification check, 11 Sep). Every `error` run is ours: "interrupted by deploy" (23 Aug), "stuck: process ended without finishing" (the 25 Aug and 19 Sep container deaths), one dead category link at Zurich (23 Aug, a 404 the collector now tolerates) and one product-page fetch at Dublin (22 Aug). One wording defect: `collector_view._story` counts a run still `running` as failed, so a source mid-sweep reads "worked before, refused now" (ATH, BCN, HKG, MAD, Extime at dump time). Nothing to repair as a source; the wording is on the running list. ## Working assumptions (rian's to overturn) - **The three are new sources at new airports**; a repaired A15 store or a second shop at an airport we show does not count. *To overturn: name the store; the fix is one task.* - **New airports stay hidden until rian looks** (`Shop.visible` defaults false; nothing here shows them). *To overturn: `app.cli shops --show `.* **Declared address slugs are written now** in the review's shape. *To overturn: change the two table lines; old shapes land.* - **No new platform**: fewer than three on Avolta, Shopify or ARI means handing back fewer. **A country not in the flag map gets no flag** (the code's rule) and a `do` item for rian. - **The scout's budget is three requests per host** (robots, discovery, one listing) under the host's delay inside window c; the discovery page is what `collect()` reads first, not a listing. *To overturn: two requests, no category page.* ## Owns `main/app/services/collectors/avolta.py` (`STORES` only), `shopify.py` (`SHOPS` only), `ari.py` (parameterisation, if chosen), `registry.py`; `collector_view.py` (`PLATFORM_NAMES` only); `services/urls.py` (`AIRPORT_SLUGS` only), `web/src/lib/urls.ts`, `web/src/lib/countries.ts`, `web/public/flags/`; `main/scripts/scout-source.py`; `main/tests/test_collectors_avolta.py`, `test_beauty_widening.py`, a new `test_collectors_shopify.py`, `test_parser_version.py` (re-pin only), new fixtures; `main/docs/COLLECTORS.md` (posture rows), `RUNBOOK.md` Collecting, `main/CHANGELOG.md`; `.logs/runs/window-2026-09-19c-scout.md`, `window-2026-09-19d-new-collectors.md`, `.logs/runs/new-collectors-2026-09-19d/`; this brief; `.logs/handoff.md` (prepend); `import/items.json` through `items.py` only. ## Must not touch `services/identity.py`, `keying.py`, `normalize.py`, `merges.py`, `ingest.py`, `collectors/{base,fetch,robots,targets}.py`, the parsing functions of `avolta.py` and `shopify.py`, `main/docs/REVIEW-PROCESS.md`, migrations, `docker-compose.yml`, `.env`, `.app.env`, `main/check.sh`; staging's and production's databases (rehearse on `dfp-devdb` from a fresh copy of the newest dump); `.logs/runs/sweep-2026-09-19b/`, `.logs/runs/uptime.log`, `notes/`; every file AW2 to AW7 own; any host `sweep plan` lists as refused, Dubai, Singapore. ## Rules that bite here - Never log in, never accept terms; robots re-read every run, modern wildcards, ANY matching Disallow is a no; crawl delay the slower of theirs and ours; a block is a refusal, recorded, never escalated; consent is data (`identity_mode` moves only with a `permission_record`; nothing here moves it). Validate on content, never status. - Listing pages, not product pages; a new source on an existing platform is configuration. - GTIN is identity; weigh a shop on barcode share before product count; the key-holder lookup and the savepoint creates are ingest's, untouched. A multi-size tile is a family: no barcode minted from a parent SKU. Currency from the page's declaration, never a symbol; a new shop an order of magnitude off its peers is a bug to report, not a bargain. - A collection runs only in an announced window launched by the orchestrator; the scout's window is the one exception the away plan grants. - A bug found becomes a test with the real record; check.sh green before every commit; one task, one commit; client-facing copy has no em dashes and never "cheap" or "free"; one word per concept (`VOCABULARY.md`: shop, place, retailer, source; product line, product variant, brand; never bare "product"). ## Questions for rian 1. Does a repaired A15 store count as one of the three? *Assumption: no; three new sources.* 2. Fewer than three pass on the existing platforms: hand back fewer, or open a new platform? *Assumption: hand back fewer.* 3. Do the new airports stay hidden through the Monday launch? *Assumption: hidden until you show them; their collectors run regardless.* 4. Do the new airports' declared addresses go to Adam with the nineteen (the running list's web-addresses item)? *Assumption: yes, appended to that item, not blocking.* ## Out of scope, noted Opening hours and the guide document (G2) for a new airport: a hand line or nothing. `admit-queue.sh` and `guard.sh` are per-window copies; a `sweep run` launcher in `app.cli` would retire them. A Shopify feed carries no currency; reading the storefront's declaration each run would close that gap.