# Stream L: the quantity model, the listed columns, the giant Listings table, and the merge desk (written 2026-09-14 Mon, from rian's decisions in the same day's thread; build plan v3 §4 is the design it extends, Stream M is the identity layer it builds on) > **Status 14 Sep:** not started. One overnight session, wave three, due 15 Sep. Runs as `/stream-l`. **Runs:** alone. Never alongside Stream A, Stream M or Stream G (they share `ingest.py`, the keying layer, `collector_view.py` and `CollectorsPage.tsx`); three sessions is the ceiling (Decision 13) and tonight this lane is the only one on the identity layer. Staging only; nothing on production; never deploys; rian holds the deploy, migration and crawl tokens. Opened in the main tree, `/srv/apps/dutyfreeprofessor`, as OVERNIGHT-RULES requires; not in a worktree. A worktree cut from master has no `backups/` (gitignored), so the dump below cannot be restored there, and this brief and the command file must be committed to master before the run (see "Before the run"). Task ids are **LT1 to LT8**: `L1` to `L11` already belong to Stream R in `import/progress.json`. **Decided by rian, 14 Sep:** three layers for every value. **Listed** is the words exactly as the shop showed them, never changed. **Standard** is what our rules made of it, recomputable at any time. **Decided** is a human override of a rule, and it survives every recomputation. Standard identity lives on the product, never on the listing; a listing-level correction is "move this listing to product X" or "ignore this listing", never a per-listing standard name. The measurable thing is a **quantity** with a unit, not a size; "size" is kept back for apparel later. A variation carries a **kind**; "shade" is a shop word for the kind **color** (American spelling in code). The quantity change lands before any merge decision is made. **This stream ends with rian able to open the giant Listings table, turn on every column, and start the manual review, with a merge desk beside it where he approves suggested merges in bulk as his own decisions.** ## Goal Retire, in this order, the things that can go wrong: a key change after merges are decided, a migration with two heads, a rekey that undoes hand work, a backfill that is not idempotent, an auto-fold that joins two different things because neither states a quantity, a backfill that labels our text as the shop's, a table that cannot sort 24k rows, a bulk approve that costs one full regeneration per row, and a recrawl that gets us refused. Then land what rian asked for on top of the retired risks: the quantity model (A), variation kinds with color (B), the listed columns (C), provenance and the override reader (D), the giant table with the column picker (E), the merge desk with approve-only suggestions (F), the honest recrawl plan (G), in the order H demands: **the quantity change lands before anyone merges.** At the end of the night rian sits down and: opens `/collectors`, Listings, turns every column on, filters and sorts with the database doing the work, downloads a CSV, and starts the manual review; opens Merge, Desk, sees a brands table and a lines table with a suggested merge, a reason and a score on each row, ticks the ones he agrees with, presses Approve, and they apply as his decisions; reads one handoff paragraph that says which shop he can recrawl, why the other five are not touched, and the one command list to run after deploy. ## What already exists, so you build on it - `products` carries `size_value`/`size_unit` (the stated pair, written once at create by `ingest.stated_size`) and `size_ml` (derived, the only thing the public API and the key read). `normalize.parse_size`/`size_ml_of` understand only l, cl and ml; `RawListing.size_ml` (`collectors/base.py:85`) is the only size field a collector can emit, and every structured-unit reader is volume-only (Heinemann `_VOLUME_UNITS`, Extime `_UNIT_TO_ML`, Changi `saleMeasureType == "volume"`, Shilla through `parse_size_ml`). On staging `size_unit` is ml/l/cl/NULL and nothing else; 557 live names carry a gram figure that is discarded and a few carry a wrong ml (id 7293 "Dior Sauvage Stick Deodorant 75g" stored 75 ml; id 14386 "Benefit Pop, Lock & Lash It 4g + 3.5ml" stored 35 ml). `DATA-MODEL.md:62` and the model comment at `catalog.py:295` say "75 g" is stored; it is not, and never has been. Do not build on that sentence. - `normalize.match_key` (v3) composes `house|line|variation|[|form]`; 3,416 live products key on `|0` and 65 live key groups collide there (38 beauty, 19 liquor, 4 confectionery). `form_of` gives set/refill/pack, but `pack` is volume-only; a set keys on whichever number the regex met first ("Daywear Set 50ml + 2x15ml + 5ml" keys at 30). Exact v3 strings are pinned in `test_keying.py` (`rabanne|1-million|edt|100`, `chivas-regal|12||2000|pack`), `test_merges.py` (positional calls, no vertical), `test_aliases.py`, `test_line_backfills.py`; under v4 every one of those strings gains its unit (rian, 15 Sep: one clean rule for every dimension), so every row is re-keyed once, tonight, before any merge decision exists. - `keying.py`: `Maps` (brands, variation aliases, lines; 300 s cache; `invalidate()`), `listing_key` for ingest, `product_key` for `rederive`, the alias routes and `backfill implausible_sizes`. `lines.variation_of(name, vertical)` returns `("", "")` outside beauty (`VARIATION_VERTICALS`, one global rule list); liquor ages already sit in the line (`chivas-regal|12`), by Stream M's decision. Brand and line aliases (`canonical_id`) and product merges (`merged_into_id`, `product_merges`) survive a rekey. A hand-edited product name does not: `overrides` (accounts.py:179) has zero rows, zero writers and no reader in the keying path; `award_picker.pin_from_override` is the only decoder. - A confirmed product merge across two keys does not survive the next collection: `ingest._resolve_product` looks up `match_key` on live rows only (`merged_into_id IS NULL`) and `_resolve_listing` re-pins `listing.product_id` on every sighting. The 576 `variation_unknown` pairs on staging are that class. - `listings` has seven columns. The shop's own words live only in `raw_records.payload` (one JSONB row per listing per run, never upserted), read back per row by `collector_view.collected_tile`, one branch per platform, returning name, brand, size, size_ml, price, was_price, currency; it reads no category path, barcode or variant text. Fragments exist for 11,098 of 23,771 listings; **12,673 have none** (CDG 6,836 of 7,036; DXB 1,489; HEINEMANN-GLOBAL 1,035; KEF 963; SYD 444; SCA 1,806). CDG's gap is history, not a bug: the 7,032-row Extime crawl (run 115, 4 Sep) predates `raw_records`, and the only run since was `collect --limit 200` on 11 Sep. `RawListing.name` is already ours (Avolta and Shilla prepend the brand, Shopify appends the variant, Extime, Avolta and Changi append a size label). `collector_view` imports `verify`, `verify` imports `ingest`, so `ingest` cannot import `collector_view`. - `/collectors` (owner-only, `sources.manage`): the Listings tab pages 50 rows (max 200), filters q/brand/collector/airport/differs, no sort, no export; the differs filter loads every fragment into Python. The Merge tab is the keyboard queue (y/n/s, one level at a time) over `merge_candidates` from `suggest.generate` and `merges.record_candidates`; confirm goes through `merge_session.confirm`; a brand or line confirm regenerates every suggestion inside the request; `generate` withdraws any undecided pair whose reason is in `RULE_REASONS` and the rules no longer produce, and rescores the rest. Barcode veto: two differing barcodes need a note. Staging waits with 27 brand, 2,036 line, 1,057 product pairs undecided, zero decided. - CLI: `app.cli backfill ` registry `BACKFILLS` (cli.py:722), every backfill idempotent with `--check`; `rederive`, `suggest`. Current Alembic head `a2b3c4d5e6f7`, pinned with the exact chain in `tests/test_accounts_backfills.py:71-80`. Migration ids are hand-chosen 12-hex in a rotating pattern; two lanes already picked the same one in one week. - The rehearsal database is `dfp-devdb` (127.0.0.1:5433, dfp/dfp); its own `dfp` database is stale. Newest dump: `/srv/apps/dutyfreeprofessor/backups/dfp-nightly-2026-09-14.dump` (always the absolute path; `backups/` is gitignored). Staging `dutyfreeprofessor-db` is read-only for this stream (`docker exec dutyfreeprofessor-db psql -U dfp -d dfp -At -c "SELECT ..."`). `import/` is mounted read-only into the app container at `/srv/import`. - Recrawl facts: heinemann-global, iceland-duty-free, sydney-duty-free, bordershop-scandinavia have one blocked run each (23 Aug 18:34 UTC, `robots.txt disallows /en/global/search/results`); `check_allowed` raises `SourceBlocked` before the first request, and policy says a host that refused is never contacted again (AWAY-PLAN §2; A16 owns the partnership ask). dubai-duty-free's last run is 22 Aug "ok" because nobody has run it since: its edge answers 403 to our declared identity (Decision 3, 4 Sep: no bypass, the ask at Cannes); `collector_view._story` therefore reads "nothing attempted for N days" while verification check 396 says BLOCKED. Extime is the only stale shop that can be recrawled: about five hours at its 1.5 s floor, on the droplet, by rian, inside an announced window. No cron runs any collector. ## The model, in rian's words and the columns they become | Layer | Lives on | Columns | Written by | Survives | |---|---|---|---|---| | Listed (the shop's words) | `listings` | `listed_brand, listed_name, listed_variant, listed_quantity_text` (text; the shop's size words, named for the "quantity" rule), `listed_category` (path, " > " joined), `listed_gtin`; folds for SQL `listed_brand_key` (slug-shaped: `brand_key(x).replace(" ", "-")`, the fold `brands.slug` uses), `listed_name_key`; the listed text's parsed `listed_quantity_value`, `listed_quantity_unit`; `listed_record_id` FK `raw_records` NULL (the fragment every listed column came from); `first_seen` (= `created_at`), `last_seen_at` | one reader from the fragment, at ingest and in `backfill listed`; all listed columns and `listed_record_id` written as one unit, NULLs included | everything; never recomputed from our data; NULL where no fragment | | Standard quantity | `products` | `quantity_value` Numeric(10,3), `quantity_unit` String(8) in {ml, g, pcs}, `pack_count` Integer, `pack_unit_value` Numeric(10,3), `form` String(12) in {single, pack, set, refill}, `set_contents` String(120), `quantity_state` String(12) in {stated, none, unparsed}; `name_key` String(400) (the `_flat` fold of the name, for SQL differs); `size_ml` stays, derived | ingest at create and enrich, `backfill quantities`, `rederive` | recomputable; `size_value/size_unit` is the stated pair where the name agreed with `size_ml`, else the collector's ml (`ingest.stated_size` falls back to `(size_ml, "ml")`; 7293 holds 75.00 ml for a 75 g stick), so it is our past parse, not the shop's word | | Standard variation | `products.attributes` | `variation` (canonical string, unchanged), **new** `variation_kind` in {concentration, color, flavor, age, cask, edition}; `variation_aliases.kind` (nullable) | rule, `backfill variations` | recomputable | | Decided | `overrides` (exists) | entity_type `product`, entity_key str(product_id), field in {name, line_id, variation, quantity}; who, when, reason | the desk, `merge_session.confirm`, one new route | **every rekey and rederive reads it first** | | Decided, listing level | `listings` | `pinned_product_id` FK products NULL, `pinned_by` FK accounts NULL, `pinned_at`; `ignored_at`, `ignored_by` FK accounts NULL, `ignore_reason` | four new routes | ingest honours the pin; `catalog_queries` and the collectors page skip ignored (the other readers are a filed `do`, LT3 e) | **The key, identity rules v4.** `house|line|variation|[|form]`, spelled in one place, `quantity.quantity_key(q)`: **every dimension spelled the same way, value then unit** (rian, 15 Sep: no bare numbers, one rule for every unit of measure): `100ml`, `50g`, `200pcs`. The ml figure is computed through `normalize.size_ml_of(value, unit)`, the same Python `round()` v3 used (banker's rounding): 17911 "92.5 ml" is `92ml`, 17500 "37.5 ml" is `38ml`, 15772 "11.5 ml" is `12ml`, 1583 "0.375L" is `375ml`; `quantity_value` keeps the exact figure (92.500). A pack keys on its **total** in its dimension with form `pack` (`chivas-regal|12||2000|pack` becomes `chivas-regal|12||2000ml|pack`; `24 x 20g` is `480g|pack`). So every product re-keys once under v4, in one `rederive`, and the l_v3_keys snapshot in LT4 is what proves the grouping of millilitre products is unchanged (each v3 key maps to exactly one v4 key and back, except the sets and the unknowns below). **Sets:** every set, a one-part set included, keys on its unit-suffixed sorted contents with form `set` (`5ml+15ml+15ml+50ml|set`, `100ml|set`, mixed units allowed: `3.5ml+4g|set`); `quantity_value/unit` is the sum when all parts share a unit, NULL when mixed; `set_contents` always; `size_ml` is that sum for an all-ml set (the public page keeps a figure), NULL otherwise; a set with no part sizes keys `unknown|set` with `quantity_state` none (a bare piece count, "4 Pieces Mini Travel Set", goes to `quantity_value` 4 pcs but not into the slot). The pinned `rabanne|1-million|edt|50|set` (`test_keying.py:37`) becomes `rabanne|1-million|edt|50ml+100ml|set`; the 379 live products with a set key and a stated size are counted in LT4. `unknown` when no quantity was read (never a number, so it can never look like a measurement). 100 ml never equals 100 g. **Unknown never agrees with anything:** `merges.duplicate_groups` no longer auto-folds a group where any member's `quantity_state` is not `stated`; such groups become candidates with reason `quantity_unknown`. **Ounces:** a metric statement anywhere in the text always beats an imperial one (7965 "Pure Musc 3.3 Oz 5cl" stays 50 ml; 10730 "/2.7oz 8cl" stays 80); `fl oz` is always ml; a bare `oz` is fluid ounces (ml) for beauty Perfume and for liquor and grams for confectionery, Makeup and Skincare; the converted figure snaps to the nearest nominal metric size within four percent (1.7 fl oz is the 50 ml bottle, 12783 "3.4 Oz" perfume is 100 ml, a 1 oz cream is 30 g). Nine live names state ounces today; those are the pins. `IDENTITY_RULES_VERSION = "4"`. ## Working assumptions this lane proceeds under (rian's to overturn) - **Every key carries its unit** (`100ml`, `50g`, `200pcs`; `unknown` when none was read). Decided by rian on 15 Sep in place of the earlier draft's bare millilitres: one rule for every unit of measure, and one full rekey tonight rather than an inconsistency kept for convenience. The five pinned test files are rewritten with the unit; the l_v3_keys snapshot proves millilitre products group exactly as before. - **A group where any member does not state a quantity is never auto-folded**; it becomes a `quantity_unknown` candidate for a person. *If rian wants those folded as before:* one line in `merges.duplicate_groups` and the candidates are withdrawn by `suggest`. - **Shade is color.** One kind, `color`, American spelling in every identifier, column and alias row; "shade", "colour", "tint", "teinte" are shop words folded to it; the page may label it "Shade" for makeup. *If rian wants a separate shade kind:* one entry in `VARIATION_KINDS` and one alias row; no key changes. - **Kind is metadata, not a key slot.** The key's variation slot stays one canonical string; kind lives in `attributes.variation_kind` and `variation_aliases.kind`. *If rian wants kind in the key:* a v5 rule change and a full rederive, nothing lost. - **The stored form words are `single | pack | set | refill`** (`pack` is what the key already pins; the picker labels it "multipack"). *If rian wants `multipack` stored:* a backfill rename and the key token changes for 405 products. - **Listed columns are written by one reader from the fragment, at ingest and in the backfill; `RawListing` carries only `quantity` and `quantity_text`.** This deviates from item C of the thread (collector-filled `listed_*` fields) so that ingest and backfill are provably the same function and the four network collectors owned by Stream A stay untouched. *If rian wants collector-filled fields, as item C says:* add the six fields to `RawListing` and fill them per collector from the same places the tile reads; the reader then only fills what a collector left None. - **An AI proposal is the session's own reading of the data, written to a file rian approves from.** No external model call, no network. *If rian wants an LLM pass later:* the same `propose` command takes any file of the same shape. - **The public site keeps showing `size_ml`** tonight; gram and piece quantities reach the product page in a follow-on (`schemas.py`, `seo.fmt_size`, `format.ts` are not touched). *If rian wants it tonight:* one additive field on `ProductSummary` plus the two formatters; filed as a `do`. - **The Extime recollection is rian's, on the droplet, after this chain deploys**; this stream drafts the window file and the commands and runs no collector. *If rian wants it run from staging:* the fragments would be wiped by the next staging refresh, so the answer stays no until he says so. - **The four refused shops and Dubai are not recrawled by anyone until rian decides otherwise.** *If rian overturns Decision 3 or A16:* he runs the collectors on bwlive himself; nothing here changes. ## Tasks, in order (ids on /plan; record each with `main/scripts/plan-set.py`; commit prefix `L:`) 1. **LT1 The quantity parser, the v4 key, and variation kinds; pure code, no schema.** *What:* new pure module `services/quantity.py`: `Quantity(value, unit, pack_count, pack_unit_value, form, set_contents, state)`; `parse_quantity(text, *, hint=None, category=None) -> Quantity` understanding l/cl/ml as today; g/gr/gram/kg; oz and fl oz by the ounce rule in the model section (metric beats imperial; `fl oz` is ml; bare `oz` is ml for Perfume and liquor, g for confectionery, Makeup and Skincare; snap within four percent, else round); pcs/pc/pieces/sticks/tablets/capsules/ caps/bags; `N x ` multipacks in any dimension (total in the canonical unit plus `pack_count` and `pack_unit_value`); Extime's glued multiplier "X20 Pieces" is a count of 20 pcs, never a pack without a unit; sets by the existing `looks_like_set` with every part kept and sorted ascending, mixed units allowed; refills; `state` is `stated` when a quantity was read, `none` when the text names no figure, `unparsed` when a number-with-unit was seen but not understood. **Precedence, written once in the module docstring and tested:** (1) a decided `quantity` override; (2) a structured shop field (a collector's own unit: Heinemann `contentUnit`, Extime `capacity`, Changi `saleMeasureType`) when the text is silent or agrees in dimension; (3) the text when it names a different dimension than the hint, a set, or a multipack, and the disagreement is recorded (`state` stays `stated`, detail `hint_disagrees`); (4) the hint when the text is silent. The `(size_value, size_unit)` hint from a v3 row ranks BELOW the text: it is our past parse, not the shop's word, which is what makes 7293 (hint 75 ml) come out 75 g and 14386 (hint 35 ml) come out a set. The per-vertical ceilings move here from `SIZE_CEILING_ML` with `size_is_implausible` kept as a wrapper. `quantity_key(q) -> str` is the one place the slot is spelled (tokens in the model section above); its ml token goes through `normalize.size_ml_of`, never `int()` or half-up rounding. `normalize.match_key` gains keyword-only `quantity: Quantity | None = None`; when given it writes the v4 slot and the form from `q.form`, when absent it behaves exactly as v3 (so `test_merges.py`'s positional calls stay green). `lines._strip_measures` strips gram and piece packs and `form_of` recognises them through `parse_quantity`; both move line keys for names carrying a gram pack, which is why `backfill lines` runs before `rederive` (LT4). Variation kinds in `services/lines.py`: `VARIATION_KINDS = ("concentration", "color", "flavor", "age", "cask", "edition")`. **Two meanings, decoupled:** `BEAUTY_VERTICALS = {"beauty"}` keeps the line and display branch (`line_key`, `_display_from`: audience mapping, no age stripping, variation words removed) exactly where `VARIATION_VERTICALS` had it, so adding a confectionery rule never routes the 385 confectionery products through the beauty branch; `VARIATION_RULES: dict[(vertical, category | None), list]` is what `variation_of` consults. Rules: (beauty, None) `concentration` (the existing rules, unchanged); (beauty, "Makeup") `color`, built on the separator the Heinemann-family shops actually use: after `_strip_measures`, the text after the last " / " is the shade, canonical the folded text ("800 fairest nude", "110n", "02 cheeky pink"); no "No. NN" rule and no "word after colour" rule (they would tag "Brush N°12", "Chanel N°5" and read "Riche" off "Colour Riche"); (confectionery, None) `flavor`; `age`, `cask`, `edition` registered without rules (liquor keeps ages in the line; a rule there would re-key every whisky). Skincare's " / Grasa" and " / todo tipo de piel" tails (56 rows) are skin types, not shades, and the 384 uncategorised beauty names with " / " get no color: the gate is `category == "Makeup"`. So the category must reach the rule: `lines.variation_of`, the new `variation_kind_of` and `keying.Maps.variation` gain keyword-only `category: str | None = None`; `listing_key` passes the `classify(...)` result and `product_key` passes `product.category`. `variation_of` keeps its two-tuple return. `IDENTITY_RULES_VERSION = "4"` with the history comment in `models/catalog.py`. *Why first:* the migration's columns are shaped by code that already passes, and every later task calls these functions. *Files:* new `services/quantity.py`; `services/normalize.py`, `services/lines.py`, `services/keying.py` (`Maps.variation` signature only), `models/catalog.py` (the constant only); new `tests/test_quantity.py`; `tests/test_lines.py`; `tests/test_keying.py` (line 37, the set pin); `tests/test_house_style.py`: extend the walk from `web/src` to the Python files this stream creates (`quantity.py`, `overrides.py`, `collected.py`, `listings_table.py`, `merge_desk.py`) and add the check that `colour` and `flavour` appear in no identifier, column name, key token or alias row there or in `VARIATION_KINDS`. *Proof:* `tests/test_quantity.py` on the real names, each with its real staging hint: 7293 "Dior Sauvage Stick Deodorant 75g" with hint (75, "ml") -> (75, g), key slot `75g`, detail `hint_disagrees`; 14386 "Benefit Pop, Lock & Lash It 4g + 3.5ml" with hint (35, "ml") -> set, contents `3.5ml+4g`, form `set`; "Daywear Set 50ml + 2x15ml + 5ml" -> `5ml+15ml+15ml+50ml`; "Kérastase set 250ml + 200ml + 150ml" -> `150ml+200ml+250ml`; "24 x 330ml" -> total 7920, pack_count 24, `7920|pack`; "24 x 20g" -> `480g|pack`; "1.7 fl oz" -> (50, ml), slot `50ml`; "1 fl oz" -> (30, ml); a Skincare "1 oz" -> (30, g); 12783 "Amo Flowerful ... 3.4 Oz (women)" Perfume -> (100, ml); 7965/7966 "Pure Musc 3.3 Oz 5cl / 10cl" -> 50 / 100 ml; 9051/9052 "/1.7floz 5cl / 10cl" -> 50 / 100; 10729/10730 "/2.7oz 5cl / 8cl" -> 50 / 80 (metric beats imperial in every case); 17911 "92.5 ml" -> slot `92ml`, 17500 "37.5 ml" -> `38ml`, 15772 "11.5 ml" -> `12ml`, "0.375L" -> `375ml` (and the hint (0.38, "l") never becomes 380); "200pcs" -> `200pcs`; 15114 "Advanced Ceramide ... 90 caps" -> (90, pcs); "Biscuits X20 Pieces" -> (20, pcs), not a pack; "Chivas 12 Year 70cl" -> slot `700ml`, no variation. Color on staging names, category Makeup: 15139 "Joli Blush Blusher 6 gr / 02 Cheeky Pink" and 15329/15330 "SYNCHRO SKIN ... 30 ml / 220 Linen" and "... / 230 Alder" -> variations of kind `color`, canonical "02 cheeky pink", "220 linen", "230 alder"; negatives: "Colour Riche Lipstick 3.6 gr / 800 Fairest Nude" keeps line `colour-riche` and color "800 fairest nude"; 11106 "Powder Brush N°13" no color; 15910 "Purifying Gentle Foaming Cleanser 125ml / Grasa" (Skincare) no color; "Chanel N°5 EDP" no color; "1 Million Parfum 5cl" no color. "Dark Chocolate 70%" and "Milk Chocolate" -> kind `flavor`, and a confectionery line key pinned before and after in `test_lines.py` is unchanged. The ONE_MILLION fixture keeps its three concentrations. Every existing key string in `test_keying.py`, `test_merges.py`, `test_aliases.py`, `test_line_backfills.py` gains its unit (`rabanne|1-million|edt|100` becomes `rabanne|1-million|edt|100ml`, `chivas-regal|12||2000|pack` becomes `chivas-regal|12||2000ml|pack`), the set pin at `test_keying.py:37` becomes `rabanne|1-million|edt|50ml+100ml|set` by the set rule, and the rewritten pins are the only test change: a pin whose house, line or variation part moves is a bug. *Done when:* those tests pass, `check.sh` green, committed. 2. **LT2 One migration, one head, rehearsed.** *What:* the single schema change of this stream, revision **`b3c4d5e6f7a8`** (confirmed unused: `ls main/alembic/versions | grep '^b3c4'` prints nothing), file `main/alembic/versions/b3c4d5e6f7a8_quantity_listed_decided.py`, `down_revision = "a2b3c4d5e6f7"`, schema only, every column nullable or `server_default`. `alembic heads` must print exactly `a2b3c4d5e6f7` immediately before writing and again before committing. Columns: on `products` the quantity columns and `name_key` (table above); on `listings` the listed columns, the two folds, the listed quantity pair, `listed_record_id FK raw_records.id NULL`, `last_seen_at DateTime(timezone=True)`, and the pin and ignore columns (who columns `FK accounts.id NULL`, plan §2); on `variation_aliases` `kind String(16)`. Indexes: `listings(listed_brand_key)`, `listings(listed_name_key)`, `listings(listed_gtin)`, `listings(listed_record_id)`, `listings(last_seen_at)`, `listings(pinned_product_id)`, `products(quantity_unit, quantity_value)`, `products(name_key)`. No new tables; `overrides` and `merge_candidates` are reused as they are. *Why here:* every later task writes to these columns, and a second head or a data-moving migration is the one mistake rian cannot undo at deploy time. *Files:* the migration; `tests/test_accounts_backfills.py` (head assertion becomes `["b3c4d5e6f7a8"]`, chain dict gains `b3c4d5e6f7a8 -> a2b3c4d5e6f7`); `models/catalog.py` (the ORM columns, with the `catalog.py:295` comment corrected to say what is actually stored); `docs/DATA-MODEL.md` (the model table above as prose, the correction that grams were never stored before v4) and `scripts/docmap.py --write` for the generated tables block. *Proof:* `docker exec dfp-devdb psql -U dfp -d postgres -c 'drop database if exists dfp_l' -c 'create database dfp_l'`; `pg_restore --no-owner --no-privileges` of `/srv/apps/dutyfreeprofessor/backups/dfp-nightly-2026-09-14.dump` into it; with `DATABASE_URL=postgresql+psycopg://dfp:dfp@127.0.0.1:5433/dfp_l` run `alembic upgrade head`, `downgrade -1`, `upgrade head`; `alembic heads` prints one head; the head-pin test passes. *Done when:* up/down/up is clean on `dfp_l`, `check.sh` green, committed with the migration file, the ORM and the pin in the same commit. 3. **LT3 Decided beats rule: the override reader, the tombstone follow, pin and ignore.** *What:* (a) new `services/overrides.py`: `read(db, entity_type, keys, fields)` and `set(db, entity_type, key, field, value, *, set_by, reason, collected_value)` (upsert on the unique triple, savepoint and re-read on IntegrityError, agents.md race rule). (b) `keying.Maps` gains `decided`: the product overrides for `name`, `line_id`, `variation`, `quantity` (value `{value, unit}`), loaded in `load_maps`, cached and invalidated like the rest; `product_key` applies `line_id`, `variation` and `quantity` overrides before composing the key. **A name override is display only for the key:** today the name IS the key input (`product_key` derives the variation from `product.name`, `match_key` derives `form_of(name)`, `backfill lines` recomputes `line_key(product.name)` and reassigns `line_id`), so keying from a typed name reproduces the failure (c) exists to prevent (a survivor renamed "1 Million" loses `edt`, keys `rabanne|1-million||100ml`, and the next collection creates a new product). When a `name` override exists, every name-derived key part (line key, variation, form, quantity text) is computed from the override row's `collected_value` (the spelling the listings arrive under) and `value` is what the page shows; `backfill lines` reads the same collected spelling. `rederive`, `backfill lines` and `merges._ENRICH_FIELDS` never write over a field that has an override (`name` included). (c) `merge_session.confirm` with a typed product name that is neither side's collected spelling writes a `name` override (`set_by` from `identity.actor`, reason "chosen at merge", `collected_value` the survivor's collected spelling) so `rederive` cannot re-key the survivor away from its listings; picking a collected spelling writes none. (d) `ingest._resolve_product`: when no live row matches the key, look for a tombstone with that key and follow `survivor_of`, so a human product merge holds across the next collection. Tombstones keep the key of the rules that merged them (`cmd_rederive` rekeys only `merged_into_id IS NULL`; on staging all 606 sit at v2/v3 and 49 already differ from their survivor's key), so LT4's `rederive` and `rekey_products` also recompute `match_key` and `identity_rules_version` on merged rows, keeping `merged_into_id`. The follow accepts a tombstone only when its `identity_rules_version == IDENTITY_RULES_VERSION`, its `product_merges.reason` is a human one (`confirmed`), its quantity slot is not `unknown` (unknown never agrees with known), and its survivor holds no barcode different from the listing's; when several tombstones share the key (557 of 606 share a key with a live row today) follow the newest `product_merges` row. (e) `ingest._resolve_listing`: a listing with `pinned_product_id` is pinned to `survivor_of(pinned)` every sighting and never re-pinned over; a listing with `ignored_at` keeps collecting prices but is excluded through one helper `catalog_queries.live_listings()` (`Listing.ignored_at IS NULL`) used by the site's price readers, brand pages and comparables in `catalog_queries.py`, `collector_view.products/grouped/listings` (default; a filter shows them), `suggest._facts`, `merge_session._shops_of`; list the call sites in the commit body. **Tonight ignore is honoured in `catalog_queries.py` and the collectors page only.** `services/trip.py` (the two Listing selects), `routers/catalog.py` (`stats`, the multi-airport filter), `services/indexnow.py`, `services/coverage.py` and `services/audit.py` query `Listing` directly and are on the Must-not-touch list, so an ignored listing still reaches the trip comparison, the home page counts and IndexNow pings. Say so in the ignore route's docstring and the handoff, and file one `do` (owner: the trip/SEO lane) naming those five readers to adopt `live_listings()`. (f) `merges.merge_products` moves override rows from the loser to the survivor where the survivor has none for that field and records the rest in `detail`. (g) Routes, all under `sources.manage` in `access.py`, who-columns from `identity.actor(request)` never the payload, every write audit-logged: `POST /api/collectors/products/{id}/override` {field, value, reason}; `POST /api/collectors/listings/{id}/pin` {product_id, reason}; `.../ignore` {reason}; `.../unpin`; `.../unignore`. *Why here:* the next task rekeys 17k products and the desk later produces decisions in volume; nothing may be decided before it is proven to survive. *Files:* new `services/overrides.py`; `services/keying.py`, `services/ingest.py` (the two resolve functions only), `services/merges.py`, `services/merge_session.py` (`confirm`, `_shops_of`), `services/catalog_queries.py` (the helper and its call sites only), `services/collector_view.py`, `services/suggest.py` (`_facts`), `routers/collectors.py`, `services/access.py` (append), `cli.py` (`rederive`, `backfill lines` respect overrides); new `tests/test_decided.py`; `docs/DATA-MODEL.md` provenance paragraph: rule (`identity_rules_version`), merge (`decided_by/at`), override (`set_by/at/reason`). *Proof:* on the SQLite kit with the ONE_MILLION fixture: an override on `variation` and one on `name` survive `rederive`, `rekey_products` AND `backfill lines` (name and key unchanged), where the name override renames the survivor to a spelling that drops the concentration ("1 Million"); the 1 Million 10 cl case (a bare tile keyed `rabanne|1-million||100ml` after a human `confirmed` merge of that key into the EDT survivor) resolves to the survivor, not a new product; a `confirmed` tombstone at a v3 key resolves after `rederive` has moved it to v4; a tombstone at a stale rules version is ignored; a tombstone keyed `unknown` is never followed; two tombstones on one key follow the newest merge; a pinned listing stays pinned after `_resolve_listing` is run with a different product; an ignored listing is absent from `catalog_queries` prices and present with the filter on `/collectors`; a merge moves the loser's override; a typed name at confirm leaves an override row set by rian; routes 401/403/200; `test_route_inventory` green. *Done when:* those tests pass, `check.sh` green, committed. 4. **LT4 Quantity and kinds through the pipeline, and the rehearsed rederive.** *What:* `RawListing` gains `quantity: Quantity | None` and `quantity_text: str | None` (the shop's own size wording); `collectors/base.py` fills `quantity` from the name when a collector leaves it None; the structured readers that already see a unit fill it directly: heinemann and heinemann_platform `contentUnit` by a named code table (the fixtures show only `L` and staging holds zero heinemann fragments, so the codes come from the spec, not from data): `MLT`/`ML`/`CLT`/`LTR`/`L` -> ml as today, `GRM`/`G`/`KGM`/`KG` -> g, `PCE`/`PC`/`ST`/`EA` -> pcs; any other code yields `quantity_state` `unparsed` with the code kept in `quantity_text`, never a silent None; extime `capacity`/ `capacity_unit`, and `net_weight`/`net_weight_unit` ONLY when `capacity` is absent and neither `variation.name` nor the product name states a quantity (on staging `net_weight` is a packaged weight: 100 ml pairs with 86 g, 50 ml with 374 g, 1 l with 924 g; the 44 fragments with no capacity look like content weights, macarons 72 g, but include "Collection set 295 g"), stamped `attributes.quantity_source = "extime.net_weight"` so a wrong gram is traceable; changi `saleMeasureType` Weight as g as well as Volume, shilla `Weight,Volume` in grams; avolta, shopify, ari and dubai stay name-parsed and are not edited (Shopify `variant.grams` is shipping weight and is never used). `size_ml` stays as `quantity.ml` (None for non-volume) so `_sizes_disagree`, `verify`, the Avolta and Extime `::` SKUs and the public API are untouched; ingest gains `_quantities_disagree` comparing like with like (ml with ml, g with g, None never vetoes) and uses it for the GTIN veto; `verify.compare` is not edited (it keeps `_sizes_disagree`, which still holds for volume). `ingest._resolve_product` computes the product's quantity through `parse_quantity(raw.name, hint=raw.quantity)`, passes it to `keying.listing_key`, writes the quantity columns and `name_key` at create, enriches an existing product whose `quantity_state` is not `stated`; `keying.product_key` passes the product's stored quantity (override first) so `rederive`, `rekey_products` and `implausible_sizes` need nothing more. `merges._ENRICH_FIELDS` gains the quantity columns; `merges.duplicate_groups` never auto-folds a group with a non-`stated` member and records it with reason `quantity_unknown` (added to `PRODUCT_REASONS` with a readable why); `suggest` groups unknown-variation pairs by the quantity tuple, not `size_ml`. Kinds: `Maps.variation` and `apply_variation_alias` carry `kind`; `backfill variations` seeds `kind` on the 41 rows, stamps `attributes.variation_kind`, and seeds the color aliases from Makeup rows only and the confectionery flavor wordings. `backfill quantities` (new, in `BACKFILLS`): every live product, `parse_quantity(name, hint=(size_value, size_unit), category=product.category)` under the LT1 precedence (the text beats this hint; the hint fills only a silent name), writes the seven quantity columns and `name_key`, sets `size_ml` where the dimension is ml and it was NULL, and **clears `size_ml` (and the derived `size_value/size_unit`) only where the parsed dimension is g or pcs and the stored `size_ml` equals the same figure read as ml** (the misread signature: 7293 stored 75 ml for "75g"), reporting that count; otherwise never clears a `size_ml`; never touches a product with a `quantity` override; second run reports 0. `rederive` and `rekey_products` rekey merged rows too (LT3 d), so a tombstone's key always reflects the current rules. `backfill merges --check` (append inside `backfill_merges`, `cli.py` is shared) prints each mergeable group (key, member ids, names, gtins) and each candidate group with its conflicts, not only the count line it prints today, so the group list the Rules ask you to read exists. `collector_view` facets, sorts and the `size_ml` filter read the quantity; the `' ml'` hardcoded at `CollectorsPage.tsx:1018` goes. *Why before the desk:* H, and the readers' finding that a rekey after decisions is the costliest reversal in the system. If this task passes three hours, split it at "backfill and rehearsal" and record the split on /plan with a note. *Files:* `collectors/base.py`, `collectors/heinemann.py`, `collectors/heinemann_platform.py`, `collectors/extime.py`, `collectors/changi.py`, `collectors/shilla.py` (the quantity reads only, nothing that touches the network, robots or pacing), `services/ingest.py` (veto, create and enrich writes), `services/keying.py`, `services/merges.py`, `services/suggest.py`, `services/lines.py`, `cli.py` (`backfill quantities`, `backfill variations`), `services/collector_view.py`, `pages/CollectorsPage.tsx`; tests: `test_keying.py` gains the v4 strings (`...|50g`, `...|200pcs`, `...|5ml+15ml+15ml+50ml|set`, `chivas-regal|12||2000ml|pack`); `test_collectors_avolta.py`, `_changi.py`, `_extime.py` and `_shilla.py` gain the quantity assertion on their existing fixtures (the Shilla "10G is a weight" test now expects `quantity (10, g)` and `size_ml None`; the Changi Weight test likewise; Extime pins that the 100 ml / 86 g fragment stays (100, ml) and never becomes grams); there is no heinemann test file today, so **new `tests/test_collectors_heinemann.py`** covers `heinemann.py` and `heinemann_platform.py` with an inline `contentUnit` item per code in the table above, asserting `quantity` and `size_ml`; `test_ingest_guards.py` pins the dimension-aware veto; `test_listings_view.py` tiles gain the quantity; `test_line_backfills.py` summary strings; `docs/COLLECTORS.md` names the fields. *Proof:* on `dfp_l`, in exactly the after-deploy order so the rehearsal produces the deploy's numbers (`product_key` reads the stored `line_id`, so a rehearsal without `backfill lines` gives different join and split counts): first `CREATE TABLE l_v3_keys AS SELECT id, match_key, merged_into_id FROM products`; then `backfill quantities --check`, `quantities` twice (second run zero), `variations` twice, `lines` twice, `rederive`, `backfill merges --check` and READ the printed group list, `merges`. Report from that database: products by `quantity_unit` and `quantity_state`; products with a stated quantity before and after by vertical (confectionery 16 of 385 and makeup 264 of 774 today are the numbers to beat); how many gram names were carrying a wrong ml and how many `size_ml` the backfill cleared (7293 now `75g`; 14386 now a set); the `lines` count (lines created, products pointed); how many `unknown` groups became `quantity_unknown` candidates; joins and splits by SQL against `l_v3_keys` (joins: v4 keys with more than one distinct v3 key; splits: v3 keys with more than one distinct v4 key; under v4 every key changes, so the numbers that matter are these two: joins expected 0 for millilitre products and splits expected 0 for millilitre products, each exception listed); live products whose GROUPING changed, broken down as gram/piece rows, set rows (expect the 379), confectionery rows (expect 0), and any millilitre row whose group moved (expect 0; list each one); tombstone keys changed; the number-followed-by-word tokens that fell to state `none`, so the count vocabulary grows from data; and that the two named sets key on their contents. *Done when:* those numbers are in the task note, every millilitre product groups exactly as under v3 (the snapshot proves it), `check.sh` green, committed. 5. **LT5 The listed columns, written at ingest and backfilled from the newest fragment.** *What:* new leaf module `services/collected.py` (imports only `normalize`, `quantity`, `taxonomy` helpers and `collectors.changi.channel_price`): `collected_tile` and `_platform` move here from `collector_view` (which re-imports them so `test_listings_view.py` keeps passing), plus `listed_fields(parser_version, payload) -> dict` returning `listed_brand`, `listed_name`, `listed_variant`, `listed_quantity_text`, `listed_category`, `listed_gtin`, `listed_brand_key` (slug-shaped: `normalize.brand_key(x).replace(" ", "-")`, the same fold `ingest.brand_slug` and `Maps.house` use, so it compares with `brands.slug`; `brand_key` alone keeps spaces and would flag every two-word house), `listed_name_key` (the `_flat` fold), `listed_quantity_value/unit` (`parse_quantity(listed_quantity_text or listed_name)`) per platform from the fragment only, never from `RawListing.name`: avolta `tile.name`/`tile.brand`, category from `category_url`; shopify `product.title`, `vendor`, `variant.title` unless Default Title, `product_type`, no barcode (the public feed has none); ari `jsonld.name`, `brand.name`, `category`, and no barcode either (all 1,395 ARI fragments carry only `@context, @type, brand, category, mpn, name, offers, sku`; read `sku`/`mpn` through `gtin_from_sku` only when it validates as a GTIN, and say which in COLLECTORS.md); extime `variation.product_name`, `variation.name` as the size text (else `capacity capacity_unit`), `main_offer.categories_name` joined, `main_offer.gtin` (the family barcode on `::size` rows; stored as listed, **never fed to `Product.gtin` or the veto**); shilla `brand`, `name`, `specs['Weight,Volume']`, `categories`, `ref_no`; changi top-level `name`, `brandName`, breadcrumb reversed, `level1saleMeasure`; dubai `displayName`, `brand`, `x_gTIN` (category ids only, so `listed_category` NULL); heinemann `name`, `brand`/`manufacturerName`, `contentUnit`, `typeOfGoods`, `gtin`/`ean`. Every text is `html.unescape`d the way ingest does, never trimmed or cased. The reader's output is written as one unit: every listed column plus `listed_record_id` from the current fragment, NULLs included, so a newer fragment that lacks a field (a payload shape change, a Shopify variant without an option) nulls it rather than leaving an older sighting's value beside new ones; one listing's listed columns never mix two fragments. `ingest.run_collector` calls it right after `_resolve_listing`, where the `RawRecord` is written, and stamps `last_seen_at` there (a quarantined sighting still refreshes the listed columns; that is stated in the docstring). `backfill listed` (new): newest fragment per listing by `max(id)` per `listing_id` through a portable subquery (not `DISTINCT ON`, so the SQLite test runs it), read slim (`payload - 'tile_html'` on Postgres), same reader, sets `listed_record_id` to that fragment id, leaves every column NULL where no fragment exists, also fills `last_seen_at` from the newest of `price_observations.observed_at` and `raw_records.created_at` where NULL; second run writes zero. *Why:* the giant table sorts on these; a backfill that copied Product fields into them would label our text as the shop's, so NULL is the honest value for the 12,673 listings without a fragment. *Files:* new `services/collected.py`; `services/collector_view.py`, `services/ingest.py` (the hook), `cli.py` (`backfill listed`), `docs/COLLECTORS.md` (one section, the reader per platform); new `tests/test_listed.py` (the eight platform fixtures from `test_listings_view.py`, the ingest hook on the SQLite kit, the backfill twice, a newer fragment lacking a field nulls it, and a two-word house whose `listed_brand_key` equals `brands.slug`). *Proof:* on `dfp_l`: `backfill listed --check`, then twice; report per location the count of listings with `listed_name` (expect 11,098 filled and 12,673 NULL: near-complete for BOG, PTY, YUL, SAL, DUB, LHR, ZRH, BCN, JFK, MAD, ATH, EZE, YYZ, HKG, MEX, ICN, SIN; 200 for CDG; zero for DXB, HEINEMANN-GLOBAL, KEF, SYD, SCA), the NULLs stated as "no fragment", not "empty". *Done when:* the reader is one function used by both writers, the numbers are in the note, `check.sh` green, committed. 6. **LT6 The giant Listings table, server and browser.** *What:* new `services/listings_table.py` (`collector_view.listings` delegates to it; the other tabs are untouched). One SQL statement, joined once, returning every field in groups: **listing** (id, airport, location, collector, source_sku, url, first_seen, last_seen, pinned, ignored), **listed** (the six columns, the listed quantity, the fragment's seen_at and parser), **standard** (product id, name, quantity value/unit/pack/form/set_contents/state, size_ml, gtin, gtin_source, category, vertical, rules version), **house/brand** (collected brand, house name, slug, canonical), **line** (name, slug, canonical, product count), **variation** (canonical, display, kind), **shop** (retailer, location name, currency, country, visible), **collector** (slug, platform, parser version, last run status), **price** (latest observation, was, currency, usd, fx rate, observed_at, in_stock), **provenance** (rule | merge | override per standard field, from `identity_rules_version`, `product_merges` and `overrides`), **differs** (brand, name, quantity). Sort and filter in SQL on any column of the first nine groups (`sort=&dir=asc|desc`; every listed and standard column a filter: `listed_brand`, `quantity_unit`, `quantity_state`, `form`, `variation_kind`, `has_fragment` (`listed_record_id IS NOT NULL`), `ignored`, `pinned`, `differs`); `differs=brand` is `listed_brand_key <> brands.slug` (canonical house; both slug-shaped, so "Paco Rabanne" listed under the house `paco-rabanne` does not differ, while the eight slugs ending in `tm`, `bulleittm`, `crown-royaltm`, legitimately do), `differs=name` is `listed_name_key <> products.name_key`, `differs=quantity` compares the stored listed pair with the product's, all as WHERE clauses; the fragment's seen_at and parser join through `listed_record_id`; the old differs-in-Python path is deleted. `columns` (comma list or `all`), `per_page` up to 500, `total` and `with_fragment` counts from the database on every request. `format=csv` on the same route streams every matching row in id batches of 1,000 (a `StreamingResponse`, the chosen columns flattened, filename `listings-.csv`; no new path, so `access.py` is unchanged). Browser: new `web/src/components/collectors/ListingsTable.tsx` with its stylesheet imported in the component, replacing `ListingsTab` in `CollectorsPage.tsx`: a column picker with the groups, per-column ticks, group ticks, **All on** and **Reset**, persisted per browser in `localStorage` key `dfp.listings.columns` (in try/catch); a sticky header, horizontal scroll, click a header to sort and again to flip; the filter bar; page size 50/200/500; the two counts from the response; a same-origin `` to the CSV with the current filters and columns (the cookie session rides on it); per row the actions from LT3 (Pin to product by id with a reason; Ignore with a reason; Unpin; Unignore); a diff tint per differing cell. **Listed cells are always rendered exactly as stored, never trimmed, cased or unescaped differently from ingest.** Owner-only as today. *Why:* this is the surface rian opens at the end; every earlier task exists so its columns are real and sortable. *Files:* new `services/listings_table.py`; `services/collector_view.py`, `routers/collectors.py`, `pages/CollectorsPage.tsx`, new `web/src/components/collectors/ListingsTable.tsx` and `.css`; new `tests/test_listings_table.py` (the column registry: every column named once, every group non-empty, `all` returns them all, the picker's ids match the API's field names; every sort column sorts in SQL, asserted by no per-row fragment read; the filter conditions, with `differs=brand` empty for a two-word house whose shop spelling folds to it and non-empty for another house; the CSV writer produces one header and `total` rows). *Proof:* against `dfp_l` through the RUNBOOK dev loop (`DATABASE_URL` at 5433, local `uvicorn app.main:app`): `EXPLAIN ANALYZE` of the unfiltered first page with every column on under one second; `GET /api/collectors/listings?sort=listed_brand&dir=asc&per_page=500` answers on the full 23,771 rows; the CSV for `airport=BOG` has 2,491 rows, read from the database; `npx tsc -b` clean; `test_house_style.py` green; every column in the picker renders without "undefined". *Done when:* the timing and the row counts are in the note, `check.sh` green, committed. 7. **LT7 The merge desk: tables, batch approve, proposals that rules never withdraw, and the file rian approves.** *What:* (a) new `services/merge_desk.py`: `GET /api/collectors/desk/brands` (one row per house: name, canonical, lines, products, listings, best undecided pair with other side, score, reason, readable why, candidate_id; filters text, has_suggestion, min_score; sort in SQL) and `GET /api/collectors/desk/lines` (one row per line within a house, same shape, plus the product-level pairs under it; filters house, vertical, kind, reason, min_score). (b) `POST /api/collectors/merge/batch` `{confirm: [{candidate_id, preferred_name?, note?}], reject: [{candidate_id, note?}]}` up to 200. `merge_session.confirm` and `reject` have no such switch today: `confirm` calls `suggest.generate` and `db.commit()` unconditionally for brand and line pairs and `reject` commits too. LT7 adds keyword-only `regenerate: bool = True` and `commit: bool = True` to both, defaulting to today's behaviour so the keyboard queue is unchanged; the batch applies each pair in id order with `regenerate=False, commit=False`, refuses per pair (barcode veto `NOTE_REQUIRED`, `LINE_HOUSES_DIFFER`, `PAIR_DECIDED` stay; the rest proceed), regenerates suggestions **once** at the end, commits once, invalidates keying once, records `decided_by`/`decided_at` per pair as today, one audit row per decision. (c) `POST /api/collectors/merge/propose` and the CLI `python -m app.cli propose --file [--check]`: inserts candidates `{level, left_id, right_id, score, why}` with reason **`proposed`** (outside `RULE_REASONS`, so `suggest.generate` never withdraws or rescores them; `PRODUCT_REASONS` renders the why) and `detail.proposed_by`, `left_id < right_id`, skipping a pair already present or decided, never merging anything. (d) The MergeTab gains a "Desk" view: the two tables, a tick per row, "tick all at or above score", "tick all proposed", Approve ticked, Keep apart ticked, the barcode veto asking for a note inline, decided rows greyed with who and when, decided-today and remaining counts from the database; the keyboard queue stays as it is. Every write under `sources.manage` in `access.py`. (e) **The proposal file rian approves in the morning**, written by the session itself (never a delegated model, never a rule) after reading both sides in `dfp_l`: `import/proposals/2026-09-15-brands.json` (at most 150 brand pairs: same house spelled two ways, a diacritic, a legal suffix, a house within a group) and `import/proposals/2026-09-15-lines.json` (the lines pairs it agrees with at score 0.8 and below, at most 300), each entry with a one-line why a person can check without the code, barcode-vetoed pairs excluded, nothing applied. `import/` is mounted read-only at `/srv/import`, so the after-deploy `propose --file` reads it in place. If the night runs long the files are the first cut: the desk still works on the rule candidates and `propose` stays usable for a later file. *Why after LT4:* decisions are made against keys as they stand (H); the batch route retires the O(N) regeneration, and the `proposed` reason retires the withdrawal trap. *Files:* new `services/merge_desk.py`; `services/merge_session.py`, `services/suggest.py` (`PRODUCT_REASONS`, `proposed`), `routers/collectors.py`, `services/access.py` (append), `cli.py` (`propose`), `pages/CollectorsPage.tsx`, new `web/src/components/collectors/MergeDesk.tsx` and `.css`; `import/proposals/*.json`; `docs/RUNBOOK.md` (the `propose` command, the desk paragraph); new `tests/test_merge_desk.py`. *Proof:* tests: the brands table row count equals the database; a batch of three confirms and two rejects regenerates once (count the calls) and records rian on each; a `proposed` pair survives `generate`; a barcode-veto pair without a note is reported and the others still apply; a decided pair is refused; 401/403 without `sources.manage`. On `dfp_l`: `desk/brands` returns 2,365 rows with the 27 rule candidates attached and sorts by score in SQL; `propose --file ... --check` on both files reports the insert counts and inserts nothing; the JSON validates against the route's schema. *Done when:* tests pass, both files are committed with their counts in the note, `check.sh` green, committed. 8. **LT8 The recrawl, told honestly, and the collectors page telling the truth about Dubai.** *What:* no collector runs in this stream. The four Heinemann-family shops are refused by their robots.txt and the code refuses them before the first request (runs 52 to 55, 23 Aug); Dubai refuses our declared identity (check 396, 11 Sep); policy says a host that refused is never contacted again, and A16 and Decision 3 own the asks. Write that, with the run ids and dates, into the handoff and into `docs/COLLECTORS.md`'s posture table if the five are not there. For Extime, the shop that has not refused, draft `.logs/runs/window-2026-09-15-extime-fragments.md` (says "not started"; host bwlive; the exact commands in order: `ssh deploy@bwlive`, `docker exec dutyfreeprofessor-app python -m app.cli collect --source extime-paris 2>&1 | tee -a .logs/runs/extime-fragments-2026-09-15.log` (three to five hours at the 1.5 s floor), then `verify --n 20 --mode after_collection --source extime-paris`; no deploy inside the window; then on mosiah `python3 main/scripts/staging-refresh.py --from backups/production/dfp-nightly-.dump`, then `backfill listed` and `suggest` on staging). Running list: one `do` for rian with `--link` to the window file; one `do` for the partnership asks naming A16 and Decision 3; one `decide`, weight `info`, no `--blocks`: the four Heinemann-family shops and Dubai stay uncrawled, frozen at 22 Aug prices (hidden or dated); options: the partnership asks at Cannes, or accept the frozen data; assumption: frozen. Make the collectors page tell the truth: `collector_view._story` reads the newest BLOCKED verification check when a source has no run since it and words it "refused since " instead of "nothing attempted for N days". Add the "Recrawl after L" paragraph to `RUNBOOK.md`. *Why:* a recrawl that gets us refused is the one risk this stream retires by not acting. *Files:* the window file; `services/collector_view.py` (`_story`); `tests/test_collector_view.py` (the story pin); `docs/COLLECTORS.md` (only if the posture table lacks the five); `docs/RUNBOOK.md`; the running list through `items.py`. *Proof:* `items.py list --kind do` shows the two items with links and `list --decisions` the decide; the window file says "not started" and names no command a stream may run; the story test is green. *Done when:* filed, tests pass, `check.sh` green, committed. ## Closing, after LT8 (no task id; the command's done conditions 4 to 6) `CHANGELOG.md` Unreleased lines for everything a user or the client would notice (no version bump); `scripts/docmap.py --write` and `bash main/docs-check.sh --strict` clean; `RUNBOOK.md` carries the after-deploy block below under migration `b3c4d5e6f7a8`; drop `dfp_l` (`docker exec dfp-devdb psql -U dfp -d postgres -c 'drop database dfp_l'`); stop the local API and remove any `main/static` symlink; the handoff (25 lines: numbers from `dfp_l`, the migration id, the commands in order, the Extime window and the three running-list items, that ignore reaches `catalog_queries` and the collectors page only tonight, how many gram products still publish an ml until the public-page follow-on, what is blocked, what rian must do) prepended to `.logs/handoff.md`; then `/checkpoint`. **Cut first if the night runs long, in this order, each named in the task note** (and at turn 120 with LT6 not started, cut (1) and (2) at once and say so in the LT7 note): (1) the proposal files in LT7 (keep `propose`, the `proposed` reason and the desk; rian approves rule suggestions only); (2) the CSV export in LT6 (leave `format=csv` refused with a clear message and a `do` item); (3) the brands table in LT7 (keep the lines table and the batch route); (4) LT1's `flavor` rule (color stays, since makeup is the reason kinds exist); (5) the row actions in LT6 (the LT3 routes stay, callable later). Never cut LT1, LT2, LT3's override reader and tombstone follow, LT4, LT5, the column picker and SQL sorts in LT6, the batch route in LT7, or LT8. **Not in this stream, filed as follow-ons:** gram and piece quantities on the public product page and in JSON-LD (`schemas.py`, `seo.fmt_size`, `format.ts`); a variation-alias level in the merge desk; an undo for a confirmed merge; the refusal ledger table (issue owned by Q); the scheduler (V4). ## After deploy (rian, in this order, each safe to repeat; the handoff repeats it) ``` cd /srv/apps/dutyfreeprofessor && ls -lt .logs/runs/ | head && pgrep -af app.cli # nothing collecting docker exec dutyfreeprofessor-db pg_dump -U dfp -Fc dfp > backups/dfp-$(date +%F)-pre-L.dump # bump main/app/version.py and rename "## Unreleased" in main/CHANGELOG.md to that version (rian; check.sh's doc gate reads both) srv-gw deploy --project dutyfreeprofessor --build # runs alembic upgrade head -> b3c4d5e6f7a8 docker exec dutyfreeprofessor-app alembic current # must print b3c4d5e6f7a8 (head); stop if not docker exec dutyfreeprofessor-app python -m app.cli backfill quantities --check docker exec dutyfreeprofessor-app python -m app.cli backfill quantities docker exec dutyfreeprofessor-app python -m app.cli backfill variations docker exec dutyfreeprofessor-app python -m app.cli backfill lines docker exec dutyfreeprofessor-app python -m app.cli rederive docker exec dutyfreeprofessor-app python -m app.cli backfill merges --check # read the groups first docker exec dutyfreeprofessor-app python -m app.cli backfill merges docker exec dutyfreeprofessor-app python -m app.cli backfill listed --check docker exec dutyfreeprofessor-app python -m app.cli backfill listed docker exec dutyfreeprofessor-app python -m app.cli suggest docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-brands.json --check docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-brands.json docker exec dutyfreeprofessor-app python -m app.cli propose --file /srv/import/proposals/2026-09-15-lines.json curl -s https://dutyfreeprofessor.demoing.info/api/health && srv-gw security-audit ``` No collection in progress while these run. Then open `/collectors`, Listings, and turn everything on; then Merge, Desk. The Extime window on bwlive, the staging refresh and `backfill listed` once more are LT8's `do`. ## Before the run (planning session, never the stream) Task ids are LT1 to LT8 because `L1` to `L11` already belong to Stream R in `import/progress.json` (`plan-set.py --add` refuses an existing id, and `LT1 doing` on the old ids would have flipped Stream R's finished rows). Verify before registering, then commit the brief and the command file to master (both are untracked today, so a fresh checkout or worktree would not carry them): ``` python3 -c "import json;d=json.load(open('import/progress.json'));print([t['id'] for s in d['streams'] for t in s['tasks'] if t['id'].startswith('LT')])" # must print [] git add .claude/commands/stream-l.md .logs/planning/streams/L-listings.md && git commit -m "Planning: Stream L brief and command" python3 main/scripts/plan-set.py LT1 todo "Planning 14 Sep" --add --stream L --stream-title "L · Listings, quantity and the merge desk" --session /stream-l --wave w3 --due 2026-09-15 --title "The quantity parser, the v4 key and variation kinds; pure code" python3 main/scripts/plan-set.py LT2 todo "Planning 14 Sep" --add --stream L --wave w3 --due 2026-09-15 --title "One migration b3c4d5e6f7a8, one head, rehearsed up/down/up" python3 main/scripts/plan-set.py LT3 todo "Planning 14 Sep" --add --stream L --wave w3 --due 2026-09-15 --title "Decided beats rule: override reader in keying, tombstone follow, pin and ignore" python3 main/scripts/plan-set.py LT4 todo "Planning 14 Sep" --add --stream L --wave w3 --due 2026-09-15 --title "Quantity and kinds through collectors, ingest and backfill quantities; rederive rehearsed" python3 main/scripts/plan-set.py LT5 todo "Planning 14 Sep" --add --stream L --wave w3 --due 2026-09-15 --title "Listed columns at ingest and backfill listed from the newest fragment" python3 main/scripts/plan-set.py LT6 todo "Planning 14 Sep" --add --stream L --wave w3 --due 2026-09-15 --title "The giant Listings table with the column picker, SQL sorts and CSV" python3 main/scripts/plan-set.py LT7 todo "Planning 14 Sep" --add --stream L --wave w3 --due 2026-09-15 --title "Merge desk: tables, batch approve, proposals rules never withdraw, the proposal files" python3 main/scripts/plan-set.py LT8 todo "Planning 14 Sep" --add --stream L --wave w3 --due 2026-09-15 --title "The recrawl, told honestly: Extime window drafted, the running-list items, the Dubai story" ``` ## Owns New: `main/alembic/versions/b3c4d5e6f7a8_quantity_listed_decided.py`, `main/app/services/quantity.py`, `main/app/services/overrides.py`, `main/app/services/collected.py`, `main/app/services/listings_table.py`, `main/app/services/merge_desk.py`, `main/web/src/components/collectors/*` (+ `.css`), `main/tests/test_quantity.py`, `main/tests/test_decided.py`, `main/tests/test_listed.py`, `main/tests/test_listings_table.py`, `main/tests/test_merge_desk.py`, `main/tests/test_collectors_heinemann.py`, `import/proposals/*.json`, `.logs/runs/window-2026-09-15-extime-fragments.md`. Existing: `main/app/services/normalize.py`, `lines.py`, `keying.py`, `merges.py`, `suggest.py`, `merge_session.py`, `collector_view.py`, `ingest.py` (the resolve functions, the veto, the create and enrich writes, the hook after `_resolve_listing`; nothing in the fetch or run-status paths), `catalog_queries.py` (the `live_listings` helper and its call sites only), `collectors/base.py`, `collectors/heinemann.py`, `collectors/heinemann_platform.py`, `collectors/extime.py`, `collectors/changi.py`, `collectors/shilla.py` (the quantity reads only, nothing that touches the network, robots or pacing), `models/catalog.py`, `routers/collectors.py`, `pages/CollectorsPage.tsx`, `main/docs/DATA-MODEL.md`, `main/docs/COLLECTORS.md` (the reader and posture sections), the tests named per task, `tests/test_accounts_backfills.py` (the chain pin only), `tests/test_keying.py`, `test_lines.py`, `test_listings_view.py`, `test_merges.py`, `test_suggest.py`, `test_ingest_guards.py`, `test_line_backfills.py`, `test_collector_view.py`, `test_house_style.py`, the collector tests for the five collectors above. ## Shared, append your own lines only `main/app/services/access.py` (your route keys in the class dicts; never restructure), `main/app/cli.py` (`BACKFILLS` entries `quantities`, `listed`; the `propose` subcommand), `main/web/src/lib/urls.ts`, `main/tests/fixtures/open_routes.json` (untouched unless a route is made public, which none is), `main/docs/RUNBOOK.md` (your commands, the after-deploy block, the desk and recrawl paragraphs), `main/CHANGELOG.md` Unreleased, `import/progress.json` through `plan-set.py`, `.logs/handoff.md` prepend-only. Re-read a shared file immediately before editing it, make the edit in one write, commit within the same task; never hold an uncommitted change to a shared file across tasks. ## Must not touch `services/collectors/fetch.py`, `robots.py`, `registry.py`, `avolta.py`, `shopify.py`, `ari.py`, `dubai.py` (their network behaviour and pacing are Stream A's); `models/schemas.py`, `services/seo.py`, `services/trip.py`, `services/feeds.py`, `web/src/lib/format.ts`, `ProductCard.tsx`, `ProductPage.tsx`, `SavingsPage.tsx`, `BrowsePage.tsx` (the public site keeps `size_ml`); `services/verify.py`, `audit.py`, `quality.py`; `routers/*` other than `collectors.py`; `web/src/vendor/*` and the accounts kit; `App.tsx` (no new page); `deploy/`, `docker-compose*.yml`, the crontab, `app/version.py`; `.env`, `.app.env`, `.bw-auth.env`; the live database `dutyfreeprofessor-db` beyond read-only SELECTs and one read-only `pg_dump` (rehearse everything on `dfp-devdb`, 127.0.0.1:5433, in `dfp_l`, a fresh copy of `/srv/apps/dutyfreeprofessor/backups/dfp-nightly-2026-09-14.dump`, dropped before the final handoff); production in any form; the hosts that have refused us. ## Rules - **Migrations: one head, one file, schema only.** Run `alembic heads` immediately before writing and immediately before committing; if another lane's revision has appeared in `alembic/versions/`, chain `down_revision` onto it and re-rehearse up/down/up on `dfp_l`. Never `alembic merge`. Never Python data moves inside Alembic; every data move is an `app.cli backfill ` that takes `--check` and writes zero on its second run. The head pin test moves in the same commit as the migration. - **Listed is never changed.** A listed column is written only from the fragment by the one reader; NULL where there is none; never from Product fields; never trimmed, cased or unescaped differently from ingest, on the page as in the database. - **Decided beats standard, standard beats listed; a machine never overwrites a human value** (agents.md). Every rekey and rederive reads `overrides` first. - **The key never collapses two dimensions or unknown onto known**: every quantity is value then unit (`100ml`, `50g`, `200pcs`), `unknown` when none was read, and an unknown-quantity group is a candidate, never an auto-fold; every key moves under v4 but only sets and unknowns change GROUPING. A merge is forward and recorded, never a delete (Decision 6). Read `backfill merges --check` (which LT4 teaches to print the groups) before any real run and report how many groups v4 joins that v3 kept apart and how many it splits, by SQL against the `l_v3_keys` snapshot. - **A GET never writes.** Every count on the page comes from the database on request, never typed. `decided_by`, `set_by`, `pinned_by`, `ignored_by` come from the signed-in account, never the payload. - **No collector runs tonight.** One robots.txt read per host to test code is the whole network budget. Never contact a host that has refused us. - **American spelling in code:** `color`, never `colour`; `flavor`, never `flavour`; "quantity" in every new name. `test_house_style.py` gains a check that `colour` and `flavour` appear in no identifier, column name, key token or alias row in the files this stream creates and in `VARIATION_KINDS` (existing prose in `SettingsPage.tsx`, `review.ts`, `structure.ts`, `airport_guides.py` is out of scope). No em dashes and never "free" or "cheap" in anything a person reads. `test_house_style.py` walks only `web/src` today; LT1 extends the walk to the Python files this stream creates, so the check reaches them. - **Delegate mechanical edits** (fixture trimming, the five collectors' quantity reads from a precise spec, the column registry, CSS, docstring passes) to `model: "sonnet"` and verify the diff yourself; never delegate the parser's rules, the key encoding, the override reader, the proposal files, or anything a person reads on the listings or merge page. - Everything else per `OVERNIGHT-RULES.md`: never deploy (say "green and ready" and name the after-deploy commands in order), own files only, `../.venv-dev/bin/pytest tests -q` and `main/check.sh` green before every commit, one task one commit with the `L:` prefix, handoff ≤25 lines, `/plan` updated, decisions on the running list with `--blocks` and `--weight` and the assumption you proceeded under.