# Stream M: one page per real product, and the merge session that gets us there
(written 2026-09-12 Sat, from rian's decision of the same day; build plan v3 §4 items 4 to 7 are the design it lands)

**Runs:** as one session at a time, never alongside Stream A (they share the identity layer).
**Decided by rian, 12 Sep:** one page per real product; sizes and variations chosen on that page;
every listing standardised into brand | name | variation | size; matches made automatically where
all four agree; merges at every level made by a person, with the machine suggesting and the person
confirming in bulk. The page change itself (the line page with its toggles and the redirects) is a
separate, later decision; **this stream ends with rian able to sit down and confirm merges in volume.**

## What already exists, so you build on it
- `products` is the priced thing (brand, name, size_value/unit with `size_ml` derived, `attributes`
  with `concentration` for beauty, `gtin`, `image_url`); `listings` is one product at one shop;
  `raw_records` keeps the tile as collected (name, brand, price, size) for every listing and run.
- `brands` with `canonical_id`: a brand alias already exists as a row pointing at its house.
- `merges.py`: `match_key` groups (brand fold + name + size); a group merges only when barcodes do not
  disagree and per-vertical attributes do not disagree; otherwise `merge_candidates` rows with a
  reason. `merge_products()` repoints listings and awards, tombstones the loser (`merged_into_id`,
  never deletes), **the survivor keeps or gains the barcode and inherits the image** (`_ENRICH_FIELDS`),
  and `product_merges` records it. `resolve_product_id()` makes the old id answer under the new one.
  300 candidates sit undecided today, nearly all `gtin_differs`.
- `/collectors` (services/collector_view.py, routers/collectors.py, pages/CollectorsPage.tsx): the
  read-only review area, owner-only (`sources.manage`). The deciding half is what this stream adds.
- `app.cli rederive` re-runs identity rules over stored raw with no network; `identity_rules_version`
  on Product says which rules keyed it.

## The model, in rian's words and the columns they become
| Level | What it is | Where it lives | Merge = |
|---|---|---|---|
| Brand | Paco Rabanne | `brands` (+ alias via `canonical_id`) | point one house at another, pick the preferred name |
| Line, "the real product" | 1 Million | **new** `product_lines(id, brand_id, name, slug, canonical_id)`, `products.line_id` | point one line at another, pick the preferred name |
| Variation | Eau de Toilette / Parfum / Elixir; for drinks, none (a 12 and an 18 are different lines) | `products.attributes.concentration`, canonicalised through **new** `variation_aliases(vertical, raw, canonical)` | map a raw wording to a canonical one, pick the display wording |
| Size | 100 ml | `size_ml` (already normalised: 10cl = 100 ml) | display preference only; a different `size_ml` is never merged |
| Product, the priced thing | 1 Million EDT 100 ml | `products` | the existing recorded merge |

The comparison happens at the product level and nowhere else: a 50 ml against a 100 ml is not a
saving, and an EDT against a Parfum is not the same juice. The line is the page; the product is the
row on it. **Automatic matching = same brand house, same line, same canonical variation, same
`size_ml`, no barcode disagreement.** Everything else is a suggestion for a person.

## Tasks, in order (ids on /plan; record each with `main/scripts/plan-set.py`; commit prefix `M:`)
1. **M1 The listings table.** On `/collectors`, a fifth view, Listings: one row per listing, the
   collected tile on the left (brand, name, size, price, from `raw_records`' newest payload) and what
   we made of it on the right (house, line, variation, `size_ml`, product id, airport). Filterable by
   collector, airport, brand, and "differs from collected". Read-only. This is the surface every later
   decision is checked against, so it comes first.
2. **M2 The line and the variation vocabulary.** Migration (schema only, one head, revise from the
   current head; rehearse up/down/up on `dfp-devdb`): `product_lines`, `products.line_id`,
   `variation_aliases`. The line key as a pure function in a new `services/lines.py`: brand house +
   the name with the brand prefix, size tokens, concentration words and edition/pack tokens stripped;
   **beauty strips concentration, drinks do not strip age or expression** (Glenfiddich 12 and 18 are
   two lines). Backfills `lines` and `variations`, idempotent, registered in `BACKFILLS`. Fixtures:
   the fourteen 1 Million rows (ids 7761 to 7805, 14747, 14766, 14767, 17216 to 17219, 17314, 17885),
   which must land as one brand house, one line, three variations, and the set kept apart.
3. **M3 Matching that stops splitting the same bottle.** `match_key` v3 in `normalize.py`: brand
   prefix stripped from the name, concentration words stripped once `attributes.concentration` holds
   them, the variation canonicalised before keying. `rederive` stamps v3. Rehearse on `dfp-devdb`
   and report, from the database: how many products the new key joins that v2 kept apart, and how
   many comparables that creates. The 100 ml 1 Million EDT (7804, 17217, 14747) must become one
   product at five airports, under a human-confirmed brand alias, not a guessed one.
4. **M4 Aliases at every level, with a preferred name.** Brand alias exists; add line alias
   (`canonical_id`) and the variation alias table; each alias carries who set it and when
   (`FK accounts.id NULL`, plan §2 who-columns rule). Readers resolve alias → canonical everywhere a
   brand, line or variation is shown or keyed. **The first confirmed alias is Paco Rabanne → Rabanne,
   recorded as rian's decision**, never applied silently by a rule.
5. **M5 Suggestions, rules first.** Extend `merge_candidates` with `level` (brand | line | product)
   and keep `score`, `reason`, `detail`. Generators, each a pure scorer with tests: brand candidates
   (fold key equal, or one brand's name contained in another's with shared line names, or a
   rebrand list); line candidates (same house, name overlap above a threshold after stripping);
   product candidates (the existing groups, plus same line + same `size_ml` where the variation is
   unknown on one side). Every candidate carries a one-line reason a person can read without the
   code. Idempotent `app.cli suggest` command. An LLM pass is a later enhancement, not this stream.
6. **M6 The merge session.** A sixth view on `/collectors`, Merge: the queue by level and score,
   each pair side by side with its listings, airports, prices, barcodes and images; **confirm /
   reject / skip on the keyboard**; on confirm, the preferred name chosen (default the most common
   spelling across listings); confirm applies through `merge_products()` or the alias table and
   records `decided_by` from the signed-in account and `decided_at`; reject records the same and
   never resurfaces the pair. Writes go through new `POST /api/collectors/merge/{id}/…` routes,
   registered in `access.py` under `sources.manage` and in the mutating-routes fixture. Progress and
   throughput shown (decided today, remaining). Rian's stated outcome: "plow through and confirm
   or reject".
7. **M7 Inheritance, written down and tested.** Barcode: the survivor keeps its own or gains the
   other's; two different barcodes are a veto that only a person overrides, and the override records
   which barcode won and why; **a barcode is never invented and never copied onto a listing that did
   not publish it**; a barcode that came from a merge is marked as such so markup can tell. Image:
   the survivor inherits, provenance kept (`image_source`). Both stated in `DATA-MODEL.md`.

**Not in this stream, filed as the follow-on:** the product page becoming a line page with variant and
size toggles, and every existing product address redirecting to its line (Stream B, and a decision of
rian's, because it changes the addresses Mark just agreed). Do it after one real merge session has
shown what the catalogue actually collapses to.

## Owns
`services/lines.py` (new), `services/merges.py`, `services/normalize.py` (identity parts), the new
migration, `models/catalog.py` for `ProductLine` and `products.line_id`, `services/collector_view.py`
and `routers/collectors.py` and `pages/CollectorsPage.tsx` for the two new views, the `suggest`
and backfill commands, `DATA-MODEL.md`, their tests.

## Must not touch
Collectors and `ingest.py` beyond the keying call (Stream A); `seo.py`, `main.py` routes, the product
page (Stream B); the live database (rehearse on `dfp-devdb`, restore the newest
`backups/dfp-nightly-*.dump` into a fresh copy first); `.env`, `.app.env`.

## Rules
- A merge is forward and recorded, never a delete (Decision 6). A wrong merge is costlier than a
  missed one, so the machine merges only where all four fields agree and the barcodes do not
  disagree; everything else waits for a person.
- No em dashes in anything a person reads on the page. Every count on the page comes from the
  database on request, never typed.
- Everything else per `OVERNIGHT-RULES.md`: never deploy (say "green and ready" and name the
  after-deploy commands in order), own files only, `main/check.sh` green, handoff ≤25 lines, `/plan`
  updated, decisions on the running list with `--blocks` and `--weight`.