"""The line above the product, the attribute vocabulary, and the form: the pure rules. Sources of truth: this module, `models/catalog.py` (ProductLine, AttributeAlias, ProductVariant.product_line_id), `cli.backfill_lines` / `backfill_attributes`, `tests/test_lines.py`. Decided by rian on 12 Sep (`.logs/planning/streams/M-merging.md`): every listing is standardised into brand | line | attribute | size; the line is the real product ("1 Million"), the attribute is the juice (Eau de Toilette, Parfum, Elixir), the size is the bottle; a product is one line at one attribute and one size; the page will be the line. Everything here is a pure function of the collected text, so `rederive` can re-run it over stored raw with no network and a test can pin every row that once went wrong. The key must be conservative in one direction only: two spellings of one line must meet, and two lines must never meet by accident. Where the rules cannot tell ("XV" against "15", "Aromatic Essence" against the plain line), they keep the words and leave the join to a person through the suggestion queue; a wrong merge costs more than a missed one. Beauty strips the concentration and attribute words out of the line and hands them to the attribute. Drinks strip nothing of the expression: a 12 and an 18 are different lines, and "original", "reserve", "XO", "black label" stay, because the catalogue's own stopword list once deleted exactly the words that tell an expression from its sibling and put a medal on the wrong bottle (agents.md). The audience is part of a beauty line ("Eternity for Men" and "Eternity for Women" are different bottles); its spellings are folded so shops agree. Identity rules v5 (`.logs/planning/catalogue-model-decisions-2026-09-15.md`, 15 Sep): a attribute a reader finds never stays in the line key, so a shade and a confectionery flavour the shop marked leave the line as the concentration does; a attribute is read only from what the shop marked, never guessed from a word in the name; and "triple", "tri" and "twin" are the expression unless the name is a pack or a set. Every kind a reader produces carries a display setting (`ATTRIBUTE_DISPLAY`): picked, shown or fact. Identity rules v6 (plan W14, 17 Sep) overrule the two paragraphs above where they describe deleting words: the line key is the RESIDUAL NAME, only the certain removals are made (`product_line_key`), and every open word list moved to `proposal_rules.py`, where it writes proposals and applies nothing. A shop's option arrives as a field (`RawListing.options`). """ from __future__ import annotations import os import re import unicodedata from collections import Counter from collections.abc import Iterable from app.services.normalize import ( _ABV_RE, _NOISE_RE, _SIZE_PATTERNS, _strip_brand_prefix, brand_key, ) #: The verticals that take the beauty branch of the line and display rules (the audience #: folded, no age stripping, the concentration and attribute words removed). Decoupled from #: which verticals carry a attribute (`ATTRIBUTE_RULES`, Stream L): a confectionery flavor #: rule must never route the confectionery names through the beauty branch. BEAUTY_VERTICALS = frozenset({"beauty"}) #: What kind of thing a attribute is. Metadata beside the canonical string #: (`attributes.attribute_kind`, `attribute_aliases.kind`), never a key slot. "shade" is a #: shop word for the kind `color` (American spelling in every identifier; the page may say #: "Shade" for makeup); `age`, `cask` and `edition` are registered without rules, because a #: drink keeps its age in the line and a rule there would re-key every whisky. ATTRIBUTE_KINDS = ("concentration", "color", "flavor", "age", "cask", "edition") #: How a kind behaves on a page (`.logs/planning/catalogue-model-decisions-2026-09-15.md` #: §2.4); the person's words are PICKED and SHOWN. A PICKED is picked: the shopper selects #: it and the comparison regenerates, because changing it changes the price or what a shopper #: would search for (1 Million EDT against Elixir at one quantity were 19 percent apart). A #: SHOWN is shown: information, never selected, because a shade picker would claim to #: know what a shop has in stock today (eighteen shades of one lipstick carried exactly two #: prices). Only a kind a reader produces has a behaviour; `age`, `cask` and `edition` have no #: reader (a drink keeps its age in the line), so the question never arises for them. The #: quantity is not a kind and is always picked. A rule in code, not a table: it changes with a #: deploy, and rian's own read is that it is a presentation matter. A `(vertical, category, #: kind)` exception wins over the kind's default when evidence says a kind behaves differently #: somewhere (a flavour or a colour may well be picked in a category food or eyewear brings). PICKED = "picked" SHOWN = "shown" FACT = "fact" # a fact about a variant the page neither picks nor lists (plan W8); no kind uses it yet ATTRIBUTE_DISPLAY: dict[str, str] = {"concentration": PICKED, "color": SHOWN, "flavor": SHOWN} ATTRIBUTE_DISPLAY_EXCEPTIONS: dict[tuple[str, str | None, str], str] = {} def display_of(kind: str, *, vertical: str | None = None, category: str | None = None) -> str: """`picked`, `shown` or `fact` for a kind, an exception for the vertical and category winning over the kind's default; "" for a kind no reader produces or the vocabulary does not know (a marked variant of unknown kind is shown until a person or a rule names it).""" if not kind: return "" for key in ((vertical or "", category, kind), (vertical or "", None, kind)): if key in ATTRIBUTE_DISPLAY_EXCEPTIONS: return ATTRIBUTE_DISPLAY_EXCEPTIONS[key] return ATTRIBUTE_DISPLAY.get(kind, "") #: The concentration phrases and the attribute qualifiers, longest phrase first, each to #: its label. Read before the brand and the line, because "Eau de Parfum" must never leave #: a stray "eau" or "de" in the line. _ATTRIBUTE_RULES: list[tuple[re.Pattern[str], str]] = [ (re.compile(r"\babsolu\s+de\s+parfum\b", re.I), "absolu"), (re.compile(r"\b[eé]lixir\s+de\s+parfum\b", re.I), "elixir"), (re.compile(r"\bessence\s+de\s+parfum\b", re.I), "parfum"), (re.compile(r"\bextrait\s+de\s+parfum\b", re.I), "parfum"), (re.compile(r"\beau\s+de\s+parfum\b", re.I), "edp"), (re.compile(r"\beau\s+de\s+toilette\b", re.I), "edt"), (re.compile(r"\beau\s+de\s+cologne\b", re.I), "edc"), (re.compile(r"\bedp\b", re.I), "edp"), (re.compile(r"\bedt\b", re.I), "edt"), (re.compile(r"\bedc\b", re.I), "edc"), (re.compile(r"\bcologne\b", re.I), "edc"), (re.compile(r"\bextrait\b", re.I), "parfum"), (re.compile(r"\bparfum\b", re.I), "parfum"), (re.compile(r"\bperfume\b", re.I), "parfum"), (re.compile(r"\b(?:body\s+|hair\s+)?(?:mist|brume|bruma)\b", re.I), "mist"), (re.compile(r"\b[eé]lixir\b", re.I), "elixir"), (re.compile(r"\bintense\b", re.I), "intense"), (re.compile(r"\bextr[eêè]me\b", re.I), "extreme"), (re.compile(r"\babsolu(?:e|te)?\b", re.I), "absolu"), ] _CONCENTRATIONS = ("edp", "edt", "edc", "parfum", "mist") _QUALIFIERS = ("intense", "extreme", "absolu") #: What a person reads for each canonical attribute. ATTRIBUTE_LABELS = { "edp": "Eau de Parfum", "edt": "Eau de Toilette", "edc": "Eau de Cologne", "parfum": "Parfum", "mist": "Mist", "elixir": "Elixir", "intense": "Intense", "extreme": "Extreme", "absolu": "Absolu", } # ---------------------------------------------------------------------------- the certain boundary # Identity rules v6 (plan W14; `docs/REVIEW-PROCESS.md` section 1 is the readable copy and a # test holds the two equal). A rule removes from a listed name ONLY: the brand as matched to its # row, the stated quantity, an ABV percentage, a pack figure, and the closed concentration # vocabulary; an age statement's synonyms fold to one token ("12 Years Old", "12 YO", "Aged 12 # Years" are `12yo`) and stay in the name. Every open word list that used to delete words (drink # words, regions, format words, connectors, the audience fold, packaging noise, a brand's last # word alone) lives in `proposal_rules.py` and writes proposals; none of them touches a key. # What that replaced: "triple" on a format list keyed Triple Cask as Cask, a stopword list put a # medal on the wrong bottle, and each was found by a person after the fact. # # The boundary was chosen from three rehearsed on a copy of staging (K3.2, 17 Sep; the numbers # are in the handoff and REVIEW-PROCESS.md): "a" removes nothing more, "b" adds the closed list # of drink category nouns, "c" adds the format words that never name a product. CATEGORY_NOUNS = frozenset({ "whisky", "whiskey", "whiskies", "scotch", "bourbon", "gin", "vodka", "rum", "rhum", "ron", "tequila", "mezcal", "cognac", "brandy", "armagnac", "calvados", "liqueur", "liqueurs", "liquor", "wine", "wines", "champagne", "prosecco", "cava", "beer", "lager", "cider", "spirit", "spirits", "blended", "blend", }) #: "single malt", "single grain", "blended malt": the pair is the category; "single" or "malt" #: alone may be the expression ("Single Barrel", "Pure Malt") and stays. CATEGORY_PAIRS = frozenset({("single", "malt"), ("single", "malts"), ("single", "grain"), ("blended", "malt"), ("blended", "grain")}) #: A category noun that names a cask is the expression: "Rum Cask", "Sherry Cask Finish". CASK_WORDS = frozenset({"cask", "casks", "finish", "finished", "barrel", "barrels", "wood", "matured"}) NEVER_NAMING_FORMAT_WORDS = frozenset({ "spray", "vapo", "vaporisateur", "vaporizador", "vaporizer", "bottle", "bottles", "btl", "ml", "cl", "oz", "fl", "gr", "g", "ltr", "lt", "vol", "abv", "proof", }) BOUNDARIES: dict[str, frozenset[str]] = { "a": frozenset(), "b": frozenset({"category_nouns"}), "c": frozenset({"category_nouns", "format_never"}), } #: The boundary identity rules v6 key under. Changing it is a rules version and a `rederive`. #: `DFP_REHEARSE_BOUNDARY` keys a COPY under another boundary to measure it (K3.2's three-way #: rehearsal; RUNBOOK); it is never set on a deployed app. BOUNDARY = os.environ.get("DFP_REHEARSE_BOUNDARY") or "a" def boundary_flags(boundary: str | None = None) -> frozenset[str]: return BOUNDARIES[boundary or BOUNDARY] _GLYPHS_RE = re.compile(r"[™®©℠*]") _DOTTED_RE = re.compile(r"\b(?:[a-z]\.){1,4}[a-z]\.?(?![a-z0-9])", re.I) # X.O, V.S.O.P. -> xo, vsop _APOSTROPHE_RE = re.compile(r"['’`]") _NUMBERED_RE = re.compile(r"\bn(?:o|°|º)?\.?\s*(\d+)\b") # N°5, No. 5, No5 -> no 5 _EXTRA_SIZE_RE = re.compile(r"\b\d+(?:[.,]\d+)?\s*(?:fl\.?\s*oz|oz|g|gr|kg)\b", re.I) _PACK_RE = re.compile(r"\b\d{1,2}\s*[x×]\s*\d+(?:[.,]\d+)?\s*(?:ml|cl|l)\b", re.I) _AGE_RE = re.compile(r"\b(\d{1,2})\s*(?:years?|yrs?|yo|y|ans|jahre|anos)\b(?:\s*old)?", re.I) _AGED_RE = re.compile(r"\baged\s+(\d{1,2})\b(?:\s*(?:years?|yrs?|yo|ans)\b)?(?:\s*old\b)?", re.I) _PUNCT_RE = re.compile(r"[^a-z0-9]+") _REFILL_RE = re.compile(r"\b(refill|recharge|rechargeable|refillable|recargable)\b", re.I) KEY_MAX = 150 def _fold(text: str | None) -> str: """Lowercase ASCII with the glyphs a shop decorates a name with removed.""" text = unicodedata.normalize("NFKD", text or "").encode("ascii", "ignore").decode().lower() text = _GLYPHS_RE.sub(" ", text) text = _DOTTED_RE.sub(lambda m: m.group(0).replace(".", ""), text) return _APOSTROPHE_RE.sub("", text) def _strip_measures(text: str, *, noise: bool = False) -> str: """Sizes, packs, counts and strengths, gone: a gram pack ("12 x 20g") and a piece count ("X20 Pieces", "90 caps") leave the line as a volume does. The packaging noise ("gift box", "with 2 glasses", "limited edition") is an open list and leaves only when a proposal rule asks (`noise`): a gift pack with glasses is not certainly the bottle.""" from app.services import quantity # quantity builds on normalize; the import here keeps lines above it if noise: text = _NOISE_RE.sub(" ", text) text = _PACK_RE.sub(" ", text) text = quantity._PACK_RE.sub(" ", text) text = quantity._GLUED_COUNT_RE.sub(" ", text) text = quantity._SINGLE_RE.sub(" ", text) for pattern, _ in _SIZE_PATTERNS: text = pattern.sub(" ", text) text = _EXTRA_SIZE_RE.sub(" ", text) return _ABV_RE.sub(" ", text) _SLASH_RE = re.compile(r"\s+/\s+") _SPACES_RE = re.compile(r"\s+") def _read_by_rules(text: str, rules: list[tuple[re.Pattern[str], str]]) -> list[str]: """The labels the rules find, in the order the shop wrote them; each span blanked in place so a shorter rule cannot re-read it ("parfum" inside "eau de parfum").""" found: list[tuple[int, str]] = [] for pattern, label in rules: for match in pattern.finditer(text): found.append((match.start(), label)) text = text[: match.start()] + " " * (match.end() - match.start()) + text[match.end():] return [label for _, label in sorted(found)] def _read_concentration(text: str) -> tuple[str, str]: labels = _read_by_rules(text, _ATTRIBUTE_RULES) if not labels: return "", "" raw = " ".join(dict.fromkeys(labels)) return raw, canonical_attribute(raw) def _read_tail(text: str) -> tuple[str, str]: """The variant the shop MARKED: the text after the last " / ", the separator the Shopify platform puts between a product's size option and its second option ("Joli Blush Blusher 6 gr / 02 Cheeky Pink" is "02 cheeky pink"; "Lindor Truffles 200 g / Milk" is "milk"). Since identity rules v6 the collector carries the option as a field and never glues it, so this reader is only the FALLBACK for a name stored before that change whose fragment `backfill options` could not reach; a fresh fragment never gets here. Every such tail is the shop's own option, a skin type included (v5 kept "/ Seca" in the line by a word list). Read across the beauty vertical (the category names the kind, never whether the tail is a attribute: 384 of the 783 tailed beauty rows on the 15 Sep copy had no category at all, CHANEL Rouge Allure among them) and for confectionery. Nothing is read from a name that carries no marked tail: no "No. NN" rule and no word-after-colour rule, which would tag "Brush N°13" and "Chanel N°5" and read "Riche" off "Colour Riche"; and no flavour-word rule, which tagged 156 one-off confectionery names ("15 Fine Chocolates Almond Crispy") as flavours of a line and grouped nothing (identity rules v5; the decisions doc §2.3, §2.6). Empty beats guessed: a shade Extime writes as " - 447 Mellow Shade" or Avolta as a bare "01" stays in the line until a reader for that shop's shape is pinned by its fragments.""" if not _SLASH_RE.search(text): return "", "" tail = _SPACES_RE.sub(" ", _SLASH_RE.split(text)[-1]).strip(" -,.") tail = _PUNCT_RE.sub(" ", tail).strip() if not tail: return "", "" return tail, tail def _without_tail(text: str) -> str: """The text with the marked tail `_read_tail` reads taken off, for the line key; unchanged when there is none.""" if not _read_tail(text)[0]: return text parts = _SLASH_RE.split(text) return " ".join(parts[:-1]) #: What `attribute_of` consults, by (vertical, category): the category's readers are tried #: before the vertical's own, in order, and the first reader that finds a wording wins. Each #: entry is the kind and a reader of the folded, measure-stripped text returning (raw, #: canonical). A kind of "" is a marked variant whose kind nothing names yet (a tailed beauty #: row outside Makeup: eyewear colours and frame shapes sit there today); it is still a #: attribute, and it leaves the line. ATTRIBUTE_RULES: dict[tuple[str, str | None], list[tuple[str, object]]] = { ("beauty", None): [("", _read_tail), ("concentration", _read_concentration)], ("beauty", "Makeup"): [("color", _read_tail)], ("confectionery", None): [("flavor", _read_tail)], } #: The verticals with any attribute rule (what `backfill attribute_values` walks). ATTRIBUTE_VERTICALS = frozenset(v for v, _ in ATTRIBUTE_RULES) def _rules_for(vertical: str | None, category: str | None) -> list[tuple[str, object]]: out: list[tuple[str, object]] = [] if category and (vertical, category) in ATTRIBUTE_RULES: out.extend(ATTRIBUTE_RULES[(vertical, category)]) if (vertical, None) in ATTRIBUTE_RULES: out.extend(ATTRIBUTE_RULES[(vertical, None)]) return out def _reads_tail(vertical: str | None, category: str | None) -> bool: """Whether a marked tail is a attribute in this vertical, and so leaves the line key (beauty and confectionery; the category is not consulted, so a classifier's choice can never move a shade between the line and the attribute slot).""" return any(reader is _read_tail for _, reader in _rules_for(vertical, category)) def _read_attribute(name: str | None, vertical: str | None, category: str | None) -> tuple[str, str, str]: if not name: return "", "", "" rules = _rules_for(vertical, category) if not rules: return "", "", "" text = _strip_measures(_fold(name)) seen: set[object] = set() for kind, reader in rules: if reader in seen: continue # the Makeup tail reader already ran with its kind; the vertical's kindless one never overrides it seen.add(reader) raw, canonical = reader(text) if raw: return kind, raw, canonical return "", "", "" def attribute_of(name: str | None, vertical: str | None, *, category: str | None = None) -> tuple[str, str]: """`(raw, canonical)` of a name's attribute: the vocabulary words found, in the order the shop wrote them ("elixir parfum intense"), and the one they mean ("elixir"). Both empty when the name says nothing or the vertical has no rule. The category reaches the rule (Makeup reads a shade; Skincare's " / Grasa" tails are skin types and get none). The alias table may map a raw wording elsewhere; the rule is the default it starts from.""" _, raw, canonical = _read_attribute(name, vertical, category) return raw, canonical def attribute_kind_of(name: str | None, vertical: str | None, *, category: str | None = None) -> str: """The kind of the attribute `attribute_of` reads (`ATTRIBUTE_KINDS`), "" when none.""" return _read_attribute(name, vertical, category)[0] def canonical_attribute(raw: str) -> str: """The rule's reading of a raw wording: Elixir dominates; else the first concentration named, then its qualifiers in a fixed order; a bare qualifier stands alone.""" labels = raw.split() if not labels: return "" if "elixir" in labels: return "elixir" base = next((label for label in labels if label in _CONCENTRATIONS), "") quals = [q for q in _QUALIFIERS if q in labels] return " ".join([*( [base] if base else []), *quals]) def display_attribute(canonical: str) -> str: """"edt intense" -> "Eau de Toilette Intense".""" return " ".join(ATTRIBUTE_LABELS.get(label, label.title()) for label in canonical.split()) def form_of(name: str | None) -> str: """The form a name declares that keeps it off the bottle's own row: "set" (a coffret or kit, once folded a Boss Bottled gift set into the bottle), "refill" (the pod, not the jar), "pack" (a case of several); empty for a single item.""" from app.services import quantity if not name: return "" form = quantity.parse_quantity(name).form return "" if form == "single" else form def _slot_value(value: object) -> str: return _PUNCT_RE.sub("-", _fold(str(value))).strip("-") def identity_slot(name: str | None, vertical: str | None, *, category: str | None = None, canonical: str | None = None, options: dict | None = None) -> str: """The key's identity slot (identity rules v6): `kind=value` for every CERTAIN identity attribute, sorted by kind, joined with ";". A pure function of what the shop stated, so an arrival and a stored variant always compute the same slot: * `concentration=` the closed vocabulary's canonical word (`canonical` when the caller resolved the wording through the alias table), or `tail=` the marked " / " tail of a name stored before options were fields (the fallback; never both, the tail reader runs first); * `abv=` a percentage the NAME states ("40%"; numbers as numbers, so 40.0 is 40). The statement leaves the residual name, so it must sit here or a 70 percent and an 85 percent chocolate bar of one weight would share a key. A strength a feed's own field states fills the `abv` column and vetoes a merge (`merges._conflicts`) but is not in the key: a column another listing enriched would move the key away from its own listing's next sighting; * `option:=` every option the shop published as a field (`attributes["option:*"]`). The age is not repeated here: its statement stays in the residual name (`12yo`).""" from app.services.normalize import parse_abv parts: dict[str, str] = {} kind, raw, rule = _read_attribute(name, vertical, category) value = canonical if canonical is not None else rule if raw and value: parts["concentration" if kind == "concentration" else "tail"] = _slot_value(value) abv = parse_abv(name) if abv is not None: parts["abv"] = f"{abv:g}" for key, option in (options or {}).items(): if key.startswith("option:") and option not in (None, ""): parts[key] = _slot_value(option) return ";".join(f"{k}={v}" for k, v in sorted(parts.items())) def resolve_alias(rows: dict, row_id: int | None, hops: int = 4): """The row an id points at once its `alias_of_id` chain is followed (a listed_brand's brand, a line's canonical line); None for an unknown id. Bounded, so a cycle cannot hang.""" row = rows.get(row_id) if row_id else None for _ in range(hops): if row is None or not row.alias_of_id or row.alias_of_id == row.id: break row = rows.get(row.alias_of_id, row) return row def _brand_sequences(listed_brand: str | None, brand: str | None) -> list[list[str]]: out: list[list[str]] = [] for text in (brand_key(listed_brand), _fold(listed_brand), brand_key(brand), _fold(brand)): words = _PUNCT_RE.sub(" ", text).split() if words and words not in out: out.append(words) return out def age_of(name: str | None) -> int | None: """The age a name clearly states, in years ("12 Years Old", "18 ans", "Aged 12 Years"); None for a bare number or a numeral ("XV"), which state nothing (REVIEW-PROCESS.md section 1).""" text = _fold(name) match = _AGED_RE.search(text) or _AGE_RE.search(text) return int(match.group(1)) if match else None def _strip_brand(tokens: list[str], sequences: list[list[str]]) -> list[str]: """The name without the brand AS MATCHED TO ITS ROW: a spelling's whole word sequence at the head (an article before it tolerated: "The Macallan 12"), else the whole sequence anywhere ("Flower by Kenzo"). A partial ("Armani" for Giorgio Armani) or an abbreviation ("Joh." for Johnnie Walker) is not a match: the words stay and the match is a proposal (`proposal_rules`: brand_partial).""" ordered = sorted((seq for seq in sequences if seq), key=len, reverse=True) for seq in ordered: for lead in (0, 1): if lead and (not tokens or tokens[0] != "the"): continue if tokens[lead: lead + len(seq)] == seq: return tokens[lead + len(seq):] # nothing left is the brand's namesake line for seq in ordered: for i in range(1, len(tokens) - len(seq) + 1): if tokens[i: i + len(seq)] == seq: return tokens[:i] + tokens[i + len(seq):] return tokens def _drops(tokens: list[str], index: int, flags: frozenset[str]) -> bool: """Whether the certain boundary (`flags`) removes this token. "a" removes none.""" token = tokens[index] following = tokens[index + 1] if index + 1 < len(tokens) else "" previous = tokens[index - 1] if index else "" if "category_nouns" in flags: if following in CASK_WORDS and token in CATEGORY_NOUNS: return False # "Rum Cask", "Sherry Cask Finish": the noun names the cask if token in CATEGORY_NOUNS or (token, following) in CATEGORY_PAIRS or (previous, token) in CATEGORY_PAIRS: return True if "format_never" in flags and token in NEVER_NAMING_FORMAT_WORDS: return True return False def residual_tokens(name: str | None, *, listed_brand: str | None = None, brand: str | None = None, vertical: str | None = None, category: str | None = None, noise: bool = False, partial_brand: bool = False) -> list[str]: """The folded words of a name with only the certain removals made and NO boundary list applied: what `product_line_key` filters and what every proposal rule starts from. `noise` and `partial_brand` are never set by the key: they are two of `proposal_rules`' readings (the packaging noise gone; the brand's last word alone taken off, the v5 brand strip).""" text = _strip_measures(_NUMBERED_RE.sub(r"no \1", _fold(name)), noise=noise) if _reads_tail(vertical, category): text = _without_tail(text) if vertical in BEAUTY_VERTICALS: for pattern, _ in _ATTRIBUTE_RULES: text = pattern.sub(" ", text) text = _AGED_RE.sub(r" \1yo ", text) text = _AGE_RE.sub(r" \1yo ", text) tokens, sequences = _PUNCT_RE.sub(" ", text).split(), _brand_sequences(listed_brand, brand) if not partial_brand: return _strip_brand(tokens, sequences) tokens = _strip_brand_prefix(tokens, sequences) last_words = {seq[-1] for seq in sequences if seq} return [t for t in tokens if t not in last_words] or tokens def product_line_key(name: str | None, *, listed_brand: str | None = None, brand: str | None = None, vertical: str | None = None, category: str | None = None, boundary: str | None = None) -> str: """The line's key: the RESIDUAL NAME of the certain key (identity rules v6, plan W14). The listed name folded (case, accents, punctuation, glyphs; "V.S.O.P." is `vsop`, "N°5" is `no 5`) with only the certain removals made: the brand as matched to its row, the stated quantity, a pack figure, an ABV percentage, the closed concentration vocabulary (beauty), and the marked " / " tail of a name stored before the collectors carried options as fields. An age statement folds to one token (`12yo`) and stays. The boundary (`BOUNDARY`) may remove one or two CLOSED lists more; nothing else is deleted, so "Blended Scotch Whisky", "Gift Box" and "pour Homme" stay in the key and what a word list would have made of them is a proposal. Empty is the brand's namesake line ("Jameson" itself). Under v5 this deleted 72 drink words, 66 format words, the connectors and a brand's last word anywhere, and folded the audience: every silent wrong join on record came from one of those lists.""" tokens = residual_tokens(name, listed_brand=listed_brand, brand=brand, vertical=vertical, category=category) flags = boundary_flags(boundary) kept = [t for i, t in enumerate(tokens) if not _drops(tokens, i, flags)] return " ".join(kept)[:KEY_MAX].rstrip() def brand_words_of(*names: str | None) -> frozenset[str]: """Every word of every spelling of a brand, folded, for the display to leave out.""" return frozenset(w for name in names for seq in _brand_sequences(name, None) for w in seq) def _display_from(sample: str, brand_words: frozenset[str], vertical: str | None, *, prefix_only: bool, category: str | None = None) -> str: """The sample name with the brand, the measures and every certain attribute (the beauty concentration words, a marked tail) taken out, the boundary's closed lists applied, and everything else left as the shop cased and spelt it.""" text = _GLYPHS_RE.sub(" ", sample or "") text = _strip_measures(text) if _reads_tail(vertical, category): text = _without_tail(text) if vertical in BEAUTY_VERTICALS: for pattern, _ in _ATTRIBUTE_RULES: text = pattern.sub(" ", text) tokens = text.split() folded = [_PUNCT_RE.sub("", _fold(t)) for t in tokens] flags = boundary_flags() keep = [] at_head = True for index, (token, fold) in enumerate(zip(tokens, folded)): if not fold: continue if fold in brand_words and (at_head or not prefix_only): continue at_head = False if _drops(folded, index, flags): continue keep.append(token.strip("-.:;,()[]/")) return " ".join(t for t in keep if t) def display_line_name(key: str, samples: Iterable[str] | Counter, brand_name: str, *, brand_words: frozenset[str] = frozenset(), vertical: str | None = None, category: str | None = None) -> str: """A spelling to show for a line: the most common product name with the brand, the measures, the format and the attribute words removed and the rest as the shop wrote it (articles, apostrophes and accents kept: "Le Male", "L'Interdit", "Terre d'Hermès"), accepted only when it folds back to the key; the standard brand's own name for the namesake line; the key's words in title case when no sample fits.""" if not key: return brand_name counter = samples if isinstance(samples, Counter) else Counter(samples) for sample, _ in counter.most_common(): for prefix_only in (False, True): display = _display_from(sample, brand_words, vertical, prefix_only=prefix_only, category=category) if display and product_line_key(display, vertical=vertical, category=category) == key: return display return " ".join(w.upper() if w in ("xo", "vsop", "vs", "xxo") else w.title() for w in key.split()) def line_slug(brand_slug: str, key: str) -> str: """The address a line page will answer at: the brand's slug, then the key's words.""" words = key.replace(" ", "-") return f"{brand_slug}-{words}" if words else brand_slug