# Quality and verification

Sources of truth: `main/app/services/audit.py`, `main/app/services/verify.py`,
`main/app/cli_quality.py`, `main/app/models/quality.py`, `main/tests/test_audit.py`,
`main/tests/test_verify.py`, `.logs/verification/`.

Written for the owner, who said he understands this decision less than average. The rules
come from build plan §3a and Decision 10; this page says what they mean in practice. The
short version: **two tools, one bar.** The audit looks at what we already hold and lists
the shapes a misread leaves behind. Verify goes back to the retailer for a sample and says
whether the page still agrees with us. Neither one proves the data is right; together with
a dated human look at real pages, they are what "we check" honestly means.

## What each tool can and cannot prove

**`python -m app.cli audit`** reads the database and nothing else. No network, safe to run
at any hour. It measures sixteen things and lists the rows behind each one with the
retailer's link, so the human review starts from a list instead of a query:

- **Shapes that mean a parser read the wrong field:** a location whose prices are in a
  currency that is not the location's (this is how the Mexico City store was published at a
  fraction of its real prices); a size no bottle comes in with nothing in the name explaining
  it (this is how a 70cl Glenfiddich became a 7-litre product); a listing priced two ways in
  one day with no promotion behind it; Paris pages whose declared size disagrees with the
  size the per-litre price implies; a product whose category belongs to a different family.
  The sixteenth is a split, not a threshold: how many product images were found by
  barcode and how many by name (`image_provenance`), so a drift towards name matches shows.
- **Shapes that mean a feed is lying by default:** a shop where every row says "in stock"
  (Dubai's feed was read that way for two weeks); a spread where the dearest shop is more
  than two and a half times the cheapest (the $77 sparkling wine against $21 elsewhere,
  almost certainly a retailer's own typing error).
- **Housekeeping that decays into lies:** parent tiles with no per-size rows, duplicate
  products the merge rules would fold (read through `merges.duplicate_groups`, so the audit
  counts exactly what `backfill merges` would do and never a pair the rules queued for a
  human), brand spellings the fold has not joined, drinks with a price and no size, products
  with no category, prices converted on the fallback exchange table, runs that never
  finished, and how old the newest price is at each visible airport against the three-day
  mark the site shows. A merged product is a tombstone (`merged_into_id` set) and counts in
  no product metric: counted, it kept every duplicate and fold the merge had resolved on the
  list.

Each metric has a threshold written beside it in the code. **Every threshold started at the
value measured on 2026-09-05**, so the first run is green by construction and any later rise
is a change we can date. Tightening one is a one-line edit (the same evening, three were
re-pinned to a copy of that dump after migration #3's backfills: duplicate groups 0, brand
folds 0, unexplained large formats 2). The audit never hides a row and
never blocks a shop; it measures, and the snapshot is kept (the workspace's
`.logs/verification/` when run from a checkout, the audit's default `--out`, and the
`audit_snapshots` table after migration #2) so a threshold change can be read against what
was true before it.

**What the audit cannot do:** it cannot tell that a price is wrong. If a parser reads the
crossed-out price instead of the real one on every row, the numbers are consistent and the
audit is happy. That is what the human review is for.

**`python -m app.cli verify`** goes back to the retailer. For each shop it picks a sample of
listings we publish, reads each one again through the same collector, and compares what the
page says now with what we show. **What it measures, plainly: freshness and parser drift.**
It can tell that a price moved, that the page now shows a different size or a different
product, that the listing is gone, that the page no longer parses, or that the retailer
refused us. **It cannot tell that the parser reads the wrong field**, because the re-read
uses the same parser. If the parser was wrong on Tuesday it is wrong the same way on
Wednesday, and verify sees agreement. That gap is closed by the audit's distribution checks
above and by the dated human review, never by verify alone.

## The verdicts, in plain language

Every check is stored as a row (`verification_checks`), never only logged; the run it belongs
to (`verification_runs`) records its `mode`, one of `on_demand`, `after_collection` or
`weekly`, so the weekly N=200 and a post-collection N=20 can be told apart. A fourth table,
`rejected_observations`, holds every listing a collector or ingest refused, with the stage,
the reason and the payload, so a refusal is a row to read and not a line in a log. One of:

| Verdict | Meaning | What it does |
|---|---|---|
| `PASS` | The page agrees with us. | Nothing. |
| `PRICE_MOVED` | Same product, same size, the price changed. | Reported. A new observation is written with `source_kind='verify'`, so the site shows the newer figure (pending §10 #11). |
| `GONE` | The page no longer offers this listing. | Reported. The row keeps its date. |
| `MISMATCH_SIZE` | The page shows a size that cannot be the same bottle. | **Blocks the shop's publication** until a human clears it. |
| `MISMATCH_IDENTITY` | Different barcode, or the name no longer reads as the same product. | **Blocks** until cleared. |
| `MISMATCH_CURRENCY` | The page prices in a currency other than the one we published. | **Blocks** until cleared. |
| `PARSE_FAIL` | The page loaded but the parser could not read a price, or threw. | **Blocks** until cleared. |
| `REVIEW` | Readable, but implausible: the re-read price is under half or over 2.5 times what other shops charge, moved by more than half, or the page's two price channels disagree (Paris has a JSON-LD price and a payload price; on a single-size page they must agree). | Listed for a human; no block. |
| `BLOCKED` | The retailer refused us: robots.txt, a 401 or 403, an empty challenge body. | The shop is abandoned for this run after this one check. |
| `FETCH_ERROR` | A timeout or a 5xx. Not a refusal, not a parser problem. | Reported; try again next run. |

Identity is decided in that order: currency, then barcode, then size, then name. A moved
price on the wrong bottle is a mismatch, not a move. Two names are "the same product" when
the shorter one's significant words are at least three quarters contained in the other
(the Athens Lancôme row, "Lancome La Vie Est Belle 50ml" against the page's "La Vie Est
Belle Eau de Parfum 50ml", sits exactly at that line and passes); a matching barcode
overrides the name entirely.

## The tripwire, not a rate

After a collection, verify reads **twenty** listings per shop. That number cannot measure a
rate: twenty out of twenty only proves the true pass rate is above about 86 percent. So the
twenty are not a score. They are a tripwire: **one** correctness failure (`MISMATCH_*` or
`PARSE_FAIL`) marks the shop "publication blocked" until a human looks and clears the check
(`verify-clear`, which records who and when). `PRICE_MOVED` and `GONE` are reported and never
block. The rolling pass rate over every check ever made is shown to the owner
(`verify-status`) and is reviewed after two weeks of real data. **It is never a launch
condition.**

Publication blocked means: the shop's prices are not shown as current until a human has
looked. The public site enforces it through one rule, `catalog_queries.publishable(db)`,
which hides a blocked shop from every catalogue query, the trip comparison, the coverage
page, `/api/stats`, the sitemap and the IndexNow list (mechanism and rehearsal:
`main/docs/SEO.md`, "Which shops the site shows"); `verify-status` remains where the block
and the checks behind it are listed.

## The BLOCKED rule

A verification request is a request to a retailer, and this project does not send twenty
declared-bot requests into a wall. Per host: robots.txt is re-read through the shared policy,
the crawl delay is honoured (the slower of theirs and ours), and **the shop is abandoned at
the first refusal**, with one `BLOCKED` row recorded. A shop whose latest collection was
itself blocked is not verified at all. Dubai and the Heinemann family refuse our identity
today; their `read_one` raises before any request is made.

A blocked shop is governed by its **last-observed date**: its prices stay visible with their
date, "last checked" stays frozen, and the dashboard shows the flag. Nothing is invented and
nothing is hidden; the date does the talking.

## Sampling, version 1 (hardcoded)

Per shop: first a census of every listing behind the home page's unscoped featured eight
and the cheapest and dearest listing of the top-20 savings, plus every row that moved more
than 20 percent since its previous observation. Then uniform random from the rest up to N.
The seed is stored on the run, so a sample can be re-drawn. A shop that renders pages in a
browser is capped at 20 per run. One host at a time in one process; to run hosts in
parallel, start one process per `--source`.

Cadence (host cron, Stream E): `audit` nightly; `verify --n 20 --mode after_collection` the
**morning after** a collection, not immediately after it; `verify --n 200 --mode weekly`
across the visible sources; on demand before any client-facing figure is quoted.

## What the public site may say

- Per row: **"Seen <date>"**. Exists today. It is the observation's own timestamp.
- Per shop: **"Spot-checked <date>"** with a one-line sampling note, shown only after a
  verification run in which that shop had checks and **zero** correctness failures and no
  refusal (`verify.spot_checked`). Not before.
- **Never** a per-row "Verified". **Never** a pass rate. **Never** a list of failures.

No client-facing number is quoted without a verify run or a dated human check behind it.

## The first real figures

The audit's first snapshot, 2026-09-05 on that morning's dump (13,128 products, 33,242
observations): all fifteen metrics at their thresholds by construction. What it surfaced for
the review: a 7,000 ml "15yo Sherry Cask" and two beauty rows with five-litre sizes read
from a number in the name; three shops whose every row says in stock (Bordershop, Dubai,
Panama); ten duplicate groups that should merge; 122 brand spellings that fold together; the
four visible airports all older than the three-day mark (last read 25 August).

Verify has not yet run against a retailer. Its logic is exercised by the test pack and by a
rehearsal on the dev copy with a stand-in reader; the first real run follows the drinks
refresh and is the owner's call. The pass rates recorded then go here, beside the date.

## The human review (Mon 21 Sep)

Starts from the audit's lists: brand folds the table missed, the merge candidates the rules
would not settle alone (two barcodes under one key, a set against its bottle: one product or
two?), the largest spreads, the largest discounts, and five sample rows per shop with links.
Thirty pages opened by hand, the row compared with
the page, the result written down with the date. That review is the only thing that can
catch a parser reading the wrong field consistently, which is why it is a calendar item and
not a script.