---
type: run
---

# Window W1, 19 Sep 2026 (staging, after deploy D1): the first runs of the three new collectors

Written by Stream AW1 (task AW1.5) before the window; **launched by the orchestrator after deploy
D1, never by a build agent** (`AWAY-2026-09-19.md` sections 1 and 4). Three sources, chosen by the
scout of window `window-2026-09-19c-scout.md` and configured in AW1.2 to AW1.4, collect for the
first time, in parallel, each waiting its own host's published crawl delay. Nothing else is
collected: not the twenty sources that already have a run, not any host `sweep plan` lists as
refused, not Dubai, not Singapore.

| source | host | published delay | paging | expected duration | why that long |
|---|---|---|---|---|---|
| `avolta-las` | `lasvegas.shopdutyfree.com` (LAS, USD) | Crawl-delay 60 | paged (`?p=2` allowed) | one to three hours | 8 to 9 drinks categories paged, the beauty page one, up to 40 multi-size product pages, one request a minute |
| `avolta-lgw` | `london-gatwick.worlddutyfree.com` (LGW, GBP) | Crawl-delay 30 | page-one-only (`Disallow: /*?`, as Heathrow) | thirty to sixty minutes | page one of each category at 30 s; nothing held yet, so no re-read on a first run |
| `avolta-hel` | `helsinki.shopdutyfree.com` (HEL, EUR) | Crawl-delay 60 | paged (no Disallow at all) | one to three hours | as Las Vegas |

The window is expected to last **about three hours** (the two 60 s hosts bound it; Gatwick ends
first). The delay is the slower of theirs and ours (`Robots.delay_for`), read from each host's
robots.txt at the start of its run; a robots read that now refuses ends that run as `blocked` and
the host is never asked again.

**What a first run creates:** its `sources`, `retailers` and `shops` rows
(`ingest.ensure_retailer_and_shops`, in a savepoint, tolerating a concurrent creator), then its
listings and price observations. `Shop.visible` defaults false, so **every new airport is HIDDEN**
until rian shows it (`app.cli shops --show <CODE>`); nothing in this window shows one. The
airport `Place` rows follow by hand after the window (`backfill places`, below). No row is
created by hand.

**While this window is open: no deploy of staging.** A deploy recreates the app container and
kills every collector in it, which is how the 19 Sep sweep ended; an Avolta host at 60 s costs
hours to redo. Code work continues; the deploy point D2 waits for `finish.sh`.

## Memory, and the figures the scripts carry

Collectors run inside the app container, whose cgroup cap is **4 GB** (`docker-compose.yml`,
raised on 18 Sep after eight collectors OOMed the 1 GB cap in six minutes). The sweep of 19 Sep
measured it (`sweep-2026-09-19b/memory-trace.tsv`): ten collectors sat at 1472 to 1577 MiB,
about 150 MiB each once settled, a lone one ended at 240 MiB, and a collector grows as its run
accumulates. So, in `new-collectors-2026-09-19d/`:

- `admit-queue.sh`: `CEILING_MIB=3000` (the sweep's copy said 620, written for the 1 GB cap):
  admit only while one more collector, up to about 300 MiB with growth, still fits under the
  guard's shed line. Three collectors are expected at 700 to 900 MiB in all, so the ceiling
  binds only if the container already carries a run it should not (the gate below says none
  may). It refuses any slug that is not one of the three. Its memory reader now scales a
  `GiB` figure from `docker stats` (the sweep's copy read `1.08GiB` as 1 MiB, a bug that would
  have admitted into a full container; tested on the four strings the command prints).
- `guard.sh`: traces memory every two minutes to `memory-trace.tsv`, sheds the heaviest of the
  three above **3400 MiB** (`SHED_MIB`, unchanged: act before the kernel does), and ends when the
  queue is done and none of the three is left.
- `finish.sh`: waits for the three to end, appends the closing section to this file (run rows,
  rejections by reason, barcode share per new shop, the shops' HIDDEN state), and tells rian
  with `srv-gw notify --from "AW1 W1"`.
- All three match the process names **exactly** (`avolta-las|avolta-lgw|avolta-hel`), never
  `avolta*`, so a collector of another window is neither counted, shed nor waited for.

Logs: `new-collectors-2026-09-19d/<slug>.log`, `admit-queue.log`, `guard.log`,
`memory-trace.tsv`. Watch it on `/collectors#live`.

## The commands, after D1 (the orchestrator runs these, in this order)

```bash
D=/srv/apps/dutyfreeprofessor/.logs/runs/new-collectors-2026-09-19d   # the main checkout, after D1's fast-forward

# 1. The gate: no source running, no collect process in the container. Both must print 0.
docker exec dutyfreeprofessor-app python -m app.cli status | grep -c ' running '
docker exec dutyfreeprofessor-app sh -c 'for p in /proc/[0-9]*; do tr "\0" " " < $p/cmdline 2>/dev/null; echo; done' | grep -c 'app.cli collect'

# 2. The three are registered in the deployed image (three lines, one per slug).
docker exec dutyfreeprofessor-app python -c "from app.services.collectors.registry import COLLECTORS; print('\n'.join(s for s in COLLECTORS if s in ('avolta-las','avolta-lgw','avolta-hel')))"

# 3. Announce, then launch: the queue admits one every 90 s, the guard and the finisher watch.
srv-gw notify --from "AW1 W1" "W1 open: avolta-las, avolta-lgw, avolta-hel collecting for the first time; no deploy until finish.sh reports (about three hours)"
setsid nohup bash "$D/admit-queue.sh" avolta-las avolta-lgw avolta-hel > /dev/null 2>&1 < /dev/null &
setsid nohup bash "$D/guard.sh"  > /dev/null 2>&1 < /dev/null &
setsid nohup bash "$D/finish.sh" > /dev/null 2>&1 < /dev/null &

# 4. A minute later: three admitted, three running.
cat "$D/admit-queue.log"
docker exec dutyfreeprofessor-app python -m app.cli status | grep -E 'avolta-(las|lgw|hel)'
```

`finish.sh` appends the closing section to **the main checkout's copy of this file** and the
scripts write their logs beside themselves there. Until the window is closed and that copy is
folded back (copy the closed file into the worktree and commit it with the next stream's work,
or leave it as deliberate dirt as the sweep's is), **no stream commits a change to this file or
under `new-collectors-2026-09-19d/`**, or the fast-forward at D2 fails.

## How to read the outcome (after the closing section is written)

1. `docker exec dutyfreeprofessor-app python -m app.cli status`: three rows `avolta-las`,
   `avolta-lgw`, `avolta-hel`, each `ok (N prices)`. A `blocked` row is a refusal (robots, or
   the edge to our declared identity): record it, never retry. An `error` row is ours: read
   `<slug>.log`.
2. The run rows and the rejections by reason for those run ids, and the barcode share per new
   shop (`listings` joined to `product_variants.gtin`): `finish.sh` writes all three tables
   into the closing section; the same lines by hand are
   ```bash
   docker exec dutyfreeprofessor-db psql -U dfp -d dfp -tA -F' ' -c "SELECT s.slug, r.id, r.status, r.prices_written, r.products_seen, r.skipped_no_price FROM collection_runs r JOIN sources s ON s.id=r.source_id WHERE s.slug IN ('avolta-las','avolta-lgw','avolta-hel') AND r.started_at > now() - interval '12 hours' ORDER BY s.slug, r.id;"
   docker exec dutyfreeprofessor-db psql -U dfp -d dfp -tA -F' ' -c "SELECT s.slug, o.stage, o.reason, count(*) FROM rejected_observations o JOIN collection_runs r ON r.id=o.run_id JOIN sources s ON s.id=r.source_id WHERE s.slug IN ('avolta-las','avolta-lgw','avolta-hel') AND r.started_at > now() - interval '12 hours' GROUP BY 1,2,3 ORDER BY 1,4 DESC;"
   docker exec dutyfreeprofessor-db psql -U dfp -d dfp -tA -F' ' -c "SELECT sh.code, count(*), count(l.listed_gtin), count(pv.gtin), round(100.0*count(pv.gtin)/nullif(count(*),0)) FROM listings l JOIN shops sh ON sh.id=l.shop_id JOIN product_variants pv ON pv.id=l.variant_id WHERE sh.iata IN ('LAS','LGW','HEL') GROUP BY sh.code ORDER BY sh.code;"
   ```
   What to expect: the scout's grids carried 33% (LAS), 22% (LGW) and 20% (HEL) barcode SKUs;
   a whole catalogue will differ from one grid, but a share near zero means the SKU shape
   changed and `gtin_from_sku` no longer reads it. Heathrow and Athens stand at 34 to 35% on
   the variant column. A shop an order of magnitude off its peers on price is a bug to report,
   not a bargain (agents.md).
3. `docker exec dutyfreeprofessor-app python -m app.cli shops`: `LAS`, `LGW` and `HEL` present
   and **HIDDEN**. Nothing here shows them.
4. The airport places, once nothing is running (`backfill` refuses while a collection runs):
   ```bash
   docker exec dutyfreeprofessor-app python -m app.cli backfill places --check   # expect 3 created, 3 pointed
   docker exec dutyfreeprofessor-app python -m app.cli backfill places
   ```
5. **The airport page check, on a restored copy only** (never staging: showing a shop there is
   rian's). Rehearsed on 19 Sep against the 19 Sep nightly with Heathrow standing in for the
   new shops: a hidden shop renders no page at all (`airport_head` answers `None`), and once
   shown the H1 is `Duty free at <shop name>`. From `main/` in the worktree:
   ```bash
   docker exec dfp-devdb psql -U dfp -d postgres -c "CREATE DATABASE dfp_w1check;"
   docker exec -i dfp-devdb pg_restore -U dfp -d dfp_w1check --no-owner --no-privileges < /srv/apps/dutyfreeprofessor/backups/<the first nightly after W1>.dump
   export DATABASE_URL=postgresql+psycopg://dfp:dfp@127.0.0.1:5433/dfp_w1check
   ../.venv-dev/bin/python -m app.cli shops --show LAS LGW HEL           # on the copy only
   ../.venv-dev/bin/python - <<'PY'
   from app.db import SessionLocal
   from app.services import seo
   SHELL = '<html><head></head><body><div id="root"></div></body></html>'
   for iata, h1 in (("LAS", "Duty free at Las Vegas Harry Reid"), ("LGW", "Duty free at London Gatwick"), ("HEL", "Duty free at Helsinki-Vantaa")):
       with SessionLocal() as db:
           head = seo.airport_head(db, iata)
           html = head.apply(SHELL, "https://dutyfreeprofessor.com")
           print(iata, head.canonical_path, f'<h1 class="airports-hero__title">{h1}</h1>' in html)
   PY
   docker exec dfp-devdb psql -U dfp -d postgres -c "DROP DATABASE dfp_w1check;"
   ```
   Expect `/airports/las-vegas-las`, `/airports/gatwick-lgw-london`, `/airports/helsinki-vantaa-hel`,
   each `True`. (A nightly dumps at 03:15; before then, dump staging by hand with
   `docker exec dutyfreeprofessor-db pg_dump -U dfp -Fc dfp > backups/dfp-after-w1-<date>.dump`.)

The closing section below is `finish.sh`'s to write.

## Closed 2026-09-19 14:16: the three first runs ended

3 run(s) ok, 0 not, 1180 prices written. Peak container memory 549 MiB of the
4096 MiB cap (the trace is `memory-trace.tsv`; anything shed is in `guard.log`).

Run rows (slug, run id, status, prices, seen, skipped, error):
```
avolta-hel 182 ok 442 442 0 -
avolta-las 180 ok 265 265 4 -
avolta-lgw 181 ok 473 473 3 -
```
Rejected observations by reason (slug, stage, reason, count):
```
avolta-las ingest gtin_size_veto 4
avolta-lgw ingest gtin_size_veto 3
```
Barcode share per new shop (code, listings, listed_gtin, variant_gtin, pct):
```
HEL 442 122 135 31
LAS 265 78 102 38
LGW 473 103 146 31
```
Shops (every new one must read HIDDEN):
```
HEL HEL HIDDEN
LAS LAS HIDDEN
LGW LGW HIDDEN
```

Next, by hand: `app.cli backfill places --check` then `app.cli backfill places` (one airport
place per new IATA code), then the airport page check on a restored copy (the section above).
Staging may be deployed again (D2).