--- type: run --- # Collection sweep, part two, 18 Sep 2026 night (staging) — the eleven the reboot left The 19 Sep window (`window-2026-09-19-sweep.md`) was cut off by the host reboot with six of eight sources landed. This window finishes it: **the ten Avolta airports and Extime Paris**. Nothing else is re-collected — shilla, the four Shopify shops and The Loop all ran clean and are hours old. Skipped on purpose, unchanged: the five refused sources `sweep plan` lists (never contacted), and **dubai-duty-free**, whose 403 to our declared identity is not a recorded run (running list: `issue-dubai-duty-free-is-listed-as-reachable-by-sweep-plan-bec`). Rian settles that; no bypass. ## Why this one runs in parallel, when the last one ran in sequence The last window put the ten Avolta airports through **one process in sequence**, and it never got past the first. The reason is in the hosts' own robots.txt, read through the collector's reader: | host | published Crawl-delay | our configured delay | |---|---|---| | `athens.shopdutyfree.com` (and the other eight `*.shopdutyfree.com`) | **60s** | 1.00s | | `london-heathrow.worlddutyfree.com` (LHR) | **30s** | 1.00s | | `www.extime.com` | none published (our 1.5s floor applies) | 1.00s | `Robots.delay_for` takes the slower of theirs and ours, so every Avolta request waits a full minute. ATH's 2h20m was not slow code; it was obedience — 126 held-URL rereads at 60s each. Ten airports end to end is roughly **27 hours**. **Parallel does not speed up any host, and that is the point.** Each process owns one host and still waits that host's 60 seconds between requests, so the per-host rate is exactly what the host asked for. Nine hosts at one request per minute is 0.15 req/s across the whole platform. The nine `*.shopdutyfree.com` names resolve to the same two Akamai addresses, so they share an edge — which is an argument about aggregate load, and 0.15 req/s is not aggregate load. LHR and Extime are separate domains entirely. **The binding constraint is ours, not theirs: collectors run inside the app container, under its 1 GB cgroup, alongside uvicorn.** That is the resource they can genuinely step on — and the night this window opens, the host itself went down to memory pressure. So the wave is measured, not assumed: a first group, `docker stats` watched, the rest added only on the evidence. Logs in `sweep-2026-09-19b/.log`. Watch it on `/collectors#live` (the tab now agrees with the approval gate about what is dead — it did not before tonight). **While this window is open: no deploy of staging** (a deploy recreates the container and kills every collector in it — which is exactly how the last one ended). ## 23:42–00:00 — the first attempt OOMed the container, and what it taught Eight collectors were launched on the measured figure of ~73 MB each. That figure was taken seconds after launch, before any of them had accumulated anything, and it was wrong: **a collector settles at 140–160 MB and keeps growing**. At 23:48:46 the cgroup hit its 1 GB ceiling and the kernel fired two `CONSTRAINT_MEMCG` kills, taking `avolta-ath` and `avolta-hkg`. **What the cap bought.** The kills stayed inside the container and the kernel chose the largest tasks, which were collectors — uvicorn survived and the site never stopped answering. This is the same failure the host suffered on 18 Sep with no boundary at all to contain it. **Four runs were left `running` with 50–100 prices committed.** Note what this says about the boot rule shipped earlier tonight: it catches a run whose *container* died, not a collector killed while the container lives. Those four would have held the review gate for 24 hours. (They cleared on the redeploy, because a recreate makes every earlier run pre-boot — but that is the fix working by luck of sequence, not by covering the case. Worth a real per-run liveness mark.) **The cap is now 4 GB** (`docker-compose.yml`, reasoning in a comment beside it), approved by rian, who has the host memory to spare. All eleven sources then launched together and sat at **1.08 GB** within a minute — above the old ceiling before doing any work, which is why eleven in parallel was never possible at 1 GB. `guard.sh` traces memory to `memory-trace.tsv` every two minutes and sheds the heaviest collector if the container passes 3.4 GB, so an unattended overnight run cannot repeat this. The trace is also the evidence for the running-list decision `decide-where-do-the-crawls-run-once-the-site-is-live-this-host`: production is a 4 GB / 2 vCPU droplet (verified), where a sweep of this shape does not fit beside the web app and Postgres.