# Opening hours: robots reads per airport operator domain (Stream G, task G1), Fri 2026-09-11 night

Opening hours are a new source class (rian, 11 Sep; `AWAY-PLAN.md` §3). Before any operator domain
is added, its own robots.txt is read, fresh, through the collectors' one policy
(`collectors/robots.py`: our named bot, modern wildcards, any matching Disallow is a no, 404 open,
401/403 a refusal, 5xx stop). Permission never transfers between domains. This is the whole network
budget of the task: one robots read per domain below; then, on a domain whose robots allow it,
discovery through the sitemap the robots file names (the index and at most one shard, as the
beauty discovery of 5 Sep did), then at most two directory or hours pages, each through `fetch()`
(honest UA, the host's crawl delay, a second and a half between requests where none is published).
Every URL fetched is listed under Reads below.
No retail collector runs. Nothing is written to any database by these reads.

| Airport | Operator domain read |
|---|---|
| ATH | www.aia.gr |
| BCN, MAD | www.aena.es (one domain, two airports) |
| BOG | eldorado.aero |
| CDG | www.parisaeroport.fr |
| DUB | www.dublinairport.com |
| DXB | www.dubaiairports.ae |
| EZE | www.aeropuertosargentina.com |
| HKG | www.hongkongairport.com |
| ICN | www.airport.kr |
| JFK | www.jfkairport.com |
| LHR | www.heathrow.com |
| MEX | www.aicm.com.mx |
| PTY | www.tocumenpanama.aero |
| SAL | www.aeropuertoelsalvador.gob.sv (then www.cepa.gob.sv if that host does not resolve) |
| SIN | www.changiairport.com |
| YUL | www.admtl.com |
| YYZ | www.torontopearson.com |
| ZRH | www.zurich-airport.com |

## Verdicts
(appended by the reads below)

| Domain | robots.txt | Verdict |
|---|---|---|
| www.aia.gr (ATH) | HTTP 403 | **refused**: the host refuses our reader at robots.txt; ATH goes to hand population, never retried in this run |
| www.airport.kr (ICN) | 200, `Disallow: /` for everyone | **disallowed**: ICN goes to hand population |
| www.aeropuertoelsalvador.gob.sv (SAL) | host does not resolve | **unreadable**; the second operator host www.cepa.gob.sv answered HTTP 403 at robots.txt: **refused**; SAL goes to hand population |
| www.aena.es (BCN, MAD) | 200, 27 rules (`*/airports/*`, `*/aeropuertos/*`, VIP and claims paths), no Crawl-delay | allowed outside those paths |
| eldorado.aero (BOG) | 200, flight detail and `/_nuxt/` | allowed |
| www.parisaeroport.fr (CDG) | 200, no rules | allowed |
| www.dublinairport.com (DUB) | 200, `/api/` | allowed |
| www.dubaiairports.ae (DXB) | 200, `/Sitefinity/`, culture query | allowed |
| www.aeropuertosargentina.com (EZE) | 200, no rules | allowed |
| www.hongkongairport.com (HKG) | 200, the old `/eng/` `/chi/` `/gb/` trees and `/iwov-resources/*` | allowed under `/en/` |
| www.jfkairport.com (JFK) | 200, no rules | allowed |
| www.heathrow.com (LHR) | 200, flight queries, search, rewards, myheathrow | allowed |
| www.aicm.com.mx (MEX) | 200, no rules | allowed |
| www.tocumenpanama.aero (PTY) | 200, no rules | allowed |
| www.changiairport.com (SIN) | 200, `/en/prog/` and `/bin/` | allowed |
| www.admtl.com (YUL) | 200, one password path | allowed |
| www.torontopearson.com (YYZ) | 200, app folders and payment paths | allowed |
| www.zurich-airport.com (ZRH) | 200, `/api/*`, `/sitemap.xml`, flight information | allowed; **its sitemap is disallowed, so no sitemap read there** |

No host publishes a Crawl-delay; the reads below keep a second and a half between requests.

## Reads

Discovery (the sitemap named by each robots file, at most one index and one shard per domain):
- https://www.aena.es/sites/sitemaps/sitemap_index.xml
- https://www.aena.es/sites/sitemaps/en_GB.xml
- https://eldorado.aero/sitemap.xml
- https://www.dublinairport.com/sitemap.xml
- https://www.dubaiairports.ae/sitemap/sitemap-index.xml
- https://dubaiairports.ae/sitemap/sitemap.gz
- https://www.aeropuertosargentina.com/sitemap.xml
- https://www.jfkairport.com/sitemap.xml
- https://www.heathrow.com/sitemap-index.xml
- https://www.heathrow.com/sitemap.xml
- https://www.aicm.com.mx/sitemap.rss
- https://www.changiairport.com/sitemap-index.xml
- https://www.changiairport.com/cn/sitemap.xml
- https://www.admtl.com/vforcesite/sitemap_index
- https://www.admtl.com/vforcesite/sitemap_en
- https://www.torontopearson.com/sitemap_yyz_en.xml
Pages (one directory or duty free page per domain; Hong Kong answered HTTP 403 on its shopping page and Paris an Incapsula challenge behind a 200, both recorded as refusals; the AENA detail address derived from the listing answered 404, the domain's second and last page):
- https://www.heathrow.com/at-the-airport/shops-a-z/world-duty-free
- https://www.torontopearson.com/en/while-you-are-here/toronto-airport-shops/duty-free-t1-after-intl-e76
- https://www.admtl.com/en-CA/directory?service=Duty_Free
- https://www.dublinairport.com/at-the-airport/shopping/shops-in-the-terminals
- https://www.jfkairport.com/dine-shop-relax/shops
- https://eldorado.aero/en/establecimientos/tiendas
- https://www.aena.es/es/adolfo-suarez-madrid-barajas/servicios-del-aeropuerto/tiendas-y-restaurantes/t1.html
- https://www.changiairport.com/en/dine-and-shop/shop-directory.html
- https://www.hongkongairport.com/en/shop-dine/shopping/
- https://www.parisaeroport.fr/en/passengers/shops-services/shops
## The collect run against dfp_g (rehearsal database, nothing on staging or production)
`python -m app.cli hours collect` with `DATABASE_URL` pointed at `dfp_g`: the three platforms that read (LHR, DUB, YYZ), the pages above plus Toronto's store pages, a second and a half apart. Outcome table below.
Second read of YYZ only, after the parser fix (the time item read whole; the twelve-hour clock): the sitemap and the eleven duty free store pages again.
One more page read, YYZ's duty-free-gucci store page, to see what the host served the second run.

**What the Toronto re-read found:** the host answered the store pages with HTTP 200 and a
"Radware Captcha Page" (H1 "We apologize for the inconvenience"), so the second run read no
hours; the one page fetched afterwards to see what was served confirmed it. **A block is a
refusal:** recorded here, no further request to www.torontopearson.com tonight, nothing worked
around. The collector keeps its first-run reading in `dfp_g` (row 3) and now declares a ten-second
pace and recognises the captcha page as a refusal; whether the host serves it again is for the
next announced run to find, and if it does YYZ goes to hand population.

## Outcome per airport (the state this session leaves)

| Airport | Outcome | Why |
|---|---|---|
| LHR | **collected** (`heathrow.py`) | www.heathrow.com's World Duty Free page: 14 stores across 4 terminals with clocks |
| DUB | **collected** (`dublin.py`) | www.dublinairport.com's shops page: Dublin Airport Duty Free T1 and T2, WHSmith Duty Free T1 and T2 |
| YYZ | **collected once, then refused** (`toronto.py`) | 11 store pages read on the first run; a Radware captcha on the second; recorded |
| BCN, MAD | hand | www.aena.es allows the path; the terminal listing names the duty free shops without hours; the detail address answered 404 (the domain's two-page budget) |
| ATH | hand | www.aia.gr refuses robots.txt (403) |
| ICN | hand | www.airport.kr disallows everything |
| SAL | hand | neither operator host is readable (no DNS; 403) |
| CDG | hand | www.parisaeroport.fr answers an Incapsula challenge behind HTTP 200 |
| HKG | hand | www.hongkongairport.com answers 403 on its shopping page |
| SIN | hand | www.changiairport.com's shop directory is drawn in the browser (1.3 MB, no hours in the document) |
| YUL | hand | www.admtl.com's directory is drawn in the browser (five characters of text) |
| JFK, BOG | hand | the shop pages read fine and carry no hours |
| DXB, EZE, MEX, PTY | hand | no shop directory found in the sitemap (or none named); nothing fetched beyond it |

No retail collector ran. Nothing was written to staging or production; the rehearsal rows are in
`dfp_g` only and go with it when the database is dropped.