# Stream A2: the rendered-fetch capability, then Singapore and Seoul
> **Status 10 Sep:** R1, R2, R3 and R5 landed (the sidecar, `render()`, the Shilla probe, the Seoul collector, handoff 5 Sep); R4 (Singapore on the targeted set) is blocked on rian's `decide-singapore-answered-403-on-the-19th-page-of-its-first-run`. The body below is the brief as asked.
**Runs:** now. Decided 2026-09-04 (build plan v2, Decision 5). Report to rian before any
network run against either retailer.

## Goal
Two of the three promised collectors need a real browser, not a text fetch. Build that
capability once, honestly, as a per-URL tool the collectors reach for; then prove it on
Singapore and attempt Seoul. Outcome for Adam: "one collector delivered (Paris), one built
and collecting (Singapore), one confirmed possible or confirmed refused (Seoul)".

## What was verified on 2026-09-04 (do not re-derive; do re-check robots each run)
**Singapore, iShopChangi.** `robots.txt` disallows only `/search*`, `/cart/`, `/checkout/`,
`/myaccount/`, `/registration/`, `*/thank-you/*`; product pages `/en/product/<slug>-mp<id>`
are permitted. Sitemaps: `/en/sitemap-index.xml` → `/en/sitemap-products.xml` (29,771 URLs;
~2,508 beauty-looking, ~1,650 drinks-looking by slug). A product page as served is a 62 KB
shell with 71 characters of body text, no price, no JSON-LD. The React bundle
(`/etc.clientlibs/ishop/clientlibs/react-app.min.*.js`) calls
`/bin/cagcommerce/webservices/v2/cag/products/search`, `.../cag/fetchProductLists`,
`.../cag/pdp/inventory.json`, `.../cag/product/promotions`; an axios interceptor attaches a
param `s` loaded from client storage key `93gma2jf`. A bare GET to the search endpoint 301s
to `/en/home`. **So the data arrives only after their JavaScript runs.** A browser that loads
a permitted product page and reads the rendered DOM (or observes the API responses the page
itself makes) is the honest path. Do NOT try to mint or replay the `s` token yourself.

**Seoul, Shilla.** `robots.txt`: `Allow: /` for `*` (it disallows Amazonbot,
Applebot-Extended, Bytespider). Home `/estore/kr/en/?uiel=Desktop` serves 674 KB with USD
prices visible. **Every product page** (`/estore/kr/en/p/<id>`) answers HTTP 403 with
`cf-mitigated: challenge` ("Just a moment…"), consistently, even after a cookie-keeping browse
from the home page. `/estore/kr/en/sitemap.xml` is a 128-byte empty document. This is a
Cloudflare JavaScript challenge. A real browser may pass it simply by being a browser. **If it
does not, stop.** We do not solve challenges by any other means.

## Deliverables, in this order
1. **`fetch.render(url, *, delay)` in `app/services/collectors/fetch.py`** with the SAME
   contract as `fetch()`: honest `USER_AGENT` (repointed at the bot page when Stream A lands
   it), robots re-checked per run with the strict matcher (`heinemann._robots_allows`, or the
   shared `collectors/robots.py` if Stream A has already moved it), crawl-delay honoured, content
   validated never status, `SourceBlocked` on a challenge page or an empty render. Returns the
   rendered HTML and, optionally, the JSON bodies of same-origin API responses the page made
   (so the SIN collector can read `products/search` output without calling it itself). Runs
   in a **sidecar container** (`dfp-browser`, its own service in `docker-compose.yml`):
   Playwright (Apache-2.0) with its bundled Chromium, `mem_limit: 2g`, `cpus: 1.0`, no host
   network, reachable only from the app container. Never `nodriver`/`zendriver` (AGPL) and never
   a paid unblocking service. Host has ~12 GB available; the app container stays at 1 GB.
2. **Singapore collector** `extime`-style module `changi.py`: sitemap discovery filtered to the
   targeted product set first (Decision 4: the ~40 beauty targets plus drinks that exist at
   other airports), full catalogue only later and only with rian's crawl token. Per product
   page: render once, read name, brand, price (SGD), size, GTIN if present, stock, category,
   image URL (store the URL only). Family dedupe as in `extime.py`. Emit `RawListing`s.
   Location `SIN`, currency `SGD`, retailer iShopChangi (operator: Changi Airport Group).
   `FALLBACK_RATES` needs SGD. Tests with a trimmed real rendered fixture.
3. **Seoul attempt** `shilla.py`: one rendered fetch of one product page under the crawl-delay.
   If the challenge clears → proceed as a normal collector (location `ICN`, currency USD as
   displayed). If `SourceBlocked` → record it in `.logs/issues.md` with the headers, set the
   source `enabled=false`, and STOP. Rian then proposes a replacement airport to Adam.
4. **Cost discipline.** A render is ~50× a text fetch. The browser is a per-URL tool for the
   comparisons that matter and for stale high-value refreshes; it is never a mode for a whole
   source. Log render counts per run.
5. **Docs**: `main/docs/COLLECTORS.md` gains a "rendered fetch" section (when to use it, the
   sidecar, the stop rule) and the two sources' posture rows; a handoff line per session.

## Owns
`app/services/collectors/fetch.py` (the `render` addition), new `changi.py`, new `shilla.py`,
`registry.py` (two lines), `fx.py` (SGD), the `dfp-browser` service in `docker-compose.yml`,
their tests. Coordinate with Stream A on `robots.py` if it exists; do not duplicate it.

## Must not touch
Anything under `web/`, `seo.py`, `ingest.py`, `normalize.py`, `taxonomy.py` (ask Stream A).

## Ground rules for this session
- Read first: `agents.md`, `brief.md`, this brief, then `extime.py` (the closest model) and
  `fetch.py`. Do NOT read all of `.logs/handoff.md`.
- **Conduct rules are absolute** and are a selling point: honest UA, robots including
  wildcards, crawl-delay, never log in, never accept terms, stop when refused, no stealth.
- Announce every network run to rian first and log it under `.logs/runs/`. A deploy kills a
  running collection; you do not hold the deploy token.
- Compose change (the sidecar) needs a deploy: build and test locally, then tell rian you are
  green. No Alembic migration in this stream.
- Before handing back: `../.venv-dev/bin/pytest tests -q` passes, commit prefixed `A2:`, one
  line in `.logs/handoff.md`, COLLECTORS.md updated.
- House style for anything the client reads: no em dashes.

## Added 2026-09-04 after the plan review (binding)
- **Network isolation.** The browser service joins a second compose network `dfp-render` holding
  only `app` and `browser`; it is NOT on `dfp` (the database is unreachable from it) and carries
  no `env_file`. Runs as the image's non-root user with the Chromium sandbox on: `cap_drop:
  [ALL]`, `security_opt: [no-new-privileges:true]`, `shm_size: 1g`.
- **Egress allowlist.** `context.route('**/*')` aborts any request to an IP literal, any
  RFC1918/loopback/link-local address, and any host outside the source's declared allowlist
  (its origin plus named asset hosts); log aborted hosts once per run. This also drops trackers.
- **Floor delay.** At least 10 s between renders on any host that publishes no Crawl-delay
  (Changi publishes none). Log a per-run render cap and count.
- **No dialog click-through.** Never accept, dismiss or select in a cookie, age, terms or
  flight/shopper-mode dialog. If prices appear only after one, that is a STOP and an issues.md
  note, not a workaround.
- **Price tier.** Record which price tier the default render shows; verify one product by hand
  against the page before the first SIN observation is written; add a `price_type` value for it.
- **Location default.** Your sidecar commit also changes `models/catalog.py` Location `visible`
  default to `False` (client-side SQLAlchemy default, one line, no migration). You are granted
  that one line. Paris appeared publicly by this default on 3 Sep; Singapore must not.
- **Shilla probe first.** Run the single Shilla rendered probe immediately after `render()`
  works, before the Singapore collector, so rian can answer Adam early.
- **Targeted set bootstrap.** Seed from drinks stocked at two or more of the sixteen plus the
  CDG x ATH beauty overlap brands; record the list under `.logs/runs/`; Stream A refines Monday.
- **COLLECTORS.md.** Create `main/docs/COLLECTORS.md` if absent, opening with "Sources of truth:
  app/services/collectors/, app/cli.py", then add the rendered-fetch section and the SIN/ICN
  posture rows. The docs session adds the generated table later.
- **robots.py.** Import the shared matcher from `collectors/robots.py` once Stream A lands it
  (this weekend); until then do not edit `fetch.py`'s robots handling yourself.