# Decisions for rian ## Open ### From Stream Q (2026-09-05) 11. **Verify's two small additions to Decision 10, proceeding unless you object.** (i) A tenth verdict, `FETCH_ERROR` (timeout or 5xx): reported, never a failure, never a block. Without it a retailer's bad hour would either block a shop or vanish from the record; the rule is "store, do not log". (ii) `PRICE_MOVED` writes a new observation with `source_kind='verify'` (§10 #11, assumed yes). If you say no, one line in `verify.py` stops the write; the checks table keeps the live price either way. (iii) The block is not a flag on the source but an uncleared failing check (`cleared_at`, `cleared_by -> accounts.id`), so clearing is itself a recorded human decision. **No real verify run tonight**: it is a request to a retailer and the brief names none; first real run after the drinks refresh, `--n 20`, the next morning. ### From Stream A2 (2026-09-05) 11. **Seoul is possible; tell Adam?** The Shilla probe passed (`.logs/runs/shilla-probe-2026-09-05.log`, issues.md): a browser with our declared identity gets the full server-rendered product page. Options: (a) keep Seoul in the nineteen on the rendered fetch, 10 s per page, discovery via category pages after one more announced probe; (b) still swap it for a §10 #9 candidate. **Assumption: (a)**; the Mon 7 reply can say "Seoul confirmed possible, collector in build". Nothing is redone under (b); the probe cost two renders. Two conduct points for you to know: the page's own scripts live under a robots-disallowed path, so we read the document without them (it is complete without them); and the second render declared `challenges.cloudflare.com` as an asset host so the browser could load what the page asked for. No stealth, no UA change, no cookie carried over. **Two more facts for the ask:** their robots.txt now disallows a list of AI and rendering crawlers by name (ClaudeBot, GPTBot, CCBot, Google-Extended, CloudflareBrowserRenderingCrawler, Bytespider, ...); we are not named and `*` permits product pages, but it says something about intent. And their category grids are drawn by scripts under a disallowed path, so a compliant bot cannot read a listing page: `shilla.py` discovers through the home page and the related-product rails, 200 renders a run, which reaches part of the catalogue per run, not all of it. 12. **Which Shilla price tier is "the price"?** The page shows list ($43), discount ($25.37) and "price for online member" ($25.37). **Assumption: discount price = what a walk-up traveller pays; recorded as `price_type` "discount", list as `was_price`; the member tier is not published.** If you want the member price, it is one line in the collector. 13. **Singapore renders only with its tag manager loaded.** The storefront bundle crashes on a global that the Adobe Launch script (`assets.adobedtm.com`) defines; with that one host declared as an asset host the page draws and fetches its own inventory JSON, which we read. The analytics beacon hosts it then calls (sixteen of them, Adobe, Google, Facebook, Criteo, Bing, Clarity, New Relic...) stay undeclared and aborted, so nothing is reported to anyone. **Assumption: acceptable**; it is loading what the page needs to function, with our declared UA, and it is written in `changi.py` with the reason. If you disagree, Singapore has no honest path and joins the Cannes ask list. 14. **Before the first SIN observation is written, one hand check.** Open `https://www.ishopchangi.com/en/product/johnnie-walker-blue-label-1000ml-40--mp00089136` in your own browser in Traveller mode with "At Departure" selected and confirm S$271.10 (the collector's `price_type` "departure"; non-traveller delivery shows S$333.87). The targeted list (622 URLs: 404 drinks matched from 1,580 seed products, 326 beauty slugs across the seven CDG x ATH brands) is `.logs/runs/changi-targets-2026-09-05.md`; Stream A trims it Monday. At ten seconds a page that run is under two hours. ### From Stream A (2026-09-04 night) 10. **Shopify hosts refuse our identity at robots.txt.** All four (YUL, PTY, BOG, SAL) answer HTTP 403 to `robots.txt` for our bot UA and 200 to a plain curl (checked once each, 2026-09-04 23:xx). Under the ratified policy (#13: 401/403 is a refusal) `robots.py` now raises `SourceBlocked` before the first feed request, so their next run records `blocked` and they fall under the BLOCKED rule (last-seen 22 Aug, dated prices stay visible). Options: (a) accept: four shops launch on dated prices, and they join the Cannes ask list; (b) rian checks whether `/products.json` is also 403 now (one request, not done tonight); (c) ask the operators (ARI Montreal, Motta) for permission, which would be a `permission_record` under migration #1. **Assumption tonight: (a).** If you choose (c), nothing is redone; the record unblocks them. Dubai's robots.txt also 403s, which matches what Decision 3 already assumes. ### From the plan review (2026-09-04) — decide before Mon 7 0. **For E3 (sizing), a note not a decision (rian, 2026-09-05):** the production droplet may later become rian's general live-hosting box (client WordPress sites and other live sites), one well-powered server instead of several small ones. DFP is the first thing on it either way. E3 therefore gives the figure two ways (DFP alone; DFP with headroom for a shared box) and the shared-box choice gets its own security review when it is made: separate Postgres, separate Docker networks, no shared secrets, default-deny firewall, per-site isolation as on this server. 3. Bot identity hostname, on a zone you control (§10 #3). 4. Client surfaces on production: off, or behind a login (§10 #4). 5. Hosting figure and message to Adam (§10 #6). 6. Two Shilla-replacement candidates to probe (§10 #9). 7. Which page types ship 18 Sep, one sentence on the structure page (§10 #10). 8. Ratify: BLOCKED rule (#5), verify writes observations (#11), host cron (#12), robots policy 404 open / 401-403 refusal / 5xx stop (#13), pages-2+ Avolta rows keep or purge (#14). 9. The reply pass to Adam, Mon 7 (§10 #15 has the wording). Mark's half is done: he has the `/structure` page and the `/todo` page (sent 2026-09-05). ## Requests - **E / rian, sidecar hardening before the first production run (review 2026-09-05):** the browser sidecar's request filter blocks IP literals and private/loopback addresses by name, but a permitted retailer hostname that resolved to a private address would pass (DNS rebinding). Belt and braces: a `DOCKER-USER` iptables rule on whichever host runs the sidecar, dropping traffic from the `dutyfreeprofessor-render` network to the host's own addresses (`172.17.0.1`, the LAN address, Tailscale address) and to RFC1918 ranges. A firewall change, so rian applies it; on this server it lands in `/etc/iptables/rules.v4`. Low likelihood, cheap fix, and it makes the sidecar's isolation not depend on code. - **Whoever owns `catalog_queries.py` (B or A): hide a publication-blocked shop.** Decision 10 says correctness failures hide the affected rows or source until a human clears them. `verify.blocked_sources(db)` returns `{source_id: [checks]}`; the latest-observation queries should exclude observations whose run belongs to a blocked source (or show them as dated, not current). Until it lands the block is visible only in `verify-status`. (Stream Q, 2026-09-05) - **E, host cron:** `python -m app.cli audit` nightly (writes `audit_snapshots`); `python -m app.cli verify --n 20 --mode after_collection` the morning after any collection (one process per `--source` if run in parallel; never immediately after); `python -m app.cli verify --n 200 --mode weekly` once a week. All inside the app container. (Stream Q, 2026-09-05) - **A, three small touches by Stream Q (2026-09-05), already applied, for your review:** `cli.py` gained two lines (`register_quality(sub)`, the audit/verify subparsers live in `app/cli_quality.py`); `models/catalog.py` gained `CollectionRun.skip_counts` (JSONB, the per-reason split migration #2 asked for); `ingest.py` gained `count_skip()` and calls it from `record_rejection` and the zero-price line, so `skipped_no_price` stays the total and `skip_counts` carries the reasons. `tests/test_rejections.py` pins the INSERT in `record_rejection` to the `rejected_observations` model's columns. Nothing else in your files was touched. - **C, `SiteHeader.tsx` / `AnnouncementBar.tsx`:** `header_html()` in `app/services/seo.py` mirrors the header's default state so the server-rendered product page keeps the hero at the same height when React mounts (layout shift measured at 0). When the mega menu or the announcement bar changes structure or height, tell Stream B so the mirror is updated in the same deploy; the nav link list there is the same NAV array, verbatim. (Stream B, 2026-09-04) - **Rian, `.app.env`:** add `PUBLIC_BASE_URL=https://dutyfreeprofessor.demoing.info` (and the production host's value when it exists). From the B2 deploy on, production builds canonical, sitemap and JSON-LD URLs from it and no longer trusts the Host header; until it is set they are relative and the startup log warns. (Stream B, 2026-09-04) - **A2, `models/catalog.py`:** the `Location.visible` default-False line (model + `server_default`) is DONE by Stream A inside migration #1 (`0a1b2c3d4e5f`); nothing for A2 to add there. Rian: deploy order is dump, deploy, `alembic upgrade head`, then `python -m app.cli backfill accounts`. (Stream A, 2026-09-04) - **A2, `fx.py`:** add `"HKD": 7.8` and `"MXN": 18.5` to `FALLBACK_RATES` (both are collected currencies; a fallback run currently converts neither, so their `price_usd` is NULL). The rate and its source now persist per observation and per run (`fx_rate`, `fx_source`), which is where Stream A stopped. (Stream A, 2026-09-04) - **Rian, Sat evening deploy order:** dump; deploy; `alembic upgrade head`; then `python -m app.cli backfill ` for `accounts`, `vertical`, `stuck_runs`, `locations`, `orphan_tiles`, `image_provenance` (each idempotent; each prints one line). Live counts on 2026-09-04: 8 stuck runs, MEX currency MXN to USD, 25 orphan tiles, 1,357 image rows. - **Stream A, `robots.py`:** expose the pattern-to-regex step (`pattern_regex(pattern) -> str`) so `fetch.disallow_regexes()` can call it instead of mirroring the four lines; a test pins them equal until then. (Stream A2, 2026-09-05) - **Stream Q:** the R2 commit's regenerated doc maps (`ARCHITECTURE.md`, `RUNBOOK.md`) already list your uncommitted `audit.py` / `cli_quality.py` / `audit` command because `check.sh` regenerates them from the working tree; nothing to do, your commit carries the code. (Stream A2, 2026-09-05) - **A2, `fetch.py`:** DONE 2026-09-05. `from app.services.collectors.robots import BOT_NAME` and build `USER_AGENT` from it, so the name we honour and the name we send cannot drift (`tests/test_robots.py::TestIdentity` pins them equal). `robots.py` imports `fetch` lazily, so the import is not circular. (Stream A, 2026-09-04) - **Docs, `agents.md`:** the robots rule says "the matcher in the Heinemann collector does it right"; the matcher now lives in `collectors/robots.py` and every collector (ARI, Dubai, Shopify included) calls `check_allowed()` at the top of `collect()`. Point the rule there. (Stream A, 2026-09-04) ## Requests - **`main/app/version.py` → `0.32.0`** (Stream Docs, 2026-09-04). `CHANGELOG.md` has carried `## 0.32.0` since the Paris collector shipped, the running container reports `0.31.0`, and the new doc gate (`main/docs-check.sh`) fails on the disagreement. Until you bump it the gate is advisory inside `check.sh` (it prints, it does not fail); the moment they agree, a red gate is a red check with no further edit. Then also set the `brief.md` NOW-block version. - **`/srv/apps/professormediagroup/CLAUDE.md`**: one redirect line added at the top by Stream Docs (that folder is not a git repository, so there is no commit to review). ## Decided - **2026-09-05 (rian): §10 #1 scope vs capacity.** Everything stays in the plan. Unpaid back-office work (owner surfaces, login screens, comments step 2, reverification scorer and merge-queue behaviour, WARN doc gates, the seven-doc restructure) is ranked last and sits in the buffer week; it is pulled forward at the **Fri 11 checkpoint if every wave-2 quote task is green**. If time runs short, unpaid work drops before any quote line. Kept regardless: the migration #1–#2 columns and the auth middleware + route-inventory test (O1). - **2026-09-05 (rian): §10 #2 the Fri 11 milestone: option (a).** Fri 11 uses the quote's own bar: fresh drinks recollection minus the refusing shops, a one-off human check of thirty pages, sixteen public on the demo host (the five refusing shops on dated prices under the BLOCKED rule), production live if E lands it. Verify's tripwire becomes the 18 Sep gate. Adam hears this in the Mon 7 reply pass, including the dated-prices point.