# Request Firewall — REMOVED in bw-dev 1.18.0 (post-mortem)

> **This feature was removed because it caused a production outage.** Everything
> below the post-mortem is retained as historical design context only. Do not
> re-implement load-shedding without satisfying the hard gate at the end.

## What went wrong (aismartventures.com, 2026-06-11)

The generated mu-plugin `wp-content/mu-plugins/bw-request-firewall.php` did a
**blocking `flock()`** (shared lock, read path) on its file-backed pressure
store (`bw-request-firewall-data/store.json`) on **every non-allowlisted
request**, to read the pressure flag.

On Flywheel's networked/overlay filesystem, `flock()` is slow and serializes.
Under real concurrency, PHP workers pile up waiting on the lock → the worker
pool is exhausted → the site goes down. The database is idle the whole time
(workers are stuck in `flock()`, not queries); only a PHP-pool restart clears
it. The write-path locks (sheds) make it worse, not better.

It passed "validation" in a dev container because `flock()` on a **local**
filesystem is instant. This is a **dev-vs-prod filesystem gap** — the bug is
invisible except under real load on the real host. That is the root lesson:
load-tested-on-real-infra-or-it-didn't-happen.

These sites also have **no fast shared-memory store** — Flywheel loads the
`memcached` extension but runs no server, and APCu is absent — so the adaptive
engine falls back to the file store, which is the thing that wedges the host.
There is no safe shared-store option on these sites.

## What replaced it: nothing (the feature is gone)

1.18.0 makes the `request_firewall` module **inert** (registers no hooks,
generates no files, zero per-request filesystem I/O) and ships a one-time,
idempotent, fail-safe **decommission** that — on every site, automatically, even
on an SFTP/rsync update — deletes the stub mu-plugin and its data dir and
disables the module. See `CHANGELOG.md` (1.18.0) and the module file header.

The real problem on these sites is **resource/stack** (heavy Kadence + ACF +
many plugins, no object cache), not 404 floods. The fix is host resources +
object caching + a lighter build — **not an in-PHP firewall**.

## Hard gate on ANY future firewall code (non-negotiable)

- **No blocking locks and no per-request filesystem I/O on the hot path. Ever.**
- Any per-request logic must be **load-tested on real Flywheel infrastructure**
  (or a filesystem that reproduces slow `flock`) under concurrency — never a dev
  container. The whole outage came from trusting dev-container validation.
- **Adaptive shedding may only exist where a verified shared-memory backend is
  present** (a live set/get probe must succeed). On hosts without one, the
  shedding code must not run at all.
- If anything is ever kept, keep **only** a stateless static-deny layer (regex
  on the request path → 410 for unambiguous junk like `.env`, `.sql`,
  `/vendor/...phpunit`) — no store, no `flock`, no per-request I/O — and
  re-justify + re-test it before redeploy.

---

# (Historical) Original design — Request Firewall (bw-dev module `request_firewall`)

Autonomous load-shedding that stops the bot-flood outage pattern: a flood of
requests for URLs that **don't exist as real pages**, each forcing a full
~2.5s WordPress bootstrap to return a 404, until the PHP worker pool is
exhausted and every visitor 503/504s.

## What it does

bw-dev generates and maintains a tiny, self-contained mu-plugin at
`wp-content/mu-plugins/bw-request-firewall.php` that screens every request
**before WordPress loads plugins, the theme, or runs the main query**. Three
layers, each able to exit early:

1. **Static deny + per-site rules** — junk that's never legitimate on any WP
   site (backup/secret/VCS/RCE-probe paths) → instant `410`. Plus per-site
   `deny`/`redirect` rules via the `bw_firewall_rules` filter. Always on.
2. **Known-good fast-path** — if the path is a real page (on the allowlist) or
   recently returned 200 → pass straight through, always.
3. **Pressure-mode load-shedding (the engine)** — when the site is under
   abnormal load, any path that's not real and not recently-200 → instant
   `410` (+ `Retry-After`, header `X-BW-Firewall: shed`) before WP boots. When
   not under pressure, unknown paths pass through normally.

This covers all three real incident shapes with zero per-URL config:
one hammered dead URL, a 39k-distinct enumeration scan, and 19k distinct
real-looking-but-dead URLs. The unifying fact: **in all of them, the flooded
URLs are not real pages.** You can't enumerate the junk (it's infinite), but
you can enumerate what's real.

## Known boundary (read this)

This does **not** defend a flood aimed at a *real, expensive* page (e.g. live
search results). That case is rare for these sites and largely handled by
Flywheel page caching. Out of scope.

## The allowlist

`wp-content/mu-plugins/bw-request-firewall-data/allowlist.json` — published
posts/pages/CPTs, taxonomy term archives, the homepage/blog page, plus
standard prefixes (`/wp-admin/`, `/wp-json/`, `/wp-content/uploads/`, …) and
patterns (sitemaps, `robots.txt`, …). Rebuilt automatically on
`save_post`/`edited_term`/`delete_post`/permalink change (debounced ~90s) and a
daily cron backstop. Stored as **JSON read via `file_get_contents`** (not a PHP
`include`) so opcache can never serve a stale or deleted version — the
fail-open guarantee depends on seeing the file's true current state.

**FAIL-OPEN (non-negotiable):** if the allowlist is missing, empty, unreadable,
or older than 48h, pressure-mode passes everything through. A flood is
survivable; blocking the real site is not.

## Pressure detection

A shared counter store — **memcached** (present on Flywheel) → **APCu** →
**flat file with flock** — counts request outcomes. The mu file can't know the
response code before WordPress runs, so bw-dev's `shutdown` recorder records
the outcome (`is_404()` vs 200) into the store; the mu file reads the aggregate
pressure flag. No database (the DB is what's under stress).

- **Enter pressure** when 404s cross the threshold in a 10-min window (default
  300; normal baseline ~2,000/**day**, incidents ran thousands/**hour**), or a
  single hot path crosses the hot threshold (default 200).
- **Sustain:** the mu file re-extends the pressure flag on each shed so ongoing
  shedding keeps pressure alive (no flapping).
- **Exit automatically** when the flood stops — sheds stop, the flag expires,
  normal resumes. **Zero human action.**

## Safety / control

- **Emergency kill switch:** add `define( 'BW_FIREWALL_OFF', true );` to
  `wp-config.php`. The mu file returns immediately — no file deletion needed.
  (On hosts with `opcache.validate_timestamps=0`, reset opcache for it to take
  effect instantly; Flywheel reflects wp-config edits.)
- **Logged-in users always pass** (valid `wordpress_logged_in_*` cookie) — admin
  and editing are never affected by pressure mode.
- **Never fatal:** the mu file is trivial and defensive; a bad regex fails open
  (`@preg_match`), a store failure fails open (no pressure → pass).
- **Notification, not action:** optional email/webhook when pressure engages
  (one per episode) so you hear it from bw-dev, not the client. Protection is
  already live by the time you read it.

## Per-site rules

### From the admin (turnkey — no code)

**Settings → BW Dev → Request Firewall → Per-site rules.** One rule per line —
applied **always**, not only under load (Layer 1):

```
/kitchen-equipped => /            # 301 redirect a dead URL to the homepage
/old-page => /new-page/           # 301 redirect to another path
/old-page => https://x.com/y      # 301 redirect to an absolute URL
/spammy-probe-path                # bare path (no arrow) → 410 deny
# lines starting with # are comments
```

A relative redirect target resolves to this site via `home_url()`; an absolute
`https://` target is used as-is. Saved rules bake into the stub immediately. This
is the right home for a dead-but-valuable URL with lingering referral traffic —
use it instead of a hand-rolled per-site emergency-shield mu-plugin.

### From code (programmatic)

Add to a site mu-plugin / theme `functions.php`. The `bw_firewall_rules` filter
layers on top of the admin rules:

```php
// Static deny patterns (path-only PCRE). Each → 410.
add_filter( 'bw_firewall_patterns', function ( $p ) {
	$p['my_probe'] = '~/some-known-bad-path$~i';
	return $p;
} );

// Explicit deny / redirect rules.
add_filter( 'bw_firewall_rules', function ( $r ) {
	$r[] = array( 'match' => '~^/kitchen-equipped/?$~i', 'action' => 'redirect', 'target' => home_url( '/' ) );
	return $r;
} );

// Extra legitimate paths to always allow (normalized: lowercase, trailing slash).
add_filter( 'bw_firewall_allowlist', function ( $paths ) {
	$paths['/special-route/'] = 1;
	return $paths;
} );
```

Filter changes are baked into the regenerated mu file on the next admin load (or
run `wp bw-dev firewall deploy`).

## Staying in sync (deploy)

The stub is regenerated whenever its content would change. It self-heals across
all the ways a site gets updated:

- **Admin browsing** — `admin_init` checks hourly and regenerates on a mismatch.
- **Settings saved** — rules/thresholds bake into the stub immediately.
- **Plugin (re)activation** — forces a fresh deploy on the next request.
- **File push without the WP upgrader** (SFTP/rsync/`wp plugin update`) — a cheap
  per-request version check (front-end + admin) detects that the on-disk stub is
  an older `FIREWALL_VERSION` than the running code and regenerates immediately,
  so a stub can't silently keep running stale logic until someone opens wp-admin.

### WP-CLI (headless / fleet rollout)

```bash
wp bw-dev firewall status            # stub version, in-sync, backend, allowlist, pressure, rules
wp bw-dev firewall status --format=json
wp bw-dev firewall deploy            # force regenerate + verify (use after pushing files)
```

`deploy` is the clean way to roll to a fleet: push the plugin files, then run
`wp bw-dev firewall deploy` per site to regenerate and verify without a browser.
It refuses (with a warning) if the module is disabled.

## Reading the admin screen

**Settings → BW Dev → Request Firewall**: firewall-file status, allowlist size +
last-built time, the counter-store backend in use, whether pressure mode is
active right now, recent activations, a **Test a URL** box (paste a real URL to
confirm it passes, or a probe to confirm it's caught), and the load-shedding
thresholds + notification settings.

## Verify on a site (runbook)

Replace `SITE` with the site URL. Real-page URLs come from the site's sitemap.

```bash
# 1. Static deny (always 410, no load needed):
curl -si SITE/.env | head -1                       # 410
curl -si SITE/backup.sql.gz | head -1              # 410
curl -si SITE/vendor/x/eval-stdin.php | head -1    # 410

# 2. Real pages pass:
curl -si SITE/ | head -1                           # 200
curl -si SITE/<a-real-page>/ | head -1             # 200

# 3. Unknown junk WITHOUT load → normal 404 (passes through):
curl -si SITE/kitchen-equipped | head -1           # 404 (no X-BW-Firewall)

# 4. Under load (lower threshold to ~5 on the settings tab, then send ~6
#    distinct junk requests) the 6th+ flips to shed:
for i in $(seq 1 8); do curl -s -o /dev/null -w "%{http_code} " SITE/junk-$i-$RANDOM; done
# → 404 404 404 404 404 410 410 410   (auto-engages at the threshold)
curl -si SITE/anything-unknown | grep -i x-bw-firewall   # X-BW-Firewall: shed

# 5. Real page STILL passes under pressure:
curl -si SITE/<a-real-page>/ | head -1             # 200

# 6. Kill switch: add define('BW_FIREWALL_OFF', true) to wp-config.php →
curl -si SITE/.env | head -1                       # 404 (firewall off)

# Reset the threshold to 300 when done.
```

## Where this sits in the defense stack

- **Front line (not this):** Cloudflare WAF + cache + origin lock. Best because
  junk never reaches Flywheel — but it's per-zone config, and doesn't help when
  a scanner hits the origin IP directly.
- **Backstop (this):** runs inside WordPress, on every site, regardless of
  Cloudflare, and catches direct-to-origin hits.
- **No `.htaccess`/nginx access** on Flywheel — the mu-plugin is the earliest
  hook we control.