# dailysplice — v8 substrate feedback > Running log of friction hit while building **inside** the v8 conventions. > **This is feedback about the substrate, not about this project** — it gets > compiled into `/srv/.logs/planning/server-evolution-2026.md` or the ops queue > (`/srv/.logs/ideas.md`) once the build settles. > > Scope discipline: only entries where the *server framework, standards library, > or gateway* caused the friction. Bugs in this project's own code belong in > `handoff.md`, not here. > > Each entry: what happened, what it cost, and a candidate fix. > (Pattern proven 2026-07-30; it is what turns one project's > pain into a standards fix for every later project. Delete this file if the > project never hits substrate friction.) --- ## 2026-08-04 — `srv-gw deploy --build` is broken fleet-wide by Docker Compose v2.40 > **RESOLVED same day** — rian fixed the gateway in a parallel session; verified > here with a real `srv-gw deploy --project dailysplice --build`. > > **The candidate fix below was wrong, and the real resolution is more > interesting.** The flag was **removed, not repositioned**: the gateway runs > compose under `subprocess.run(capture_output=True)`, so stdout is not a TTY and > BuildKit already emits full plain output — `--progress=plain` was a **no-op in > this context** (measured: identical line counts with and without). The > collapsing it was meant to defeat only happens in an interactive terminal. The > useful half of that change — error tail plus the full log at > `/.logs/deploy-failed.log` — was never the problem and still works. > Full writeup, including the two process lessons, is the 2026-08-04 entry in > `/srv/.logs/ideas.md`. > > Kept because the *cause* is durable: the argv was assembled by list index > (`cmd.insert(4, ...)`), which hid the top-level-vs-subcommand ordering > constraint, and nothing exercised the deploy path it changed. **What happened.** Every `srv-gw deploy --project --build` fails immediately with `unknown flag: --progress`. The gateway builds the command as ``` docker compose -f --env-file up -d --remove-orphans --build --progress=plain ``` but in Compose **v2.40.3** (installed: Docker 29.0.2) `--progress` is a **top-level** flag, not an `up` flag. `docker compose up --help` no longer lists it; `docker compose --help` does. The flag was added to the gateway on 2026-07-30 for a good reason — BuildKit's default renderer collapses step output, which is a hard stop for a developer who cannot run docker at all — so it should be repositioned, not removed. **What it cost.** The first deploy of this project, plus the diagnosis. Worse, it is not project-specific: **no custom app on this server can be deployed with `--build` right now**, and a developer hitting this has no fallback, because they cannot run `docker` themselves. This is the exact failure mode the `--progress` flag was added to prevent. **Candidate fix.** One line in `/srv/gateway/gateway.py` (~3650): move `--progress=plain` from the end of `cmd` to immediately after `compose`, i.e. build `["/usr/bin/docker", "compose", "--progress=plain", "-f", ...]`. Note the existing `cmd.insert(4, "--env-file")` / `insert(5, ...)` indices shift by one and must be updated in the same change. Worth a smoke test in the deploy path so a future Compose upgrade fails loudly in CI rather than silently in production. **Workaround used here.** Build directly as the owner (`docker compose --progress=plain ... build`), then `srv-gw deploy` without `--build` to recreate the container. The gateway must do the *recreate* step because it alone can read the 600 `srv-gateway`-owned `.bw-auth.env`. --- ## 2026-08-04 — `app-client-register --secret-out` cannot write over a staged file **What happened.** `coding.md` says to stage a secrets file with `REPLACE_WITH_*` placeholders at tight permissions. Compose also requires every `env_file` to exist before the container will start. So `.bw-auth.env` was staged as `660 rian:dailysplice-dev` before the first deploy — and then ``` srv-gw app-client-register --name dailysplice ... --secret-out /srv/apps/dailysplice/.bw-auth.env ``` failed with `[Errno 1] Operation not permitted: '.../.bw-auth.env'`. The gateway writes the file as `srv-gateway` and chmods it, which it cannot do to a file owned by someone else. **The dangerous part: it failed HALF-WAY.** The client was **already registered** (and the host **already auto-un-gated to PUBLIC**) before the file write was attempted. The error message says only "Operation not permitted", giving no hint that the registration itself succeeded — so the obvious next move, re-running the command, would collide with an existing client. Recovery was `rm .bw-auth.env` followed by `app-client-rotate-secret --client-id dailysplice --secret-out `, which mints a fresh secret and writes it correctly. **What it cost.** A failed registration, a confusing error, and a secret that was generated and then thrown away. **Candidate fixes.** (a) Have `--secret-out` write via a temp file + `os.replace` in the target directory, so it does not need to own the existing file; or unlink the target first. (b) Pre-flight the destination *before* creating the client, so the command fails with nothing done rather than half done. (c) Failing both, make the error message state that the client WAS created and name the rotate-secret recovery. (d) `CHECKLIST.md` should say explicitly: **do not pre-stage `.bw-auth.env`** — let the gateway create it — and note that compose therefore needs the register step before the first `up` (or `required: false` on the `env_file` entry). --- ## 2026-08-04 — the security audit's deploy-state drift check is off by the UTC offset **What happened.** Immediately after building AND deploying, `srv-gw security-audit` reported: > `architecture/deploy_state_app/dailysplice: source changed ~419 min AFTER the > running image was built -- built but NOT deployed.` It reported `coachpapa` with the **same ~419 minutes**. 419 min ≈ 6.98 h, which is exactly this host's `America/Vancouver` UTC offset (UTC−7 in August). Verified false: `docker image inspect` gives `2026-08-04T20:02:52-07:00`, the newest real source file is 24 seconds *later* and is a host-run script that is not in the image at all, and the running container demonstrably carries the newest code and the current `VERSION`. **What it cost.** Little here, because it was checked — but the check is **permanently false-positive for every project**, which is worse than not having it: an always-red drift check trains everyone to ignore the one warning that is supposed to catch genuinely stale deploys. **Candidate fix.** In the audit's deploy-state check, parse the image's `.Created` as an **aware** datetime and compare against `st_mtime` converted to the same zone (both are epoch-comparable — use `os.stat().st_mtime` vs `datetime.fromisoformat(created).timestamp()`, never a naive `datetime`). Add a regression case asserting a just-built image reports zero drift. While there, exclude non-image paths (`.venv/`, `node_modules/`, `.pytest_cache/`, and `scripts/` that only run on the host) from "newest source", or the check will keep firing on files that could not possibly be in the image.