# Legacy migration — schema + transforms + rehearsal loop

**Status:** shipped (M2) · **Founding spec:** `/srv/apps/work/docs/rebuild/`
02-domain-model.md (as amended), 05-migration-plan.md step 2,
06-test-plan.md §10, 09-build-plan.md M2.

## What this is

The R1 schema (alembic `0003`) and the idempotent legacy→R1 transforms
that rebuild `public.*` from the landed `legacy.*` snapshot — plus the
new-side diff-harness emitter and the full nightly rehearsal loop
(snapshot → scrub → legacy → transform → extract both → compare →
status line). The founding spec is authoritative; this doc records
what shipped and where it lives.

## Schema (alembic 0003, models in `app/models/` — one file per aggregate)

parties, tenants, engagements, compensation_terms (`kind` =
cost|billout worker-terms model), rate_overrides (own table, judgment
#25), projects (uniqueness `(tenant_id, client_party_id, lower(name))`,
judgment #33), import_batches, time_entries (raw_* immutable import
payload, `source_timezone`, per-side `cost_currency`/`billout_currency`
stamps, `invoice_line_id` lock), invoices + invoice_lines (R1 keeps
today's invoice shape + from/to party links, judgment #24), cc_*
(near-verbatim; `(batch_id, line_index)` now unique), comments
(tenant_id + nullable anchors), audit_log (unified audit/undo — full
before-images for bulk ops; writers land at M5), migration_runs (the
transform run-report table). **No sets/tasks tables** (judgment #32).
Also completes `users.person_party_id` (FK → parties + unique;
deferred from M1, ADR #002 #5).

Money = `Numeric(12,2)` + `CHAR(3)` currency stamped as a pair (CHECK
enforced); `source_rate`/`source_amount` keep 4dp (raw import
provenance — rebuild delta #2 decided). Entry times stay timezone-naive
(07 #17). Legacy `_usd` column names die: `manual_total`, `income`,
`amount`, `cost_amount`/`billout_amount` + currency columns.

## Transforms (`app/services/migrate/`, CLI `scripts/migrate_legacy.py`)

Ordering per 05 step 2 (parties → tenant → users; direct_client edges;
worker engagements + terms incl. Gary cost 0/billout 25; projects +
rate_overrides; import batches; time entries; invoices + time-kind
lines; CC; comments). Key rules, all tested:

- **Idempotent**: every run truncates the target tables (users is
  column-scoped-upserted — login-provisioned rows and preferences
  survive; parties is DELETEd because users references it) and rebuilds
  from `legacy.*` in ONE transaction. Failure = rollback; the report
  row is written outside the transaction either way.
- **Person parties from team_members ONLY**, deduped by lower(email);
  unmatched source_user_emails create nothing (rows stay Blocked on
  their raw_* identity).
- **Operator/client name collisions merge** into one party ("one party
  wearing two hats"); same-named clients under different operators stay
  separate.
- **Stamps copy VERBATIM** (preflight fails if any legacy stamp exceeds
  2dp); all-CAD currency stamping (07 #21) with per-currency row counts
  in every report.
- **Straggler rule (judgment #22)**: transferred-but-not-invoiced count
  is computed and REPORTED (expected 0); a nonzero count attaches those
  rows to the EXISTING per-org "Legacy transfers" invoice
  (create-if-absent with a louder warning) — never an abort.
- **Reconciliation**: every legacy table row gets a disposition; every
  legacy COLUMN is enumerated in
  `app/services/migrate/dispositions.py` and verified against the
  landed schema at runtime — "not mentioned anywhere" fails the run.
  Unexplained deltas = nonzero exit + rollback.
- **Invoice from/to seeding** follows the pinned algorithm (from =
  tenant party always; to = non-tenant operator scope, else client
  scope, else NULL for owner review — chips beat entry-derived scope).

## Harness (new side + rehearsal)

- `harness/common.py` — shared canonical-JSON builders (both extractors
  use them; serialization cannot drift).
- `harness/extract_new.py` — the new-side emitter: project_id groups
  for assigned rows, byte-identical `raw_*` triple keys for unassigned,
  invoices via the invoice_lines lock chain, operators/clients from
  direct_client engagements, one team-member rates row per subcontract
  engagement.
- Nightly (03:30) now runs the FULL rehearsal; the same pipeline fires
  at boot when the sidecar lacks a `legacy` schema (or
  `RUN_SNAPSHOT_ON_START=1`). Status lines append to
  `harness/out/nightly-status.jsonl`.
- Host-side rehearsals: `scripts/sync_legacy_host.py` mirrors the live
  tables via psycopg COPY (host pg_dump is v16 and can't dump the PG17
  source; auth.users is scrubbed by construction — only
  id/email/created_at are ever selected).

## First real-data result (2026-07-07)

4,593/4,593 time entries accounted; every legacy table delta 0; second
run byte-identical (ex-meta). Harness GREEN across all history after
pinning ONE real judgment-#22 straggler that the run surfaced: entry
`adeb5a3f-5b7d-4807-8a7a-493ac7b42036` (2026-01-28, 502s, CAD 3.49,
Villa del Mar hosting support — transferred 2026-05-21, on no invoice),
attached to "Legacy transfers" invoice
`b6dda19a-…e846f5eb`, allowlisted row-for-row per 05 step 0.
**Owner review:** rian should confirm that entry belongs on Legacy
transfers (or invoice it properly in the old app — the next rehearsal
then reports zero and the allowlist entries go stale/removed).

## Changelog

- 2026-07-07 (0.3.0): M2 shipped — R1 schema, transforms 2.1–2.9,
  new-side emitter + common builders, snapshot-on-boot, nightly full
  rehearsal, host-side legacy sync, 43 new transform/schema/harness-parity tests
  (and the 4 existing harness tests folded into the pytest gate).