0068. Production decision-model calls are replayed on a local machine
Status: Accepted (2026-09-26)Context
Rental features call TypeSafe Jev. Kev is an open-weight model with the same API (design:docs/superpowers/specs/2026-09-26-decision-model-providers-design.md).
To choose between them, we replay real production calls against Kev and compare.
Each typesafe_calls row stores the full input: listing addresses and descriptions,
chat text with up to 5 earlier messages, and duplicate-check property addresses (a
subject and up to 10 candidates from other landlords). ADR-0066 and
ADR-0067 gate the use of real data in calibration behind Gremlin’s confirmation.
Kev runs on a developer’s local GPU machine. The replay tool reads production with
the developer’s normal DB login.
Decision
Allowcmd/replay-typesafe to read typesafe_calls inputs and label columns from
production, and send them to a Kev instance on the developer’s own machine.
- Inputs are sent only to a model on the developer’s own machine. They never go to
another third party. The tool enforces this, with no override:
--base-urlmust be a loopback host (127.0.0.1,[::1],localhost).- The dialer checks the actual IP it connects to, so a DNS change cannot bypass the flag check.
- Redirects are refused and environment proxies are ignored.
- A GPU on another machine is reached through an SSH tunnel, so the URL stays loopback.
- The same rule applies to
cmd/eval-listing-check,cmd/eval-message-checkandcmd/eval-duplicate-checkwhen the source is not the built-in synthetic set (dbor any file) and a custom--base-urlis set. - Output files hold call ids, ref ids, probabilities and run settings only. No state text and no API key is written.
- The tool runs SELECTs only. It writes nothing to production.
- To hand-label a disagreement, the developer may read that one message by id with their existing production DB access. Hand labels go in a local file as ids and a label only. The same Gremlin confirmation gates this.
- Gremlin is the accountable owner of this trade-off, as in ADR-0064.
Consequences
- Personal data (addresses, chat text) passes through a developer laptop’s memory and local model. A lost or shared laptop exposes only ids and probabilities on disk, not the text.
- The developer’s DB login can write. Safety relies on the tool’s queries being SELECT-only. A read-only role was considered and declined for simplicity.
- Only the last 90 days of calls can be replayed (
typesafe_callsretention). - A Kev on another machine needs an SSH tunnel. That is a small setup cost, accepted so the loopback check has no exceptions.
- Hosting Kev for production traffic is a new third party. That needs its own ADR.