0068. Production decision-model calls are replayed on a local machine

Status: Accepted (2026-09-26)

Context

Rental features call TypeSafe Jev. Kev is an open-weight model with the same API (design: docs/superpowers/specs/2026-09-26-decision-model-providers-design.md). To choose between them, we replay real production calls against Kev and compare. Each typesafe_calls row stores the full input: listing addresses and descriptions, chat text with up to 5 earlier messages, and duplicate-check property addresses (a subject and up to 10 candidates from other landlords). ADR-0066 and ADR-0067 gate the use of real data in calibration behind Gremlin’s confirmation. Kev runs on a developer’s local GPU machine. The replay tool reads production with the developer’s normal DB login.

Decision

Allow cmd/replay-typesafe to read typesafe_calls inputs and label columns from production, and send them to a Kev instance on the developer’s own machine.
  • Inputs are sent only to a model on the developer’s own machine. They never go to another third party. The tool enforces this, with no override:
    • --base-url must be a loopback host (127.0.0.1, [::1], localhost).
    • The dialer checks the actual IP it connects to, so a DNS change cannot bypass the flag check.
    • Redirects are refused and environment proxies are ignored.
    • A GPU on another machine is reached through an SSH tunnel, so the URL stays loopback.
  • The same rule applies to cmd/eval-listing-check, cmd/eval-message-check and cmd/eval-duplicate-check when the source is not the built-in synthetic set (db or any file) and a custom --base-url is set.
  • Output files hold call ids, ref ids, probabilities and run settings only. No state text and no API key is written.
  • The tool runs SELECTs only. It writes nothing to production.
  • To hand-label a disagreement, the developer may read that one message by id with their existing production DB access. Hand labels go in a local file as ids and a label only. The same Gremlin confirmation gates this.
  • Gremlin is the accountable owner of this trade-off, as in ADR-0064.
Running the replay on real data is gated on the same Gremlin confirmation as ADR-0066 and ADR-0067 calibration.

Consequences

  • Personal data (addresses, chat text) passes through a developer laptop’s memory and local model. A lost or shared laptop exposes only ids and probabilities on disk, not the text.
  • The developer’s DB login can write. Safety relies on the tool’s queries being SELECT-only. A read-only role was considered and declined for simplicity.
  • Only the last 90 days of calls can be replayed (typesafe_calls retention).
  • A Kev on another machine needs an SSH tunnel. That is a small setup cost, accepted so the loopback check has no exceptions.
  • Hosting Kev for production traffic is a new third party. That needs its own ADR.