0074. One Redis on the home NAS for Fly and K8s, reached from Fly over WireGuard

Status: Accepted (2026-10-31)

Context

Bun and Go use Redis for rate limits (ADR-0073), idempotency keys, the partner API-key limiter (rate_limit:apiKey:*), and app-JWT revocation markers (authInvalidBefore:*, written by Bun, read by Go). Revocation only works when the Bun that writes a marker and the Go that reads it share one Redis. Before this decision:
  • Fly Go and both K8s backends used a free-tier Upstash database. In October 2026 it hit its 500,000-request limit, most likely from K8s Bun’s health probes going through the rate limiter (an estimate, not measured). After that, every command failed: K8s Bun crash-looped, and Go’s v1 API ran degraded (partner key routes answered 503).
  • Fly Bun had no Redis, so its limits and revocation markers stayed per machine.
  • K8s moved to a Redis container on the home NAS (192.168.0.142). Fly still could not reach it, because the NAS is only on the home LAN.
Issue #192 also needed counts shared across Fly machines, and recommended a paid Redis near sin rather than the home Redis.

Decision

  1. One Redis for all four backends: redis:7.4-alpine on the UGREEN NAS, run with --requirepass, AOF (--appendonly yes) and --maxmemory-policy noeviction (Go’s ValidateForRateLimiter requires the last two). Key prefixes don’t overlap: Bun uses ratelimit:*, idempotency:*, authInvalidBefore:*; Go uses rate_limit:*.
  2. K8s reaches it on the LAN at 192.168.0.142:6380. UGOS runs its own Redis on 127.0.0.1:6379, so Docker can’t publish 6379 on the NAS. The URL is in each overlay’s sealed secret (REDIS_URL).
  3. Fly reaches it over Fly’s private network. The NAS is a Fly WireGuard peer, gremlin-nas-redis (org personal, gateway sin, address fdaa:3a:8068:a7b:93d7:7da2:2336:f302). A linuxserver/wireguard container dials out to Fly, and the Redis container shares its network (network_mode: service:wireguard), so Fly connects to [fdaa:3a:8068:a7b:93d7:7da2:2336:f302]:6379. The LAN port 6380 is published on the WireGuard container. Peer DNS names (*._peer.internal) don’t resolve from the machines, so the URL uses the IPv6 literal. Fly’s REDIS_URL is a Fly secret on gremlin-go-api and gremlin-bun-api.

Considered options

  • Paid Upstash (pay-as-you-go) or fly redis create near sin. Managed, close to the machines, and not tied to the home network, but it costs money, and K8s would cross the internet to reach it.
  • Port-forward the NAS Redis on the home router. Exposes Redis to the internet behind a password sent in plain text (no TLS), needs DDNS for the residential IP, and most home routers can’t limit the port to Fly’s egress IPs.
  • Keep Fly on Upstash and K8s on the NAS. A ban made on one side wouldn’t revoke tokens on the other, and Upstash’s quota would run out again.

Consequences

  • Revocation markers, rate-limit counts and idempotency keys are now shared across Fly and K8s, and across machines.
  • The home NAS and its internet link are a single point of failure for both deployments. When Redis is unreachable:
    • Bun’s app limiter, Better Auth’s limiter and Go’s IP limiters count in per-process memory.
    • Revocation checks fail open.
    • Go’s partner key limiter fails closed (503 rate_limit_unavailable).
    • All of these recover on the next request once Redis is back; no restart is needed. go-redis redials on demand, and Bun reconnects at most every 5 s. Go’s startup log “v1 API is degraded” only reports the state at boot; nothing keeps it.
  • Each Fly command costs about 13 ms (measured from sin). Bun’s app limiter makes about 3 commands per request; Go’s limiters make one, within a 150 ms timeout.
  • Every machine in the Fly org can reach this Redis over 6PN, so the Redis password is the only control. The WireGuard private key lives only on the NAS (wg0.conf in the Redis project’s WireGuard config folder). If it’s lost, remove the peer (fly wireguard remove personal gremlin-nas-redis), create a new one, and update both Fly secrets with the new address.
  • Changing the Redis password means resealing REDIS_URL in both K8s overlays and resetting it on both Fly apps.
  • Revisit this decision before any app takes real traffic (AGENTS.md §6). A managed Redis near sin removes the home dependency.