0074. One Redis on the home NAS for Fly and K8s, reached from Fly over WireGuard
Status: Accepted (2026-10-31)Context
Bun and Go use Redis for rate limits (ADR-0073), idempotency keys, the partner API-key limiter (rate_limit:apiKey:*), and app-JWT revocation markers (authInvalidBefore:*,
written by Bun, read by Go). Revocation only works when the Bun that writes a marker and
the Go that reads it share one Redis.
Before this decision:
- Fly Go and both K8s backends used a free-tier Upstash database. In October 2026 it hit its 500,000-request limit, most likely from K8s Bun’s health probes going through the rate limiter (an estimate, not measured). After that, every command failed: K8s Bun crash-looped, and Go’s v1 API ran degraded (partner key routes answered 503).
- Fly Bun had no Redis, so its limits and revocation markers stayed per machine.
- K8s moved to a Redis container on the home NAS (
192.168.0.142). Fly still could not reach it, because the NAS is only on the home LAN.
sin rather than the home Redis.
Decision
- One Redis for all four backends:
redis:7.4-alpineon the UGREEN NAS, run with--requirepass, AOF (--appendonly yes) and--maxmemory-policy noeviction(Go’sValidateForRateLimiterrequires the last two). Key prefixes don’t overlap: Bun usesratelimit:*,idempotency:*,authInvalidBefore:*; Go usesrate_limit:*. - K8s reaches it on the LAN at
192.168.0.142:6380. UGOS runs its own Redis on127.0.0.1:6379, so Docker can’t publish 6379 on the NAS. The URL is in each overlay’s sealed secret (REDIS_URL). - Fly reaches it over Fly’s private network. The NAS is a Fly WireGuard peer,
gremlin-nas-redis(orgpersonal, gatewaysin, addressfdaa:3a:8068:a7b:93d7:7da2:2336:f302). Alinuxserver/wireguardcontainer dials out to Fly, and the Redis container shares its network (network_mode: service:wireguard), so Fly connects to[fdaa:3a:8068:a7b:93d7:7da2:2336:f302]:6379. The LAN port 6380 is published on the WireGuard container. Peer DNS names (*._peer.internal) don’t resolve from the machines, so the URL uses the IPv6 literal. Fly’sREDIS_URLis a Fly secret ongremlin-go-apiandgremlin-bun-api.
Considered options
- Paid Upstash (pay-as-you-go) or
fly redis createnearsin. Managed, close to the machines, and not tied to the home network, but it costs money, and K8s would cross the internet to reach it. - Port-forward the NAS Redis on the home router. Exposes Redis to the internet behind a password sent in plain text (no TLS), needs DDNS for the residential IP, and most home routers can’t limit the port to Fly’s egress IPs.
- Keep Fly on Upstash and K8s on the NAS. A ban made on one side wouldn’t revoke tokens on the other, and Upstash’s quota would run out again.
Consequences
- Revocation markers, rate-limit counts and idempotency keys are now shared across Fly and K8s, and across machines.
- The home NAS and its internet link are a single point of failure for both
deployments. When Redis is unreachable:
- Bun’s app limiter, Better Auth’s limiter and Go’s IP limiters count in per-process memory.
- Revocation checks fail open.
- Go’s partner key limiter fails closed (503
rate_limit_unavailable). - All of these recover on the next request once Redis is back; no restart is needed. go-redis redials on demand, and Bun reconnects at most every 5 s. Go’s startup log “v1 API is degraded” only reports the state at boot; nothing keeps it.
- Each Fly command costs about 13 ms (measured from
sin). Bun’s app limiter makes about 3 commands per request; Go’s limiters make one, within a 150 ms timeout. - Every machine in the Fly org can reach this Redis over 6PN, so the Redis password is the
only control. The WireGuard private key lives only on the NAS (
wg0.confin the Redis project’s WireGuard config folder). If it’s lost, remove the peer (fly wireguard remove personal gremlin-nas-redis), create a new one, and update both Fly secrets with the new address. - Changing the Redis password means resealing
REDIS_URLin both K8s overlays and resetting it on both Fly apps. - Revisit this decision before any app takes real traffic (AGENTS.md §6). A managed Redis
near
sinremoves the home dependency.