0073. API hosts are DNS-only in Cloudflare; rate limits key on Fly-Client-IP

Status: Accepted (2026-10-12); amended 2026-10-31 (decisions 4 and 5, issue #192)

Context

Each backend is served on two domains. Both hosts reach the same Fly app:
  • Bun: bun-api.kokweng.net and bun-api.rumah.homes, app gremlin-bun-api.
  • Go: go-api.kokweng.net and go-api.rumah.homes, app gremlin-go-api.
Fly issues their TLS certificates. DNS for these hosts is on Cloudflare, and the records were created proxied (orange cloud). The Bun backend runs two rate limiters on /api/auth/*:
  • App middleware (rateLimitMiddleware): keys on CLIENT_IP_HEADER = 'Fly-Client-IP'. Fly overwrites this header on every request, so the caller can’t forge it.
  • Better Auth’s built-in limiter: for example 3 sign-ins per 10 s. It keyed on the leftmost X-Forwarded-For, Better Auth’s default.
With the Cloudflare proxy on, both keys were wrong:
  • Fly-Client-IP was Cloudflare’s edge IP, so many users shared one bucket.
  • The leftmost X-Forwarded-For comes from the caller, so any client could get a fresh bucket on every request.

Decision

  1. API hosts are DNS-only (grey cloud). bun-api.* and go-api.* on both zones point straight at Fly. Cloudflare still fronts the Workers (landlord.*, tenant.*, the rumah.homes apex marketplace, panel.*).
  2. Both limiters key on Fly-Client-IP. Better Auth gets advanced.ipAddress.ipAddressHeaders = [CLIENT_IP_HEADER], the same header the app limiter already uses.
  3. Go keys on Fly-Client-IP too (added 2026-10-13). The Go backend sets Echo’s IPExtractor from CLIENT_IP_HEADER (Fly-Client-IP in backend/go/fly.toml), so every c.RealIP() (global and per-route rate limiters, audit IP, declaration IP) uses it. When the header is unset or malformed it uses the socket peer, never X-Forwarded-For / X-Real-IP. Before this, Echo’s default trusted those headers: on the live Go API, 120 rapid requests from one client got 25 × 429, while the same 120 with a forged X-Forwarded-For each got 0 × 429.
  4. Worker server routes forward the visitor’s IP behind a shared secret (added 2026-10-31). Worker fetches to Go or Bun send X-Gremlin-Client-IP (the visitor’s CF-Connecting-IP) and X-Gremlin-Proxy-Secret. Both backends use the forwarded IP only when CLIENT_IP_PROXY_SECRET is configured and the secret matches (constant-time compare) and the value parses as an IP; otherwise they fall back to CLIENT_IP_HEADER as before, without an error. The secret is a Fly secret on both apps and a Worker secret on the rental portal (super-app-rental, plus --env preview) and marketplace (gremlin-rental), never a fly.toml / wrangler.toml value. Workers always strip inbound X-Gremlin-* headers before setting their own.
    • Go: ClientIPExtractor(header, proxySecret) sets Echo’s IPExtractor, so every c.RealIP() gets the forwarded IP.
    • Bun: clientAddress resolves it for the app limiter. Better Auth can’t verify a secret, so the /api/auth/* route (and the synthetic get-session call in the auth middleware) clones the request with x-gremlin-auth-client-ip set to the resolved IP, overwriting any inbound copy, and ipAddressHeaders names only that header.
  5. Counts are shared through Redis when REDIS_URL works (added 2026-10-31). Go’s four IP limiters (global, listing check, interpret, error reports) use a Lua token bucket on rate_limit:ip:<name>:<ip> with Redis’s clock. Bun’s app limiter uses ratelimit:*, and Better Auth uses rateLimit.customStorage on ratelimit:better-auth:* (not secondaryStorage, which would move sessions out of Postgres). On a Redis error each falls back to per-process memory, so an outage loosens limits instead of failing requests. The partner key limiter stays fail-closed.

Considered options

  • Keep the proxy and key on CF-Connecting-IP. This keeps Cloudflare’s WAF and DDoS protection. But gremlin-*.fly.dev bypasses Cloudflare, so a caller could forge the header there. It would also need an origin check (Cloudflare IP allow-list or authenticated origin pulls), and changes to both limiters and the fly.toml design.
  • Leave as is. Some users share a bucket, and the sign-in limit can be bypassed.

Consequences

  • No Cloudflare WAF, DDoS protection or caching in front of the APIs. API responses aren’t cacheable, and Fly’s edge gives basic protection. If the APIs need a WAF later, revisit this ADR and take the CF-Connecting-IP option with an origin check.
  • Fly renews certificates without the _acme-challenge CNAME.
  • Server-to-server calls depend on the secret. Until CLIENT_IP_PROXY_SECRET is set on both Fly apps and both Workers, Worker requests still arrive with the Worker’s Fly-Client-IP, so users behind one Cloudflare egress share a bucket. A secret mismatch fails the same way, silently. Rotate it on all four at once.
  • Shared counts need Redis. All four backends use one Redis on the home NAS (ADR-0074). While it is unreachable, counts fall back to memory and the effective ceiling is (number of machines) times the configured figure. All Better Auth instances share one store, so one IP’s sign-in attempts count together across both domains.
  • Switching a record back to proxied silently breaks both limiters again. Check this ADR first.