0073. API hosts are DNS-only in Cloudflare; rate limits key on Fly-Client-IP
Status: Accepted (2026-10-12); amended 2026-10-31 (decisions 4 and 5, issue #192)
Context
Each backend is served on two domains. Both hosts reach the same Fly app:- Bun:
bun-api.kokweng.netandbun-api.rumah.homes, appgremlin-bun-api. - Go:
go-api.kokweng.netandgo-api.rumah.homes, appgremlin-go-api.
/api/auth/*:
- App middleware (
rateLimitMiddleware): keys onCLIENT_IP_HEADER = 'Fly-Client-IP'. Fly overwrites this header on every request, so the caller can’t forge it. - Better Auth’s built-in limiter: for example 3 sign-ins per 10 s. It keyed on the
leftmost
X-Forwarded-For, Better Auth’s default.
Fly-Client-IPwas Cloudflare’s edge IP, so many users shared one bucket.- The leftmost
X-Forwarded-Forcomes from the caller, so any client could get a fresh bucket on every request.
Decision
- API hosts are DNS-only (grey cloud).
bun-api.*andgo-api.*on both zones point straight at Fly. Cloudflare still fronts the Workers (landlord.*,tenant.*, therumah.homesapex marketplace,panel.*). - Both limiters key on
Fly-Client-IP. Better Auth getsadvanced.ipAddress.ipAddressHeaders = [CLIENT_IP_HEADER], the same header the app limiter already uses. - Go keys on
Fly-Client-IPtoo (added 2026-10-13). The Go backend sets Echo’sIPExtractorfromCLIENT_IP_HEADER(Fly-Client-IPinbackend/go/fly.toml), so everyc.RealIP()(global and per-route rate limiters, audit IP, declaration IP) uses it. When the header is unset or malformed it uses the socket peer, neverX-Forwarded-For/X-Real-IP. Before this, Echo’s default trusted those headers: on the live Go API, 120 rapid requests from one client got 25 × 429, while the same 120 with a forgedX-Forwarded-Foreach got 0 × 429. - Worker server routes forward the visitor’s IP behind a shared secret (added
2026-10-31). Worker fetches to Go or Bun send
X-Gremlin-Client-IP(the visitor’sCF-Connecting-IP) andX-Gremlin-Proxy-Secret. Both backends use the forwarded IP only whenCLIENT_IP_PROXY_SECRETis configured and the secret matches (constant-time compare) and the value parses as an IP; otherwise they fall back toCLIENT_IP_HEADERas before, without an error. The secret is a Fly secret on both apps and a Worker secret on the rental portal (super-app-rental, plus--env preview) and marketplace (gremlin-rental), never afly.toml/wrangler.tomlvalue. Workers always strip inboundX-Gremlin-*headers before setting their own.- Go:
ClientIPExtractor(header, proxySecret)sets Echo’sIPExtractor, so everyc.RealIP()gets the forwarded IP. - Bun:
clientAddressresolves it for the app limiter. Better Auth can’t verify a secret, so the/api/auth/*route (and the synthetic get-session call in the auth middleware) clones the request withx-gremlin-auth-client-ipset to the resolved IP, overwriting any inbound copy, andipAddressHeadersnames only that header.
- Go:
- Counts are shared through Redis when
REDIS_URLworks (added 2026-10-31). Go’s four IP limiters (global, listing check, interpret, error reports) use a Lua token bucket onrate_limit:ip:<name>:<ip>with Redis’s clock. Bun’s app limiter usesratelimit:*, and Better Auth usesrateLimit.customStorageonratelimit:better-auth:*(notsecondaryStorage, which would move sessions out of Postgres). On a Redis error each falls back to per-process memory, so an outage loosens limits instead of failing requests. The partner key limiter stays fail-closed.
Considered options
- Keep the proxy and key on
CF-Connecting-IP. This keeps Cloudflare’s WAF and DDoS protection. Butgremlin-*.fly.devbypasses Cloudflare, so a caller could forge the header there. It would also need an origin check (Cloudflare IP allow-list or authenticated origin pulls), and changes to both limiters and thefly.tomldesign. - Leave as is. Some users share a bucket, and the sign-in limit can be bypassed.
Consequences
- No Cloudflare WAF, DDoS protection or caching in front of the APIs. API responses aren’t
cacheable, and Fly’s edge gives basic protection. If the APIs need a WAF later, revisit
this ADR and take the
CF-Connecting-IPoption with an origin check. - Fly renews certificates without the
_acme-challengeCNAME. - Server-to-server calls depend on the secret. Until
CLIENT_IP_PROXY_SECRETis set on both Fly apps and both Workers, Worker requests still arrive with the Worker’sFly-Client-IP, so users behind one Cloudflare egress share a bucket. A secret mismatch fails the same way, silently. Rotate it on all four at once. - Shared counts need Redis. All four backends use one Redis on the home NAS (ADR-0074). While it is unreachable, counts fall back to memory and the effective ceiling is (number of machines) times the configured figure. All Better Auth instances share one store, so one IP’s sign-in attempts count together across both domains.
- Switching a record back to proxied silently breaks both limiters again. Check this ADR first.