ADR-0070: One shared uploads module for every app

Status

Accepted (2026-09-28)

Tags

uploads, storage, seaweedfs, s3, multipart, security, xss, rental, fieldforce, digital-worker, go, attachments

Decision

Every app uploads files through one Go module, internal/modules/uploads, mounted at /api/v1/uploads. Files are stored in SeaweedFS through the shared storage package.
  • Upload first, claim later. A file is stored as pending and gets an id. A feature (listing photo, proof, activity) claims the id server-side with UseCase.Claim, passing the owner it derived itself. Unclaimed uploads are swept after 24 hours.
  • The app decides the owner, never the caller. domain.Apps lists every app. Rental files belong to the user, because rental has no organization. Fieldforce, digital worker, and panel files belong to the org in the caller’s token, and are refused without one. The browser only names the app, so a landlord who is also an org admin cannot drop a rental file into the org.
  • Private by default. Files go to STORAGE_PRIVATE_BUCKET and are read only through GET /api/v1/uploads/:id/content by their owner. Only images may be public, and only in apps that allow it (PublicImages). Private uploads stay disabled (503) until the bucket is proven to refuse anonymous reads, using the canary probe now shared as storage.VerifyPrefixIsPrivate.
  • The bytes decide the type. domain.Sniff reads the first 512 bytes. The client’s Content-Type is ignored. HTML, SVG, XML, JavaScript, and executables are always refused, because they can run script if served back. The download sends nosniff, a sandboxing CSP, and attachment for anything that is not image, video, or audio.
  • Limits only tighten. The server sets 25MB per request (UPLOADS_MAX_SIZE) and 5GB for chunked uploads (UPLOADS_MAX_CHUNKED_SIZE). A caller may pass a smaller max_size and a narrower accept, never wider ones.
  • Large files use S3 multipart through Go. The client opens a session, sends 8MB chunks (PUT /:id/chunks/:n), and completes. Chunk 1 must come first: it is sniffed before the storage upload is even created. Go records each part only after storage confirms it (upload_parts: number, size, ETag), and completes from that record. A client resumes by asking which chunks arrived. We never ask storage: SeaweedFS ListParts was seen returning no parts for uploads that had them, and a chunk cut off by a pause can leave a short part.
  • Object keys are uploads/{app}/{owner_type}-{owner_id}/{id}{ext}, so apps and owners never share a prefix and a user id can never collide with an org id.

Context

Three modules already stored files, each its own way. Rental photos and proofs have dedicated endpoints. Fieldforce has a shared-looking attachments table with its own cleanup job. The new FileUpload component in packages/ui needs one endpoint that every app can call. Reusing attachments was considered and rejected for now:
  • org_id is NOT NULL, so rental (no organization) cannot write to it.
  • file_type allows only photo and signature.
  • It stores a public file_url; private files have none.
  • It has no columns for filename, size, content type, visibility, or chunk progress.
  • Its cleanup lives in the fieldforce module and only knows the public bucket. It would delete a private file’s row and leave the object behind forever, and it stops running when fieldforce is turned off.
Cleanup itself is kept: files that were uploaded but never used should not stay on the server. The uploads module runs its own sweep that knows both buckets and aborts unfinished multipart uploads.

Considered options

  • Extend attachments in place. One table sooner, but it changes fieldforce behaviour while fieldforce still trusts the browser’s Content-Type (#153). Done later instead: fieldforce moves onto uploads and attachments is dropped, which also fixes #153.
  • Upload endpoints per feature (as rental does). Rejected: every feature re-implements sniffing, limits, chunking, and cleanup, and each copy can get the security wrong.
  • Presigned URLs straight to SeaweedFS. Saves Go bandwidth, but the storage endpoint is not meant to be browser-facing, the type check would happen after the bytes land, and private bucket policy would have to be exposed. Revisit if Go bandwidth becomes the bottleneck.
  • tus protocol. Resumable by design, but adds a dependency and a second storage layout for what S3 multipart already gives us.

Consequences

  • New tables uploads and upload_parts (migrator CreateUploadsTables), module flag MODULE_UPLOADS (on for the full, smb, and rental profiles), and a 10-minute sweep.
  • Adding an app is one line in domain.Apps. Adding a file type is one entry in filetype.go.
  • Feature modules must claim uploads they keep; anything unclaimed disappears after a day.
  • A private file can only be read by its owner through this module. Features that share files with other people (a tenant viewing a landlord’s agreement) keep streaming through their own endpoints after checking their own access rules.
  • Browsers reach the endpoint through each app’s server proxy, so the same-origin session works and no CORS setup is needed.