​
ADR-0030: Digital Worker extraction watermark — advance-on-success, with divergent outage recovery

​
Status

Accepted (pending Phase 1 implementation)

​
Tags

digital-worker, dedup, watermark, privacy, cost, idempotency, chat

​
Decision

Proactive task-extraction deduplicates via a per-channel watermark (last_extracted_message_ts on dw_channel_bindings):
  • Local triage may read a context window of older messages, but only messages after the watermark can become drafts.
  • The watermark advances only after a successful, persisted extraction pass.
This produces two deliberately divergent outage behaviors:
CauseOrg intentWatermarkOn recovery
Caps/credits exhausted (402), BYOK out-of-credits (502), provider error, local-triage downOrg wants the worker running, just can’t extract right nowStays putBackfills the gap
Feature flag turned off”Stop observing us”Jumps past the off-windowNo backfill
Recovery backfill is bounded by per-org retention (messages aging past the window are dropped + logged) and processed in batches to avoid a cost spike at top-up.

​
Why

The (provider, external_message_id) idempotency key stops duplicate event processing, but not duplicate drafts: triage runs over rolling windows, so one commitment sits in several windows and would be re-drafted each time. The watermark separates “messages I may read for context” from “messages I’m allowed to turn into work,” killing overlapping-window duplicates at the source. The subtle, must-record part is the divergence between two kinds of outage. Both leave the worker temporarily not-extracting, but they mean opposite things:
  • Cap/credit exhaustion carries no intent to stop observing — the org wants the worker running and simply couldn’t pay for the LLM call. Silently dropping commitments made during the outage would be a billing hiccup eating real work, so the watermark stays put and the backlog is recovered on top-up.
  • Flag-off carries an explicit intent to stop observing — backfilling the off-window on re-enable would violate that intent and the privacy boundary, so the watermark jumps past the gap.
A future implementer could easily treat both outages identically and break one of the two behaviors. The distinguishing principle is did the org express intent to stop observing? Rejected alternatives:
  • Semantic dedup against existing open/promoted drafts. Needs embedding/similarity matching; MVP complexity not justified — the human approver dismisses the rare re-discussion duplicate. Rejected for MVP.
  • Always backfill on feature-flag re-enable. Violates the org’s “stop observing” intent and the privacy boundary. Rejected.
  • No backfill on cap/credit exhaustion. Loses real commitments over a billing blip. Rejected.

​
Consequences

  • Extraction is at-least-once: a failed pass re-tries its window, so a rare duplicate draft can appear, which the approver dismisses.
  • A long outage exceeding per-org retention permanently drops the oldest un-extracted messages (logged skipped) — bounded backlog by design.
  • Recovery after a long outage is a potential cost spike, mitigated by batching and the local-triage pre-filter (ADR-0028).

​
Rules for agents

  • Advance the watermark ONLY after a successful, persisted extraction pass.
  • Any “temporarily couldn’t extract” cause MUST leave the watermark put for retry; ONLY an explicit feature-flag-off advances it past the gap.
  • Recovery backfill MUST be bounded by per-org retention and processed in batches.