Operate the agent mailbox

The durable agent mailbox (ADR 0074) is the server-side store agents use to hand each other messages, results and questions. It lives in the control plane, independent of any per-task sidecar, so a queued message survives a runner re-attach, a sidecar restart or a server redeploy and is delivered at the recipient’s next turn.

This page covers what an operator watches and tunes. The user-facing how-to is Use the agent mailbox.

The mailbox on /metrics

The mailbox series render on the same /metrics endpoint as the rest of the instance telemetry (see Observability: Prometheus, Grafana and the nightly load checks). Their labels are fixed:

• No series is labelled by mailbox key or workspace id — both are unbounded and would leak cardinality into Prometheus.
• The drop counter’s only label is reason (currently over_ceiling).
• The depth gauge’s only label is kind, which is derived from the address prefix and folded to other for anything unrecognised.

Metric	Type	Signal
supacloud_mailbox_queued{kind="task|run|session|team|other"}	gauge	Messages awaiting delivery, per address kind
supacloud_mailbox_sent_total	counter	Fresh rows persisted (an idempotent replay is not a second send)
supacloud_mailbox_delivered_total	counter	Rows a recipient channel accepted
supacloud_mailbox_handled_total	counter	Rows the recipient’s harness acknowledged
supacloud_mailbox_expired_total	counter	Queued rows discarded by the retention TTL
supacloud_mailbox_dropped_total{reason="over_ceiling"}	counter	Sends refused because the mailbox was full
supacloud_mailbox_released_total	counter	Rows released back to queued after a deferred delivery
supacloud_mailbox_drains_total	counter	Drain passes run
supacloud_mailbox_drain_duration_seconds	histogram	Drain-pass latency
supacloud_mailbox_fanout_size	histogram	Team fan-out width (active member channels reached)

No RLS, application-level tenancy

The mailbox tables carry workspace_id and rely on application-level filtering in the query layer (migration 167’s rule), not row-level security. The metrics are process-global counters; they are not per-workspace series.

Reading queue health

• A mailbox is waiting. supacloud_mailbox_queued stays above zero and supacloud_mailbox_drains_total keeps advancing while supacloud_mailbox_delivered_total does not. The recipient has no live channel; it is delivered when one attaches.
• Sends are being refused. supacloud_mailbox_dropped_total{reason="over_ceiling"} rises. The mailbox is at its per-mailbox queue ceiling and the send is refused with an explicit error — backpressure, not silent loss. Drain the queue or raise the ceiling (the agent_mailboxes.max_queued row, default 64).
• Deliveries are bouncing back. supacloud_mailbox_released_total rises without a matching supacloud_mailbox_delivered_total. A team with no active member or a rejecting channel released the row; it stays queued for a later attempt rather than being dropped.
• Delivered but never handled. supacloud_mailbox_delivered_total exceeds supacloud_mailbox_handled_total. The frame reached the channel but the runner never acknowledged it. Delivery is at-least-once: a lapsed lease re-sends a row the channel may already have seen.
• Retention is discarding work. supacloud_mailbox_expired_total rises. A mailbox has had undelivered rows longer than its workspace retention window.

Tune retention

Every message is stamped with an expires_at when it is sent, from the workspace’s retention window or the pinned default of 30 days. The window is a per-workspace setting, stored in workspace_settings.message_retention_days (migration 398):

• NULL (the default) means the pinned 30-day window.
• A set value is bounded 1..3650 days.

The window applies to new sends only

Retention is read at send time. Lowering the value does not re-date messages already in flight — an existing row keeps the expires_at it was stamped with. Plan a lower window around the messages you actually want to expire.

A background sweep runs once a minute. It expires queued rows past their TTL, purges terminal rows (and ended sessions) on the same window, and drains mailboxes with no live channel. Retention deletes only by age; a closed gate does not delete its answer early — there is one retention knob, not two.

Setting the window today

No console surface writes message_retention_days yet; set the column directly in workspace_settings (the CHECK enforces 1..3650, or leave it NULL for the default) and it takes effect for the next send.

Delivery mechanics worth knowing

• Drain cadence. Attach and send drain a mailbox immediately; a mailbox with no live channel is caught by the 60-second sweep. A message can therefore wait up to that cadence plus the recipient’s own attach before it is delivered.
• Lease. A drain claims rows under a 60-second lease with FOR UPDATE SKIP LOCKED, so concurrent replicas skip each other’s rows and a duplicate sweep never double-delivers. A lapsed lease lets a later pass re-claim the row.
• Head-of-line. A delivery failure releases the whole claimed tail back to queued, not just the failing row, so one bad channel cannot strand the rest in delivering.
• Team fan-out. A team: row is delivered to every member with a live channel all-or-nothing; if none is active the row is released, never half-delivered.

Troubleshooting

Symptom	Likely cause	What to check
supacloud_mailbox_queued climbing, no deliveries	Recipient has no live channel	Is the task/session running? The 60 s sweep will drain on attach
dropped_total{reason="over_ceiling"} climbing	Mailbox at its max_queued ceiling	Drain the mailbox or raise agent_mailboxes.max_queued
released_total climbing, delivered_total flat	Dormant team or rejecting channel	Check team members’ liveness and the runner’s channel health
Delivered ≠ handled	Runner never acked	Check the runner’s intervention handling; the lease will re-send
expired_total climbing	Messages outliving their retention window	Confirm the recipient is reachable; review message_retention_days