Don't Block the UI: Async Email Unlocks with pg-boss

Ayush Mania

The Symptom

Saral lets recruiters bulk-add candidates to an email sequence. Each candidate's email is unlocked at the moment they are added - a separate API call to a vendor, three to eight seconds per profile, plus a credit charge. The first version called the vendor synchronously inside the request handler. Adding fifteen candidates blocked the request for two minutes. The browser sat on a spinner. If the user navigated away or the tab refreshed, half the unlocks completed and half did not, with no clear way to recover.

Spotting the Bug

This is the kind of bug that looks small in the planning doc and large in production. The shape of the failure - half-completed work with no recovery path - is the giveaway.

The Move to a Queue

The fix is straightforward in shape. Enqueue one job per candidate, return to the user immediately, let workers do the unlocks in the background. We already had pg-boss running for sequence email sends, so the queue was free.

Ts

// POST /sequences/:id/candidates  body: { candidateIds: [...] }

for (const candidateId of candidateIds) {
  if (await queueHasActiveJob(sequenceId, candidateId)) continue; // idempotent
  await enqueue("sequence.unlock", { sequenceId, candidateId });
}

return reply.code(202).send({ queued: candidateIds.length });

The endpoint returns in tens of milliseconds. The client polls a status endpoint every three seconds and updates the UI as jobs flip from queued → running → succeeded | failed_*. Three things make this work in practice that are not obvious from the shape.

Idempotency at Enqueue, Not Just Send

If a user clicks "Add 50 candidates" twice in quick succession, you do not want fifty extra jobs and fifty extra charges. The deduplication has to happen at enqueue time:

If a candidate already has an active (queued or running) job for this sequence, reuse it instead of creating a new one.

The check is one query against a partial index on (sequence_id, candidate_id) filtered to active states. It is cheap enough to do unconditionally on the enqueue path, and it makes the API safe to retry. A second layer covers the case where two requests pass the check simultaneously and both insert: a unique constraint on the same tuple, partial-indexed to active states, plus ON CONFLICT DO NOTHING. The user gets one job per candidate per sequence, however hard they click.

Per-Candidate Send Gates

A naive sequence kicks off email sends as soon as candidates are added. With async unlocks, that breaks: half the candidates do not have an email yet when the first send fires. The clean fix is to gate each send on its own candidate's unlock state, not on the batch.

Unlock state at send time	Action
Active job (queued or running)	Reschedule send +5 min
Succeeded	Send
failed_no_email	Skip; surface in UI for review
failed_transient	Reschedule send +5 min, capped retries

Per-candidate gating means one stuck unlock does not block 49 other sends. This sounds obvious in retrospect; the trap is to write the gate against a batch-level "all unlocks complete" boolean, which inverts the relationship between throughput and tail latency.

Concurrency That Backs Off Gracefully

The vendor's API rate-limits, and 429s come back with a Retry-After header. The pg-boss worker pool's teamSize is dynamic.

Start at 3.
On 429, drop to 1.
After a successful run, ramp back up.

Single retry on transient errors with a thirty-second delay. No retry on 4xx that are not 429 - a malformed request will be malformed forever. The combination of low base concurrency, dynamic backoff, and short-circuit retries handles the vendor's flakiness without saturating it. We do not have a token-bucket implementation; we have one number that goes up and down based on the most recent response code, and it is enough.

Credit Semantics Survive the Move

The unlock action costs credits. In the synchronous version the deduction was inline with the API call. In the async version it has to behave the same.

Pre-flight check at enqueue: do we have the credits to attempt this batch?
Deduction inside the worker, atomic with the unlock result.
Refund on failed_no_email or failed_transient after retry budget is exhausted.

The ledger is append-only, so a refund is a positive entry tagged rolled_back, not an edit of the original. Balance is SUM(amount) over the ledger; nothing about credits depends on a "current balance" field staying consistent.

Where to Over-Engineer

Charging users in the background, behind an API call you do not control, is the place where a rounding-error-shaped bug becomes a refund email. Spend the time here.

The Status Endpoint Is the Contract

The frontend polls one endpoint to render progress.

Ts

// GET /sequences/:id/unlocks
// → [ { candidateId, status, attempts, lastError? }, ... ]

status is one of queued | running | succeeded | failed_no_email | failed_transient | cancelled. The frontend renders a badge per candidate and surfaces a retry button on failed_transient. There is no stream, no websocket. Three-second polling is fine for batch sizes up to a few hundred and saves a separate transport.

What Was Tempting and Wrong

Two things felt right early and were not.

Reusing the existing send queue for unlocks

Surfacing per-job progress as a websocket stream

The Shape That Generalizes

The architecture is small enough to summarize.

Move the slow external call into a queue.
Make enqueue idempotent against the natural key.
Gate downstream actions per-item, not per-batch.
Adjust concurrency on the response code, not the clock.
Treat the credit ledger as append-only and refund forward.
Poll for status; do not stream.

Everything past that is the specific shape of your vendor's failure modes. The backbone holds whether you are unlocking emails, generating exports, or doing any other "do N slow things, then tell the user" workflow. The version that ships is rarely the version you would draw on a whiteboard. The version that survives a real workload is almost always shorter.

Share this post: