The code is five lines. The system behind it that actually works in production is not.
Every engineer has written the naive webhook implementation. You fire a POST request to some endpoint, check for a 200 response and call it done. Fifteen minutes of work, maybe less. You ship it, it passes staging and you forget it exists.
Then production hits. A third-party service goes down at 2am. Your client’s server is behind a cold spot in their deployment pipeline and returns a 502 for twelve minutes. Your webhook delivery loop silently drops forty events because nobody thought about what happens when the other side is unavailable. You find out three days later when someone asks why their order confirmation never arrived.
The naive implementation wasn’t wrong. It was incomplete. And the gap between “sending HTTP requests” and “reliably delivering events” is where production systems actually live.
This is a breakdown of that gap. Not a tutorial, not a “getting started” guide. If you’re building anything that other systems depend on for state changes, which covers payment events, user lifecycle hooks, inventory updates and basically everything else in a modern backend, this is the architecture you actually need.
The Illusion That Kills You
Every webhook tutorial covers exactly one scenario: the happy path. Your server sends the request. The receiver responds with 200. You move on. The tutorial ends.
Nobody writes about what happens when the receiver is mid-deploy. Nobody covers the case where the receiver’s database is locked and it returns a 500. Nobody mentions what happens when the receiver processes the event twice because you retried after a timeout that wasn’t actually a failure on their end.
The result is that most webhook implementations in production look exactly like the tutorial example: synchronous, stateless and optimistic. They assume the network is reliable. They assume the receiver is always up. They assume a non-200 response is a permanent failure.
All of those assumptions are wrong, regularly and in ways that matter.
Decouple Before You Do Anything Else
The first architectural decision is the most important one. Do not deliver webhooks synchronously inside the user request path.
If a user triggers an event and your code immediately fires a POST to a third-party endpoint, you’ve tied your response time to their uptime. Their service degrades, your API degrades. Their deployment goes sideways, your users wait. You’ve inherited a dependency you didn’t choose.
The right pattern is to decouple event creation from event delivery. When something happens in your system, write an event record to a queue or a database table. Return a response to your user. Let a separate worker process pick up that event and attempt delivery asynchronously.
This isn’t novel. It’s just not what the tutorials show, because tutorials optimize for simplicity over correctness.
The decoupled model gives you three things immediately: your API is no longer blocked on third-party latency, failed deliveries can be retried independently without involving your user-facing code and you have a persistent record of every event and its delivery state.
That last one matters more than people expect. When a client says an event never arrived, you need a source of truth. A queue with delivery logs is that source of truth. A fire-and-forget HTTP call is not.
The Only Retry Strategy That Works at Scale
Once you have events in a queue, you need to decide how to retry failures. This is where most implementations get it wrong in interesting ways.
The naive retry is immediate: fail once, try again immediately. This is almost always the wrong answer. If a server is down, hammering it with immediate retries doesn’t help it recover. It makes things worse. You’re adding load to a system that’s already struggling.
The correct pattern is exponential backoff with jitter. The backoff means each subsequent retry waits longer than the last, doubling the wait time on each attempt. The jitter adds randomness to that wait time to prevent a thundering herd.
The thundering herd problem is what happens without jitter. If you have a thousand clients all retrying at exactly the same intervals, after a receiver recovers they all hit it simultaneously. The synchronized flood can bring down a system that would otherwise recover cleanly. Jitter prevents this by spreading retries across a time window instead of clustering them at a point.
The full jitter formula the AWS architecture team documented looks like this:
wait_time = random_between(0, min(max_cap, base_delay * 2^attempt))Pseudocode only. Implement this with your language’s cryptographically random number generator, not Math.random().
In PHP, the concrete version uses random_int() (cryptographically secure, unlike rand()) and looks like this:
// Calculate the next retry delay using full jitter exponential backoff.
// random_int() is cryptographically secure and available since PHP 7.0.
function calculateBackoffSeconds(int $attempt, int $baseSeconds = 1, int $capSeconds = 1800): int
{
$ceiling = min($capSeconds, $baseSeconds * (2 ** $attempt));
return random_int(0, $ceiling);
}
// In your delivery worker, after a failed attempt:
function scheduleRetry(array $event, int $attempt, string $lastError): void
{
$nextAttempt = $attempt + 1;
if ($nextAttempt > 10) {
moveToDeadLetterQueue($event, $lastError, $attempt);
return;
}
$delaySeconds = calculateBackoffSeconds($nextAttempt);
// Push the event back into your queue with a delay.
// The exact API here depends on your queue driver.
YourQueue::pushWithDelay($event, $delaySeconds);
}In practice: base delay of one second, cap at thirty minutes, max of ten attempts. After ten failures, you stop retrying and move the event somewhere else. That somewhere else is a dead letter queue.
Dead Letter Queues Are Not a Last Resort
The term “dead letter queue” sounds like failure. Engineers treat it as an afterthought, something you add only after a production incident teaches you why you need it.
Think of it differently. A dead letter queue is an honest system. It says: this event could not be delivered, here is everything about it and it is not going to be silently discarded.
When an event exhausts its retries, it goes to the DLQ with full context attached: the original payload, the destination URL, the number of attempts, every error response received along the way. This is the debugging information you need when a client asks why their integration stopped working.
The DLQ also tells you something important about your system’s health. If events are flowing into it at a steady rate, something upstream is broken. If it’s empty, delivery is healthy. Alerting on DLQ depth is one of the most useful monitoring signals you can add to a webhook system and it costs almost nothing to set up.
DLQs also give you manual recovery options. Events in a DLQ can be re-queued for delivery once the downstream system is fixed. Without a DLQ, those events are gone. With one, they’re recoverable.
Here is the pattern for writing to and recovering from a DLQ:
// Store a failed event in the dead letter queue with enough context to debug it.
function moveToDeadLetterQueue(array $event, string $lastError, int $attemptCount): void
{
$record = [
'original_event_id' => $event['id'],
'destination_url' => $event['destination_url'],
'payload' => $event['payload'],
'attempt_count' => $attemptCount,
'last_error' => $lastError,
'failed_at' => date('c'), // ISO 8601 timestamp
];
// Persist to a durable store: a database table, AWS SQS DLQ,
// Redis list or whatever your stack uses.
// The critical property is that it survives process restarts.
YourDlqStore::insert($record);
}
// Requeue a DLQ event manually after the downstream issue is resolved.
function requeueFromDlq(int $dlqRecordId): void
{
$record = YourDlqStore::find($dlqRecordId);
// Reset attempt count. Treat this as a fresh delivery attempt.
YourQueue::push([
'id' => $record['original_event_id'],
'destination_url' => $record['destination_url'],
'payload' => $record['payload'],
'attempt' => 0,
]);
YourDlqStore::markAsRequeued($dlqRecordId);
}The YourDlqStore and YourQueue references are placeholders for whatever persistence layer your stack uses. The structure of the record is what matters: every field you might need during a 3am debugging session should be in there before the event ever reaches this function.
Signature Verification Is Not Optional
Here is a failure mode that is not about reliability. It is about security.
If a system receives a webhook and processes it without verifying where it came from, an attacker can send forged requests to that endpoint and trigger whatever logic the webhook is supposed to trigger. Fake order confirmations. Fabricated payment events. Inventory updates that never happened.
The standard mitigation is HMAC-SHA256 signature verification. The sender signs the payload with a shared secret before sending. The receiver verifies that signature before processing.
In PHP, generating the signature looks like this:
// Generate a HMAC-SHA256 signature over the payload using a shared secret
$signature = hash_hmac('sha256', $payload, $secret);And verifying on the receiver side:
// Always use hash_equals for constant-time comparison, not ===
$expected = hash_hmac('sha256', $payload, $secret);
$is_valid = hash_equals($expected, $received_signature);The reason you use hash_equals() instead of === is timing attacks. String comparison in most languages short-circuits on the first mismatch. An attacker can measure tiny differences in response time to infer how many characters of their forged signature are correct. hash_equals() always takes the same time regardless of where the strings differ.
There’s a second vulnerability here that most implementations miss: replay attacks. A valid signature proves the payload came from someone with the secret. It does not prove the payload was sent recently. An attacker who captures a signed webhook can replay it hours later and the signature will still pass verification.
The fix is to include a timestamp in the signed payload and reject payloads with timestamps older than a defined window. Stripe’s implementation uses a five-minute window and includes the timestamp directly in the signature computation. Concretely, they sign timestamp.payload_body as a single string, which means you can't change the timestamp without invalidating the signature.
Implementing the same pattern:
signed_string = timestamp + "." + raw_payload_body
signature = hmac_sha256(signed_string, secret)
// On verification:
if abs(now - timestamp) > 300:
reject // payload is too old
verify hmac signaturePseudocode. Verify the exact timestamp format and tolerance against your system’s requirements before implementing.
What Happens When One Consumer Is Slow
At any reasonable scale, you will have consumers that are reliably slow. Not down, not failing. Just slow. They accept the event, they return 200, but they take eight seconds to do it.
If your delivery worker waits eight seconds per request and you have a queue backing up, you have a throughput problem. One slow consumer can clog your entire delivery pipeline if your architecture doesn’t isolate them.
The fix is per-endpoint rate limiting and connection timeouts. Set a short timeout on each delivery attempt. If the receiver doesn’t respond within three to five seconds, treat it as a failure and retry with backoff. Don’t let one slow endpoint hold up delivery to other, faster consumers.
For high-volume systems, isolate slow or high-failure endpoints into separate worker pools. If one consumer is consistently unreliable, it should degrade in isolation. It should not steal capacity from the rest of your delivery system.
A circuit breaker helps here too. Track the failure rate per endpoint over a rolling window. When an endpoint’s failure rate crosses a threshold, stop sending to it temporarily and let the circuit breaker trip. After a cooldown period, allow a single test request through. If that succeeds, gradually resume delivery. If it fails, hold.
Here is a self-contained PHP circuit breaker for a single endpoint:
// A per-endpoint circuit breaker with three states: CLOSED, OPEN, HALF_OPEN.
// Store one instance of this per destination URL, keyed by endpoint in your cache or DB.
class WebhookCircuitBreaker
{
private const CLOSED = 'closed';
private const OPEN = 'open';
private const HALF_OPEN = 'half_open';
private string $state = self::CLOSED;
private int $failureCount = 0;
private int $failureThreshold = 5; // trip after 5 consecutive failures
private int $cooldownSeconds = 60; // wait 60s before allowing a test request
private int $openedAt = 0;
// Call this before each delivery attempt.
// Returns false if the breaker is OPEN and the cooldown has not expired.
public function canAttempt(): bool
{
if ($this->state === self::CLOSED) {
return true;
}
if ($this->state === self::OPEN) {
if (time() - $this->openedAt >= $this->cooldownSeconds) {
// Cooldown expired. Allow one test request through.
$this->state = self::HALF_OPEN;
return true;
}
return false;
}
// HALF_OPEN: one test request is already in flight, let it through
return true;
}
// Call this after a successful delivery.
public function recordSuccess(): void
{
$this->failureCount = 0;
$this->state = self::CLOSED;
}
// Call this after a failed delivery attempt.
public function recordFailure(): void
{
$this->failureCount++;
$shouldTrip = $this->state === self::HALF_OPEN
|| $this->failureCount >= $this->failureThreshold;
if ($shouldTrip) {
$this->state = self::OPEN;
$this->openedAt = time();
$this->failureCount = 0;
}
}
public function isOpen(): bool
{
return $this->state === self::OPEN;
}
}Usage in your delivery worker:
// Fetch the circuit breaker for this specific endpoint (from cache/DB in real usage)
$breaker = $this->getCircuitBreaker($event['destination_url']);
if (!$breaker->canAttempt()) {
// Circuit is open. Skip this delivery and reschedule for later.
// Do NOT count this as a failed attempt against the event's retry budget.
scheduleRetry($event, $event['attempt'], 'circuit_open');
return;
}
try {
$response = $this->httpClient->post($event['destination_url'], $event['payload']);
if ($response->getStatusCode() === 200) {
$breaker->recordSuccess();
} else {
$breaker->recordFailure();
scheduleRetry($event, $event['attempt'], 'http_' . $response->getStatusCode());
}
} catch (\Exception $e) {
$breaker->recordFailure();
scheduleRetry($event, $event['attempt'], $e->getMessage());
}Note that when the circuit is open you skip the attempt without burning a retry. The event waits. The endpoint gets a break. When the cooldown expires, a single probe goes through. If it succeeds the circuit closes and delivery resumes normally.
This pattern prevents your system from endlessly hammering a broken endpoint during a prolonged outage. It also gives the downstream system room to recover without being flooded on restart.
The Observability You Actually Need
Three numbers tell you most of what you need to know about a webhook delivery system.
Delivery success rate per endpoint. If one endpoint’s success rate drops significantly while others stay healthy, that endpoint has a problem. If all endpoints drop together, you have a problem.
P99 delivery latency. How long does it take from event creation to confirmed delivery at the 99th percentile? This captures the tail behavior. An average that looks fine can hide a P99 that’s blowing through acceptable limits.
Dead letter queue depth. Monitor this over time, not just as a point-in-time value. A flat DLQ is healthy. A growing DLQ means retries are not succeeding and you need to investigate.
Add alerting on all three. DLQ depth above some threshold, success rate below some floor, latency above some ceiling. These three alerts will catch most production problems before clients report them.
One more thing: log enough context on every delivery attempt to reconstruct what happened without having to guess. The destination URL, the event type, the HTTP response code and body, the attempt number, the timestamp. When debugging a missed event three days later, this log is the only asset you have.
The Gap Is the Product
The naive implementation ships in fifteen minutes. The reliable system takes a lot longer. Most teams don’t build the reliable system until after an incident makes the cost of the naive one visible.
The interesting part is that the gap between them is not really about code. A retry queue is not complicated. Signature verification is a few function calls. Dead letter queues are a configuration setting in most message brokers. None of these pieces are hard.
What’s hard is believing they’re necessary before they’ve already failed you.
Every production incident in a webhook system I’ve seen was obvious in hindsight. The design wasn’t subtle. The failure mode was just never considered before it happened.
The architecture described here is not premature optimization. It’s the minimum viable version of a system that doesn’t lie to the services depending on it. The naive version is a prototype. This is the real thing.
Build the real thing the first time.
If you’ve been burned by a webhook system failing silently, the event you never got is probably still sitting in a queue somewhere with no retry logic, no DLQ and no one who knows to look for it.
