The most dangerous assumption in API integration is that a 200 OK means the job is done. It does not. It means the job started.
I shipped a webhook integration for a client in 2023. Payment gateway triggers webhook, our system receives it, processes the transaction, updates the record. Clean. Simple. Done.
Six months later, their payment gateway provider sent me an email. They had been retrying a batch of failed deliveries for the past three days. Our endpoint had been returning 500 errors during a maintenance window. Forty-seven transactions were in a retry queue. Some had been delivered twice by the time we noticed. Two of them had been processed twice.
The client had two duplicate charges to investigate, one angry customer and a reconciliation problem that took a week to untangle.
We had a webhook. We did not have a webhook system. There is a difference and it matters a lot.
What “Fire and Forget” Actually Means
Webhooks are described as fire and forget because that is how they feel to implement. You register a URL with a third party. They POST to it when something happens. You respond with a 200. Done.
The problem is that “fire and forget” describes the sender’s perspective, not yours. To the payment gateway, it is fire and forget. They sent the request. Your endpoint responded. Their responsibility ends there, unless they have retry logic built in. And even then, their retry window has limits.
From your side, you are not forgetting anything. You are receiving a payload that represents a real-world event. A payment. An order status change. A document signed. A subscription cancelled. These events have business consequences. They need to be processed. They need to be processed exactly once. They need to be processed even if your server hiccups for thirty seconds.
Most webhook implementations handle the happy path. The server is up, the payload is valid, the processing succeeds, the 200 goes out. Everybody happy.
Production does not only run the happy path.
The Four Ways Webhooks Actually Break
Before you can build a reliable webhook receiver, you need to understand what breaks.
Your server goes down. Deployments, memory exhaustion, unhandled exceptions that crash the process. When your endpoint is unavailable and the sender has no meaningful retry strategy, the event is lost. If the sender does retry, you now have to handle duplicates.
Your server is slow. Some webhook senders have short response windows before they mark a delivery as failed and retry. If your processing logic takes too long because it is doing synchronous database writes, calling a third-party API or running inside a PHP process that is under load, the sender retries. Now you have a duplicate.
Your processing logic fails after you return 200. You respond quickly to avoid the timeout, then hand off to a queue. The queue worker crashes mid-processing. The event was acknowledged as received but never actually processed. Nobody knows.
Duplicate delivery. Even reliable senders like Stripe deliver webhooks more than once under certain conditions. Their documentation explicitly states this. If your processing logic is not idempotent, the second delivery causes real damage. A charge posted twice. An email sent twice. An inventory count decremented twice.
These are not edge cases. These are the four scenarios you need to handle by default.
The Incident I Keep Telling Developers About
The duplicate charge story is the one I use most. Let me walk through exactly what happened.
Our webhook endpoint received a payment.completed event and did three things synchronously: updated the payment record, triggered a fulfillment job and sent a confirmation email. All three happened inside the controller, before returning the response.
The fulfillment job hit an external API. That API was slow one afternoon. The request took long enough that Stripe’s response window expired. Stripe does not publish a hard number in their docs. They just say “quickly return a 2xx.” The practical limit before they mark a delivery failed and retry is short. We hit it.
The second delivery arrived while the first was still processing. Both passed our basic validation. Both updated the payment record (which happened to be idempotent because it was a simple status update). Both triggered fulfillment. The customer got two shipments.
The fix required three things we did not have: a delivery log to detect duplicates, a signature check to verify the payload was genuine and processing moved out of the request cycle entirely.
We rebuilt the endpoint in an afternoon. We should have built it that way the first time.
Building a Webhook Receiver That Does Not Embarrass You
Here is the actual implementation in Laravel. Not theory. Not pseudocode. The real structure.
Step 1: Verify the signature first
Every serious webhook provider signs their payloads. Stripe uses HMAC-SHA256. Do not process anything before verifying the signature. This protects you from replayed requests, forged payloads and testing against your production endpoint.
public function handle(Request $request): JsonResponse
{
$payload = $request->getContent();
$signature = $request->header('Stripe-Signature');
$secret = config('services.stripe.webhook_secret');
try {
$event = \Stripe\Webhook::constructEvent($payload, $signature, $secret);
} catch (\Stripe\Exception\SignatureVerificationException $e) {
return response()->json(['error' => 'Invalid signature'], 400);
}
// Safe to process now
ProcessWebhookJob::dispatch($event->id, $event->type, $event->data->object);
return response()->json(['received' => true]);
}Return 200 immediately after dispatching the job. Do not process inline.
Step 2: Log every delivery before you touch it
Before any processing logic runs, write the raw payload to a webhook_deliveries table. This gives you a permanent audit trail, lets you replay events and is your first line of defence against duplicates.
// Migration
Schema::create('webhook_deliveries', function (Blueprint $table) {
$table->id();
$table->string('provider');
$table->string('event_id')->unique(); // provider's event ID
$table->string('event_type');
$table->json('payload');
$table->enum('status', ['pending', 'processed', 'failed'])->default('pending');
$table->text('error')->nullable();
$table->timestamps();
});Step 3: Enforce idempotency using the event ID
Every webhook provider gives each event a unique ID. Stripe uses evt_xxxxx. Use it.
class ProcessWebhookJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public function __construct(
private string $eventId,
private string $eventType,
private array $data
) {}
public function handle(): void
{
$delivery = WebhookDelivery::firstOrCreate(
['event_id' => $this->eventId],
[
'provider' => 'stripe',
'event_type' => $this->eventType,
'payload' => $this->data,
'status' => 'pending',
]
);
// Already processed. Bail out safely.
if ($delivery->status === 'processed') {
return;
}
try {
$this->process($delivery);
$delivery->update(['status' => 'processed']);
} catch (\Throwable $e) {
$delivery->update([
'status' => 'failed',
'error' => $e->getMessage(),
]);
throw $e; // Let Laravel's queue retry handle it
}
}
private function process(WebhookDelivery $delivery): void
{
match ($delivery->event_type) {
'payment_intent.succeeded' => $this->handlePaymentSucceeded($delivery->payload),
'customer.subscription.deleted' => $this->handleSubscriptionCancelled($delivery->payload),
default => null,
};
}
}The firstOrCreate call on event_id is your idempotency gate. If the event was already processed, the job exits cleanly. If it is a genuine first delivery, it proceeds. If the same event arrives three times, only one processing cycle runs.
Step 4: Use a dedicated queue worker for webhooks
Do not throw webhook jobs onto your default queue alongside email sends and report generation. Webhooks have stricter time constraints. Give them their own queue and their own worker.
// config/queue.php, webhook-specific connection
'webhook' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'webhooks',
'retry_after' => 90,
'block_for' => null,
],
# In your supervisor config
php artisan queue:work --queue=webhooks --tries=5Note: --backoff on the CLI only accepts a single integer. For step-based exponential backoff, define it on the job class itself using public array $backoff = [30, 60, 120];. The job property takes precedence over the CLI flag.
The Outbound Side: When You Are the One Sending Webhooks
If you are building a platform that sends webhooks to your clients, and eventually you will be, the problem flips. Now you need to handle their servers going down.
The same principles apply in reverse.
Store every outbound delivery in a webhook_dispatches table. Include the response code, response body and delivery timestamp. Schedule retries with exponential backoff. Stop retrying after a threshold (usually 24 to 72 hours) and alert the client that their endpoint is unreachable.
Sign your payloads with HMAC-SHA256 and document the verification steps in your API docs. Clients who care about security will check. Clients who do not will at least have the mechanism available when something goes wrong.
Build an endpoint in your dashboard where clients can view recent deliveries and trigger a manual replay. This saves your support team three emails every time a client’s server has a rough morning.
The minimum viable outbound webhook sender in Laravel looks like this:
class DispatchWebhookJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public int $tries = 5;
public array $backoff = [30, 120, 300, 600, 3600];
public function __construct(
private WebhookDispatch $dispatch
) {}
public function handle(): void
{
$payload = json_encode($this->dispatch->payload);
$signature = hash_hmac('sha256', $payload, config('webhooks.secret'));
$response = Http::withHeaders([
'Content-Type' => 'application/json',
'X-Webhook-Signature' => $signature,
'X-Webhook-Event' => $this->dispatch->event_type,
])->timeout(15)->post($this->dispatch->endpoint_url, $this->dispatch->payload);
$this->dispatch->update([
'response_status' => $response->status(),
'response_body' => $response->body(),
'delivered_at' => now(),
'status' => $response->successful() ? 'delivered' : 'failed',
]);
if (!$response->successful()) {
throw new \Exception("Webhook delivery failed with status {$response->status()}");
}
}
public function failed(\Throwable $exception): void
{
$this->dispatch->update(['status' => 'exhausted']);
// Notify the client their endpoint is unreachable
}
}What “Reliable” Actually Looks Like
There is a short version of everything above that you can use as a checklist the next time you are building a webhook integration.
Verify signatures before processing anything. Log every delivery with the provider’s event ID. Return 200 immediately and process asynchronously. Use the event ID as your idempotency key. Separate webhook processing onto its own queue. Build retry and replay into outbound webhook senders.
None of this is complicated. None of it requires a new library or a paid service. It requires treating webhooks as what they are: distributed messaging with all the failure modes that implies, rather than HTTP requests that always succeed.
The fire-and-forget model is useful shorthand for how webhooks are designed. It is a terrible model for how you implement them.
The day your client’s server goes down is not a question of if. It is a question of whether your system handles it cleanly or creates a reconciliation problem that someone spends a week fixing.
Build the system that handles it cleanly. You will thank yourself at 2am.
