Most background job failures are not bad luck. They are bad architecture baked in from day one.
There is a specific kind of incident that every backend developer eventually faces. The queue has been running for months. Nothing looks wrong. Then one Tuesday afternoon, a critical feature stops working quietly, and when you dig in, you find thousands of stuck jobs, duplicate emails sent to users and a database that has been slowly choked by queue polling since the day you launched. Nobody noticed because nobody was watching.
The failure was not bad luck. The failure was there from the beginning, built into every architectural decision the team made when “we just need background jobs” felt like a small problem.
Queue architecture is one of those topics that only gets serious attention after something breaks badly. This is about fixing it before that happens.
The Queue Driver Trap
Most applications start with the database queue driver. It is the path of least resistance. No extra infrastructure, no Redis setup, no configuration headaches. You run a migration, set QUEUE_CONNECTION=database in your .env, and you are done.
This is fine for local development. It is a liability in production.
The database queue driver works by polling a jobs table. A worker runs a query on an interval to fetch the next available job. At low volume, this is invisible. At moderate volume, with a few workers running concurrently, those constant SELECT and UPDATE statements start to show up in your slow query log. At high volume, the queue table becomes a contention bottleneck, and every background job competes for locks with your actual application queries running on the same connection pool.
The problem compounds because teams add workers to keep up with backlog, which generates more polling queries, which increases load, which creates more delays. The symptom is a slow queue. The cause is using a transactional database for a workload it was not designed to handle.
Redis solves this. It is purpose-built for exactly this kind of work: fast, in-memory, non-blocking, with list-based data structures that map cleanly to queue semantics. In Laravel, switching from the database driver to Redis is a one-line environment change once Redis is running.
QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PORT=6379If you are on a small VPS and Redis feels like overhead, it is not. A single Redis instance handling queue operations for a mid-sized application consumes less than 50MB of memory under normal load. The database you saved by avoiding Redis is running orders of magnitude more expensive queries because of that choice.
Idempotency Is Not Optional
Here is a scenario. Your application sends a welcome email when a new user registers. That email dispatch is handled by a queued job. The job runs, calls your mail provider’s API, and then the worker crashes before it marks the job as complete. The queue driver sees the job as unfinished and runs it again. Your user receives two welcome emails.
This is not an edge case. This is normal queue behavior. Every queue system that offers reliability guarantees does so through at-least-once delivery. That means any job can run more than once. Your job code has to handle that.
Idempotency means the job produces the same result whether it runs once or twenty times. This is not automatic. You have to design it.
For external API calls, this means storing whether the call was made before dispatching it again:
class SendWelcomeEmail implements ShouldQueue
{
public function handle(): void
{
$key = 'welcome_email_sent:' . $this->user->id;
if (Cache::has($key)) {
return;
}
Mail::to($this->user)->send(new WelcomeMail($this->user));
Cache::put($key, true, now()->addDays(7));
}
}This is a simplified example. For payment processing or anything with financial consequences, use a dedicated idempotency token stored in your database, not the cache. Cache eviction under memory pressure can wipe your lock, and then you are back to duplicate charges.
The rule is direct: if your job touches money, external APIs or sends notifications, it must be idempotent. Period. The queue will retry. Your code needs to be ready for that.
Dead Letter Queues: Your Operational Safety Net
Most Laravel applications use the failed_jobs table as their answer to failure handling. A job exceeds its retry limit, it lands in failed_jobs, and it sits there until someone notices or the table grows large enough to be alarming.
This is not a dead letter queue. It is a graveyard.
A dead letter queue (DLQ) is an active operational tool. Failed messages land in a separate, monitored queue. You set alerts on that queue’s depth. When the depth crosses a threshold, something on your team responds. The SLA I use: no message in a DLQ should be older than 24 hours without someone actively triaging it.
In AWS SQS, the DLQ setup is explicit. You configure a source queue to forward messages that exceed a maxReceiveCount to a dedicated DLQ ARN. The DLQ is a normal SQS queue that you treat as high-priority.
{
"RedrivePolicy": {
"deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:my-app-dlq",
"maxReceiveCount": 3
}
}In Laravel with Redis, the equivalent is the failed queue plus active monitoring via Horizon's failed job metrics. The behavior is similar but the operational tooling around SQS DLQs is more mature, particularly when integrated with CloudWatch alerts.
What teams consistently skip is the replay strategy. When the root cause of failures is fixed (a third-party API was down, a schema change broke a serialized payload), you need a way to replay DLQ messages back to the original queue. In SQS, that is a manual redrive policy. In Laravel, that is php artisan queue:retry all or queue:retry {id}.
Neither of these is automatic. Build the runbook before you need it, not after.
Worker Configuration That Actually Holds Up
Long-running queue workers accumulate memory. PHP does not have the same memory management guarantees as a Go service or a Rust process. If your worker processes thousands of jobs without restarting, it will eventually grow large enough to get killed by the OS or start behaving unpredictably due to leaked references.
The fix is not to write better PHP. The fix is to configure workers to restart themselves after a controlled interval.
Laravel’s queue worker accepts two flags that most teams do not use:
php artisan queue:work --max-jobs=500 --max-time=3600--max-jobs tells the worker to stop gracefully after processing 500 jobs. --max-time tells it to stop after 3600 seconds (one hour), regardless of job count. When the worker stops, Supervisor restarts it. Memory resets. Problem avoided.
Your Supervisor configuration should look something like this:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --max-jobs=1000 --max-time=3600 --sleep=3 --tries=3
autostart=true
autorestart=true
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/worker.log
stopwaitsecs=3600The stopwaitsecs value is important. It must be greater than or equal to --max-time. Without it, Supervisor will send a SIGKILL to your worker mid-job if a graceful stop takes longer than the default timeout.
One more thing teams consistently get wrong: running a single worker process. One worker means one job at a time. If that job blocks for 30 seconds waiting on an external API, everything behind it waits. Set numprocs to match the concurrency your workload needs, but keep it under the number of available Redis connections your application is configured to handle.
The SQS vs Redis Decision Framework
Redis and SQS are not interchangeable. They have different guarantees, different failure modes and different operational profiles.
Redis is the right choice when:
- You need low-latency job dispatch and processing
- You want queue observability through a dashboard (Horizon provides this out of the box)
- Your infrastructure already runs Redis for caching
- You are on a single server or small cluster
SQS is the right choice when:
- You need durability guarantees beyond what a single Redis instance provides
- You are already deep in the AWS ecosystem and want native integrations with CloudWatch and IAM
- You need to decouple producers and consumers across different services or accounts
- You are processing jobs that take longer than the Redis queue’s practical limits
The gotcha with SQS that burns teams regularly: the visibility timeout. When SQS delivers a message to a worker, it hides that message from other consumers for a configurable period. The default is 30 seconds. If your job takes 45 seconds to process, SQS makes the message visible again before your worker finishes. Another worker picks it up. Now you have two workers running the same job simultaneously.
The fix is to set the visibility timeout to comfortably exceed your longest expected job runtime. If your slowest job takes 5 minutes, set the visibility timeout to at least 10 minutes. Better, set it per-message from within the job itself as you get a clearer picture of actual runtimes.
// Extend visibility timeout inside a long-running SQS job
$this->job->getJobId(); // SQS message ID
// Call ChangeMessageVisibility with new timeout before processing completesIf you are using SQS through Laravel, the queue config handles the initial visibility timeout but you need explicit extension logic for long-running jobs.
Observability Is Where It All Falls Apart
You cannot fix what you cannot see. Most queue setups have exactly one observability mechanism: the failed_jobs table. Developers check it when users report something is broken.
That is not observability. That is reactive debugging.
What you actually need:
Queue depth over time. If jobs are accumulating faster than workers process them, you have a problem before users notice it. Alert when queue depth exceeds a threshold that represents your acceptable processing lag.
Job throughput by type. Knowing that your queue processed 10,000 jobs today is less useful than knowing it processed 9,800 ProcessPayment jobs and 200 SendNotification jobs. When throughput for a specific job type drops, you can trace it directly.
P95 processing time per job type. Average processing time hides outliers. A job that usually takes 200ms and occasionally takes 45 seconds will show a perfectly acceptable average while consistently blowing your SQS visibility timeout. P95 tells you the truth.
Failed job rate as a percentage of total jobs. An absolute count of failures is noisy. A failure rate above 1% is worth investigating. Above 5% is an incident.
Laravel Horizon gives you most of this for Redis queues. It tracks throughput, runtime and failure metrics through a real-time dashboard at /horizon. The snapshot command populates historical metrics:
// routes/console.php
Schedule::command('horizon:snapshot')->everyFiveMinutes();For SQS, CloudWatch metrics cover queue depth (ApproximateNumberOfMessagesVisible), messages in-flight (ApproximateNumberOfMessagesNotVisible) and age of the oldest message (ApproximateAgeOfOldestMessage). The last one is the metric most teams never configure an alert for, and it is the most useful signal for detecting queue stalls.
Stop Building for Day One
Queue architecture failures follow a consistent pattern: decisions made for day-one convenience calcify into production liabilities. The database driver stays in production because switching feels risky. Jobs never get idempotency logic because they have not failed yet. The DLQ never gets set up because it seems like premature optimization.
None of these feel urgent until a bad deploy, a third-party outage or an unexpected traffic spike turns them into an incident at 2am.
The three changes worth making this week:
- Audit your queue driver. If it is
databasein production, schedule the Redis migration. It is a one-day effort with a measurable impact on database query load. - Pick three of your most critical job types and make them explicitly idempotent. Payment jobs, notification jobs and anything that calls an external API are the highest priority.
- Set up a DLQ or equivalent monitoring with a concrete alert threshold. Failed job count plus time-in-DLQ are the two metrics that matter.
The queue does not care how busy your team is. It will keep processing, keep failing and keep accumulating problems at exactly the rate your current architecture allows. The choice is whether you address this before or after users notice.
