What the defaults are actually doing to your task queue in production
You deployed at noon on a Tuesday. Rolling restart, one pod at a time. Nothing alarming in the logs. Then someone messages: the report generation jobs queued before the deploy never ran. No failures. No retries. Just gone.
Celery ate them. Quietly.
The reason is buried in a default setting most engineers never read past the quickstart. And it is not the only one.
Celery 5.6.3, released March 26, 2026, ships with defaults optimised for getting started, not for running things in production. The documentation is honest about this. The tutorials are not. So teams copy the quickstart config, build on top of it, and then spend days debugging symptoms that trace back to a handful of settings nobody thought to change.
Here are the five that cause the most damage.
1. task_acks_late Is False, So Your Tasks Vanish on Worker Restart
The default:task_acks_late = False
When a Celery worker picks up a task, it sends an acknowledgment to the broker immediately. Right then, before executing a single line of your code. From the broker’s perspective, that task is done.
If the worker crashes mid-execution (OOM kill, deployment restart, segfault), the broker never requeues it. The task is acknowledged. Gone. No retry. No error in your monitoring. Just silence.
This is what swallowed those Tuesday jobs. The worker picked them up, acked them, then got replaced by the new pod before they finished running.
Setting task_acks_late = True flips this. The acknowledgment only goes out after the task completes successfully.
But there is a catch. By default, even with acks_late=True, Celery still acknowledges the task if the worker process is terminated by a signal. A process killed by SIGKILL (the OOM killer, for instance) will not magically succeed on retry. Celery assumes deliberate termination means you do not want the task rerun.
To override this and ensure tasks are requeued even when the process is killed:
# celery config: add both together, never one without the other
app.conf.update(
task_acks_late=True,
task_reject_on_worker_lost=True,
)With both set, a task picked up by a dying worker gets returned to the queue. With only acks_late=True, you still lose tasks on hard process termination.
The trade-off is real: your tasks must be idempotent. If the task ran to 90% completion before the worker died, it will run again from zero. If your task sends a payment or fires a webhook, you need to handle double execution. But losing tasks silently is worse than running them twice. At least you can design around idempotency.
What to set:
task_acks_late = Truetask_reject_on_worker_lost = True- Make tasks idempotent before enabling both
2. worker_prefetch_multiplier Is 4, Which Starves Your Fast Tasks
The default:worker_prefetch_multiplier = 4
Each Celery worker process prefetches four tasks from the broker at startup and keeps them in memory. The idea is throughput: grab tasks in batches, reduce round trips to the broker.
The problem shows up the moment your queue has two task types with different execution times.
Say you have a send_notification task that completes in 50 milliseconds and a generate_pdf task that takes 40 seconds. Both go into the default queue. A worker process grabs four tasks via prefetch, and if three of those happen to be generate_pdf jobs, that worker is locked up for 120 seconds. The send_notification tasks sitting behind them in that worker's prefetch buffer cannot move to another worker. They are reserved. Which sounds fine until you realize a 50ms notification is now waiting two minutes because it lost a coin flip at pickup time.
This is not a queue throughput problem. It is a prefetch reservation problem.
The fix depends on your setup:
Option A: set prefetch multiplier to 1
app.conf.worker_prefetch_multiplier = 1One task at a time per worker process. Slightly lower throughput for fast tasks, but fair scheduling. Good starting point for any queue that mixes fast and slow tasks.
Option B: separate queues per task type
# define separate queues at task level
@app.task(queue='fast')
def send_notification(user_id, message):
...
@app.task(queue='slow')
def generate_pdf(report_id):
...Then run dedicated workers per queue with the -Q flag:
# fast tasks: more workers, low prefetch
celery worker -Q fast -c 8 -O fair
# slow tasks: fewer workers, prefetch 1
celery worker -Q slow -c 2 -O fairOption B is more operationally complex but gives you independent scaling and cleaner failure isolation. Option A is the right default for teams that have not measured the problem yet.
The multiplier of 4 made sense when Celery tasks were fast, homogeneous and latency did not matter. Most production queues are none of those things.
3. The Visibility Timeout Will Run Your Long Tasks Twice
The default (Redis broker):visibility_timeout = 3600 seconds (1 hour)
The visibility timeout is the number of seconds after a worker picks up a task before the broker assumes the worker died and requeues it. One hour. Seems generous.
Until your task takes 75 minutes.
At the 60-minute mark, Redis decides the worker is dead (even though it is not) and puts the task back in the queue. Another worker picks it up and starts from scratch. Now you have two workers running the same task simultaneously. If that task creates database records, sends emails or charges a card, you have a problem.
The issue compounds with acks_late=True, which you may have set specifically to protect against worker crashes. A reasonable thing to do. Except that if the visibility timeout expires while the task is still alive and running, Redis requeues it regardless of ack status, and now you have two workers doing the same work simultaneously with no error signal anywhere.
The fix: set visibility_timeout to a value longer than your longest-running task. If your slowest task runs for two hours, set it to three:
app.conf.broker_transport_options = {
'visibility_timeout': 10800, # 3 hours in seconds
}More importantly, know what your actual long-tail task duration looks like. If you do not have that metric, add it. Task duration percentiles matter more than average duration. The task that runs in 59 minutes 99% of the time and takes 62 minutes on the 1% spike is the one that will cause this.
Celery 5.5 introduced worker_soft_shutdown_timeout (disabled by default, set to 0.0 seconds) which gives workers a grace period to finish executing tasks before shutdown. That helps during deploys. It does not help for visibility timeout expiry while a task is already running.
What to set:
visibility_timeoutto at least 2x your observed maximum task duration- Monitor task duration at the p95 and p99 level, not just average
- For tasks that genuinely need hours, break them into smaller sub-tasks with checkpoints
4. No Time Limits Means One Stuck Task Holds a Worker Hostage
The default:task_time_limit = None, task_soft_time_limit = None
If a Celery task hangs on a network call that never resolves, gets stuck in a loop, or blocks on a database lock indefinitely, the worker just sits there holding it. The prefetch queue backs up behind it. Other tasks wait. Nothing raises an error anywhere that you’d notice without actively looking for it, and the worker’s health check reports green the whole time.
Celery provides two settings for this:
task_soft_time_limit raises a SoftTimeLimitExceeded exception inside the task, giving it a chance to clean up before dying.
task_time_limit forcibly terminates the worker process if the task exceeds it.
Use them together. Soft limit for graceful cleanup, hard limit as the backstop:
from celery.exceptions import SoftTimeLimitExceeded
@app.task(
bind=True,
soft_time_limit=300, # 5 minutes: raises SoftTimeLimitExceeded
time_limit=360, # 6 minutes: kills the process if still running
)
def process_report(self, report_id):
try:
result = run_expensive_computation(report_id)
return result
except SoftTimeLimitExceeded:
# log, save partial state, release any locks you hold
cleanup_partial_state(report_id)
raise # re-raise so Celery marks the task as failedSetting these globally applies to all tasks unless overridden:
app.conf.update(
task_soft_time_limit=300,
task_time_limit=360,
)The right values depend on what your tasks actually do. A task that hits an external API probably should not run for more than 30 seconds. A task generating a large report might legitimately need 10 minutes. Set the limits based on measurements, not intuition. Without them, one badly-timed third-party API outage can stall your entire worker pool.
5. retry_backoff Is False, Which Creates a Thundering Herd on Failure
The default:retry_backoff = False (note: retry_jitter = True by default)
When a task fails and retries, how long does Celery wait between attempts?
With retry_backoff=False, the answer is: whatever you set in default_retry_delay, which defaults to 180 seconds. Every retry. At a fixed interval.
This sounds fine until 500 tasks fail simultaneously because a downstream service went down. They all retry at T+180s, then T+360s, then T+540s, every wave arriving with exactly the same load that broke the service the first time. The recovering system never gets a chance to breathe.
Exponential backoff fixes this. When retry_backoff=True, the first retry waits 1 second, the second waits 2 seconds, the third waits 4 seconds, doubling each time up to retry_backoff_max (which defaults to 600 seconds).
The jitter setting (retry_jitter, already True by default) randomizes those delays slightly so retries from different workers do not all align. That part is already working for you. The backoff itself is not.
@app.task(
bind=True,
autoretry_for=(Exception,),
max_retries=5,
retry_backoff=True, # exponential: 1s, 2s, 4s, 8s, 16s...
retry_jitter=True, # already True by default, explicit is clearer
retry_backoff_max=600, # cap at 10 minutes
)
def call_external_api(self, payload):
response = requests.post('https://api.example.com/process', json=payload)
response.raise_for_status()
return response.json()One thing to watch: autoretry_for=(Exception,) catches all exceptions. In practice you want to be more specific. Retry on network errors and rate-limit responses. Do not retry on ValidationError or business-logic failures. Those will never succeed and will burn through max_retries before dying the same way.
@app.task(
bind=True,
autoretry_for=(requests.exceptions.ConnectionError, requests.exceptions.Timeout),
max_retries=5,
retry_backoff=True,
retry_backoff_max=600,
)
def call_external_api(self, payload):
...A fixed retry delay with no backoff is not just inefficient. During an incident, it actively prevents recovery.
A Minimal Production Config
For reference, here is what these five settings look like together:
# celery_config.py
app.conf.update(
# Reliable delivery: ack after completion, requeue on crash
task_acks_late=True,
task_reject_on_worker_lost=True,
# Fair scheduling: no prefetch hoarding
worker_prefetch_multiplier=1,
# Visibility timeout: longer than your slowest task
broker_transport_options={
'visibility_timeout': 10800, # 3 hours
},
# Time limits: adjust based on your actual task durations
task_soft_time_limit=300,
task_time_limit=360,
)Retry settings live at the task level because different tasks have different failure modes. There is no single global retry_backoff=True. That is intentional design on Celery's part. A data ingestion task and a payment task do not share the same failure envelope.
One warning before you deploy this: task_acks_late=True with task_reject_on_worker_lost=True means tasks may run more than once. That shifts the idempotency responsibility onto your task code entirely. If your tasks write to a database, add a unique constraint on the task's logical ID. If they call external APIs, check whether the operation already completed before firing the request again. Fix that before changing the ack settings, not after.
Celery’s defaults are optimised for demos. Most of the engineers who copy those defaults into production never come back to change them. Until something disappears.
Thank you for being a part of the community
Before you go:
Whenever you’re ready
There are 4 ways we can help you become a great backend engineer:
- The MB Platform: Join thousands of backend engineers learning backend engineering. Build real-world backend projects, learn from expert-vetted courses and roadmaps, track your learning and set schedules, and solve backend engineering tasks, exercises, and challenges.
- The MB Academy: The “MB Academy” is a 6-month intensive Advanced Backend Engineering Boot Camp to produce great backend engineers.
- Join Backend Weekly: If you like posts like this, you will absolutely enjoy our exclusive weekly newsletter, sharing exclusive backend engineering resources to help you become a great Backend Engineer.
- Get Backend Jobs: Find over 2,000+ Tailored International Remote Backend Jobs or Reach 50,000+ backend engineers on the #1 Backend Engineering Job Board.
