Every deploy you run is a coin flip between clean and catastrophic. Here’s why.
We had a payment service that processed about 400 requests per second. Every Friday afternoon, the platform team pushed infrastructure updates. And every Friday afternoon, a handful of users saw their checkout fail with a 502.
Nobody panicked. The error rate was under 0.1%. Monitoring dashboards stayed green. The on-call engineer marked it as transient. For three months, we blamed the load balancer.
It was not the load balancer.
The payment service had no graceful shutdown handler. When Kubernetes sent SIGTERM during a rolling update, the process died mid-request. Connections dropped. Database transactions hung open until the connection pool timed them out. A few unlucky users got charged without receiving a confirmation, because the webhook callback never fired.
Three months of “transient” errors. All because nobody taught the server how to stop.
SIGTERM Is Not a Suggestion
When your process receives SIGTERM, the operating system is asking it to terminate. Not demanding. Asking. Your process gets a chance to finish what it’s doing, close connections, flush buffers and exit cleanly.
Most backend frameworks do nothing with this signal by default.
Go applications exit immediately. Python applications running under Uvicorn or Gunicorn depend entirely on the server configuration. Laravel queue workers check a boolean flag on the next loop iteration, which means a long-running job might not see the signal for minutes. Node.js applications with active keep-alive connections will sit there indefinitely, waiting for connections that never close on their own.
The result is the same across every stack: your application gets told to stop, and it either dies too fast or refuses to die at all.
In Kubernetes, this distinction has teeth. The default terminationGracePeriodSeconds is 30 seconds. Your pod gets SIGTERM, then a 30-second countdown starts. If your process hasn't exited by then, Kubernetes sends SIGKILL. No negotiation. No cleanup. The process is gone, and whatever it was doing is gone with it.
The Three Ways Your Server Dies Badly
Most shutdown failures fall into one of three patterns. Understanding which one your application has is the first step to fixing it.
Pattern 1: The Instant Death. Your server receives SIGTERM and exits immediately. In-flight HTTP requests get a connection reset. Database transactions are abandoned mid-write. Background jobs vanish. This is the default behavior in Go, C and Rust applications unless you explicitly handle the signal.
Pattern 2: The Zombie. Your server ignores SIGTERM entirely. It keeps accepting new requests during the grace period. It keeps processing. Then SIGKILL arrives and everything dies at once, which is even worse than Pattern 1 because now you have 30 seconds of additional in-flight work that also gets destroyed. This happens in Node.js applications where server.close() is never called, and in Python applications running behind process managers that swallow the signal.
Pattern 3: The Half-Shutdown. Your server stops accepting new HTTP requests but forgets about everything else. The background worker is still pulling from the queue. The WebSocket connections are still open. The scheduled task is still running. The HTTP layer shut down cleanly, but the rest of the application is still alive when SIGKILL arrives.
Pattern 3 is the most common in production applications, because it’s the one that looks correct until something breaks.
What a Clean Shutdown Actually Looks Like
A proper graceful shutdown has four phases, and they have to happen in order.
Phase 1: Stop accepting new work. Mark the service as unhealthy so the load balancer stops routing traffic. In Kubernetes, this means your readiness probe should start returning a non-200 status. Stop pulling from message queues. Reject new WebSocket connections.
Phase 2: Drain in-flight work. Let active HTTP requests finish. Let currently processing queue jobs complete. Let open database transactions commit or roll back. Set a timeout for this phase. If a request hasn’t finished in 15 seconds, it probably never will.
Phase 3: Close resources. Close database connection pools. Disconnect from Redis. Flush any buffered logs or metrics. Close file handles.
Phase 4: Exit. Return a zero exit code so the orchestrator knows it was intentional.
The order matters. If you close the database pool before draining HTTP requests, every in-flight request that needs a database call will fail. If you stop the readiness probe but keep pulling from the queue, you’re doing new work that might get killed.
Go: The Standard Library Already Solved This
Go’s net/http package has had Server.Shutdown() since Go 1.8. It closes all open listeners, then closes all idle connections, then waits for active connections to return to idle before returning. You provide a context with a timeout.
// Graceful shutdown in Go using the standard library
func main() {
srv := &http.Server{Addr: ":8080"}
go func() {
if err := srv.ListenAndServe(); err != http.ErrServerClosed {
log.Fatalf("listen: %v", err)
}
}()
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
<-quit
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Fatalf("shutdown: %v", err)
}
}This handles Phase 2 and Phase 4. But it does not handle Phase 1 or Phase 3. You still need to fail your health check and close your database pool yourself.
The mistake I see most often: teams call srv.Shutdown() and assume the job is done. They forget that Shutdown() only drains HTTP connections. If your application has background goroutines processing queue messages or running periodic tasks, those keep running until SIGKILL arrives.
Python: FastAPI’s Lifespan Is Not Optional
FastAPI uses a lifespan context manager for startup and shutdown logic. Everything before the yield runs at startup. Everything after runs at shutdown.
# FastAPI lifespan for startup and shutdown resource management
from contextlib import asynccontextmanager
from fastapi import FastAPI
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup: initialize resources
db_pool = await create_db_pool()
app.state.db = db_pool
yield
# Shutdown: clean up resources
await db_pool.close()
await flush_metrics()
app = FastAPI(lifespan=lifespan)This replaced the older @app.on_event("startup") and @app.on_event("shutdown") decorators, which were deprecated in favor of the lifespan pattern.
The problem is that the lifespan shutdown only runs after Uvicorn finishes draining requests. If you’re running Uvicorn behind Gunicorn with the --graceful-timeout flag, Gunicorn controls the grace period. If you're running Uvicorn standalone in a container, the SIGTERM goes directly to Uvicorn, and its default behavior is to stop accepting connections and wait for in-flight requests. But there's a known issue: when Uvicorn runs with the --workers flag, signal handling can break, and the shutdown may not propagate to child workers correctly.
If you’re running FastAPI in production, test your shutdown path. Send SIGTERM to your container and watch what actually happens. Don’t assume it works.
Laravel: Queue Workers and the Boolean Trap
Laravel’s queue worker is a while(true) loop. When it receives SIGTERM or SIGQUIT, it sets a $shouldQuit boolean to true. The worker checks this flag at the top of each loop iteration.
This means the worker will finish the current job before stopping. That’s the correct behavior. But if your job takes 10 minutes to run, the worker won’t check the flag for 10 minutes. If Kubernetes has a 30-second grace period, the worker gets killed mid-job.
The fix is to either increase terminationGracePeriodSeconds to match your longest job, or design your jobs to checkpoint their progress so they can resume after being killed.
There’s another trap. If you set block_for to 0 in your queue configuration, the worker blocks indefinitely waiting for the next job. During that blocking wait, it cannot process signals. SIGTERM sits in the signal queue until a job arrives, gets processed, and the loop comes back around to check the flag.
For your artisan commands, Laravel provides the trap() method:
// Signal trapping in a Laravel artisan command
$this->trap([SIGTERM, SIGINT], function (int $signal) {
$this->info("Received signal {$signal}, cleaning up...");
$this->cleanup();
return false;
});This gives you a hook to run cleanup code. Use it for any long-running command that manages resources.
The Kubernetes Timing Problem Nobody Mentions
Here’s the part that catches experienced engineers off guard: Kubernetes sends SIGTERM and updates the Endpoints object at the same time, not sequentially.
Your pod gets SIGTERM. Simultaneously, the Kubernetes control plane tells kube-proxy and ingress controllers to stop routing traffic to your pod. But those updates propagate through the cluster asynchronously. Some nodes learn about the change in milliseconds. Others might take a few seconds.
During that window, your pod is shutting down, but some clients are still sending it requests. If your shutdown handler immediately stops accepting connections, those requests fail with connection refused errors.
This is why preStop hooks exist. A common pattern is to add a short sleep in the preStop hook:
# Kubernetes preStop hook to allow endpoint propagation
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 5"]That 5-second delay gives the cluster time to propagate the endpoint removal. Your pod keeps accepting requests for 5 more seconds, and by the time it actually starts shutting down, the load balancer has already stopped sending new traffic.
The preStop hook runs before SIGTERM is sent to your container. The total time for preStop plus your shutdown handler must fit within terminationGracePeriodSeconds. If your preStop sleeps for 5 seconds and your grace period is 30 seconds, your application has 25 seconds to drain and exit.
Health Checks That Lie During Shutdown
Your readiness probe should fail the moment shutdown begins. Not after you’ve drained connections. Not after you’ve closed the database. The moment you receive SIGTERM, your readiness probe should start returning 503.
Most implementations get this wrong. They tie the readiness check to the database connection. As long as the database is reachable, the probe returns 200. So during shutdown, the probe keeps saying “I’m healthy” even though the server is about to die.
A simple pattern is to use an atomic boolean:
// Atomic boolean for health check during shutdown
var isShuttingDown atomic.Bool
func healthHandler(w http.ResponseWriter, r *http.Request) {
if isShuttingDown.Load() {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
}
// In your signal handler:
isShuttingDown.Store(true)Set the flag as the very first action in your signal handler, before you start draining or closing anything.
Testing Shutdown Is Not Optional
You would not deploy an API endpoint without testing it. But most teams deploy shutdown handlers without ever sending a SIGTERM to their local container.
Here’s a minimal test sequence:
- Start your application in a container.
- Send it a few requests that take 5 seconds to respond. Add a deliberate delay.
- While those requests are in-flight, send
docker kill --signal=SIGTERM <container>. - Verify that every in-flight request gets a complete response.
- Verify that no new connections are accepted after the signal.
- Check your database for orphaned transactions.
- Check your queue for jobs that were processing but never marked complete.
If any of those checks fail, your Friday deployments are gambling with your users’ data.
The Real Cost of Ignoring This
Dropped requests during deploys are not transient errors. They’re architectural failures with a known fix. Every 502 during a rolling update is a connection your server didn’t close properly. Every orphaned database row is a transaction your server didn’t commit. Every duplicate queue message is a job your server didn’t acknowledge.
The fix is usually under 50 lines of code. A signal handler. A shutdown flag. A drain timeout. A preStop hook.
Your server starts hundreds of times during development. It starts once in production and runs for weeks. But it dies every time you deploy. The code that handles that death deserves the same attention as the code that handles a request.
Teach your server how to stop. It already knows how to start.
