How We Handle Container Restarts: Graceful Shutdown and Zero Downtime
Container restarts are a routine operation: you deploy a new version, the old container is stopped and the new container is started. But if the restart is not handled correctly, in-flight requests are dropped, users see errors, and the app appears to be down. Deployxa handles container restarts with graceful shutdown, which ensures zero dropped requests and zero downtime. Here is how it works.
The direct answer is that Deployxa's container restart process has four steps: signal (send SIGTERM to the container), drain (stop sending new requests, let in-flight requests complete), timeout (wait a configurable period for in-flight requests to finish), and kill (send SIGKILL if the container does not exit). The platform also uses blue/green deployments, which means the new container is started before the old container is stopped, ensuring there is always a healthy container to handle requests. For more on blue/green deployments, see our article on Traefik v3 dynamic routing.
The Restart Process
Here is the step-by-step process for a container restart during a blue/green deployment.
Step 1: The new container starts
When you deploy a new version, the new container (green) starts alongside the old container (blue). The new container is not yet receiving traffic; it is starting up and initializing.
Step 2: The readiness engine checks the new container
The readiness engine runs the 14-point check on the new container. If the check passes (grade A or B), the traffic swap proceeds. If the check fails, the old container continues to handle traffic, and the deployment is aborted.
Step 3: Traffic switches to the new container
Traefik switches traffic from the old container to the new container atomically. New requests go to the new container, while in-flight requests to the old container are allowed to complete.
Step 4: The old container receives SIGTERM
The old container receives a SIGTERM signal, which tells it to shut down gracefully. The container should stop accepting new requests, finish in-flight requests, close database connections, and exit.
Step 5: The drain period
The platform waits for a configurable drain period (default 30 seconds) for the old container to finish in-flight requests and exit. During this period, the old container is still running but not receiving new traffic.
Step 6: The old container is killed (if needed)
If the old container does not exit within the drain period, the platform sends a SIGKILL signal, which forcefully terminates the container. Any in-flight requests that were still running are dropped (but this should be rare, because the drain period is usually long enough).
How to Implement Graceful Shutdown in Your App
For the graceful shutdown to work, your app needs to handle the SIGTERM signal correctly. Here is how to implement it in Node.js.
const server = app.listen(process.env.PORT || 3000);
let isShuttingDown = false;
app.use((req, res, next) => {
if (isShuttingDown) {
res.set('Connection', 'close');
return res.status(503).json({ error: 'Server is shutting down' });
}
next();
});
process.on('SIGTERM', () => {
console.log('SIGTERM received, shutting down gracefully...');
isShuttingDown = true;
server.close(() => {
console.log('All connections closed, exiting...');
process.exit(0);
});
// Force exit after 30 seconds
setTimeout(() => {
console.error('Forcing exit after 30 seconds');
process.exit(1);
}, 30000);
});This code does three things: (1) it sets a flag when SIGTERM is received, (2) it rejects new requests with a 503 status (which tells the load balancer to route to another container), and (3) it waits for in-flight requests to complete before exiting.
Common Pitfalls and Troubleshooting
The first pitfall is not handling SIGTERM. If your app does not handle SIGTERM, the container is killed immediately (after the drain period), which drops in-flight requests. The fix is to handle SIGTERM and implement graceful shutdown. The second pitfall is not rejecting new requests during shutdown. If your app continues to accept new requests during shutdown, those requests might not complete before the container is killed. The fix is to set a flag (e.g., isShuttingDown) and reject new requests with a 503 status. The third pitfall is not closing database connections. If your app does not close database connections during shutdown, the connections are left open, which can cause issues for the database. The fix is to close database connections in the SIGTERM handler. The fourth pitfall is not setting a timeout. If your app waits indefinitely for in-flight requests to complete, the container might not exit within the drain period, which means it is forcefully killed. The fix is to set a timeout (e.g., 30 seconds) and to force exit after the timeout. The fifth pitfall is not testing graceful shutdown. If you do not test it, you might not know that it does not work until a production deployment drops requests. The fix is to test graceful shutdown by sending SIGTERM to your app during a load test and verifying that no requests are dropped.
How Container Restarts Integrate with Blue/Green Deployments
Container restarts and blue/green deployments work together to ensure zero downtime. In a blue/green deployment, the new container (green) is started before the old container (blue) is stopped, which means there is always a healthy container to handle requests. The traffic swap is atomic (via Traefik), which means no requests are dropped during the swap. The old container's graceful shutdown ensures that in-flight requests to the old container complete before it is killed. For more on blue/green deployments, see our articles on Traefik v3 dynamic routing and the agentic blue/green deployment pipeline.
Advanced Container Restart Patterns
Beyond the basics, the container restart system supports several advanced patterns. The first is connection draining. During a restart, the system drains in-flight requests (stops sending new requests, lets in-flight requests complete) before killing the container. The drain period is configurable (default 30 seconds), which means long-running requests (e.g., file uploads, report generation) have time to complete. The system also sends a Connection: close header during the drain period, which tells clients to close the connection after the current request and reconnect to a different container.
The second pattern is health-aware restarts. Before restarting a container, the system checks the container's health (via the readiness engine). If the container is unhealthy, the system restarts it immediately (because it is not serving traffic anyway). If the container is healthy, the system starts the new container first, waits for it to be healthy, and then drains and kills the old container. This ensures there is always a healthy container serving traffic.
The third pattern is rolling restarts. For apps with multiple containers, the system restarts containers one at a time (not all at once), which means there are always healthy containers serving traffic during the restart. The rolling restart waits for each container to be healthy before restarting the next one, which ensures zero downtime.
The fourth pattern is restart backoff. If a container crashes repeatedly (e.g., due to a bug), the system uses exponential backoff for restarts (e.g., 1s, 2s, 4s, 8s, 16s, 32s, 64s), which prevents a crash loop from consuming excessive resources. After a maximum number of restarts (e.g., 5), the system stops restarting and alerts the team.
The fifth pattern is restart reasons. The system logs the reason for each restart (e.g., deployment, health check failure, manual restart, OOM kill, crash), which is essential for debugging. The restart reason is displayed in the dashboard and is available via the MCP server.
How Container Restarts Integrate with the Readiness Engine
Container restarts and the readiness engine work together to ensure zero downtime. During a restart, the readiness engine runs the 14-point check on the new container. If the check passes (grade A or B), the new container is added to the load balancer and starts receiving traffic. If the check fails, the new container is not added, and the old container continues to serve traffic. This ensures that a restart never degrades the app's reliability. For more on the readiness engine, see our article on the health check system.
Lessons Learned
Building the container restart system taught us several lessons. First, graceful shutdown is essential. Without graceful shutdown, in-flight requests are dropped during restarts, which causes errors for users. With graceful shutdown (SIGTERM, drain period, request completion), in-flight requests complete before the container is killed, which means zero dropped requests. Second, the drain period needs to be long enough. If the drain period is too short (e.g., 5 seconds), long-running requests (e.g., file uploads) are dropped. The default drain period (30 seconds) is sufficient for most apps, but apps with very long-running requests might need a longer period. Third, blue/green deployments are the gold standard. By starting the new container before stopping the old one, there is always a healthy container serving traffic, which means zero downtime. This is more reliable than rolling restarts (which restart containers one at a time) and much more reliable than in-place restarts (which restart the container in place, causing a brief downtime). Fourth, crash loop detection is important. If a container crashes repeatedly, restarting it immediately creates a crash loop that consumes excessive resources. Exponential backoff prevents the crash loop, and a maximum restart limit ensures the team is alerted. Fifth, the restart reason is essential for debugging. Without the restart reason, you do not know why a container was restarted (was it a deployment? a health check failure? a crash? an OOM kill?). With the restart reason, you can quickly identify and fix the root cause. For more on Deployxa's engineering, see our articles on Traefik v3 dynamic routing and the build cache architecture.
Conclusion: Zero Downtime with Graceful Shutdown
Container restarts can drop requests if not handled correctly, but Deployxa's graceful shutdown process ensures zero dropped requests and zero downtime. By sending SIGTERM, draining in-flight requests, and using blue/green deployments, the platform ensures that your app is always available, even during deployments. For more on Deployxa's engineering, see our articles on the build cache architecture and the health check system. Learn about how we handle custom domains and the audit log system in our companion articles. Explore our free developer tools to speed up your workflow. Try Deployxa Drop for an instant live preview with zero signup.