The Agentic Incident Response Pipeline
Production incidents at 3 AM are the worst part of running an app. You get woken up by an alert, you SSH into the server, you read the logs, you diagnose the issue, you apply a fix, and you go back to sleep, hoping the fix worked. This is slow, error-prone, and exhausting. What if an AI agent could handle the first three steps (diagnose, fix, notify) automatically, and only wake you up if the fix does not work? With the Deployxa MCP server and an LLM, this is straightforward to build. Here is how to build an agentic incident response pipeline that auto-diagnoses, fixes, and notifies.
The direct answer is that an agentic incident response pipeline is a system that detects production incidents (via monitoring), diagnoses the root cause (via log analysis and LLM reasoning), applies a fix (via code changes or configuration updates), verifies the fix (via health checks), and notifies the team (via Slack or PagerDuty). The pipeline is bounded: it handles known incident types (e.g., container crash, database connection failure, high error rate) and escalates to a human for novel incidents. The Deployxa MCP server provides the monitoring and diagnostic tools, and the LLM provides the reasoning. For more on agentic patterns, see our article on building an AI agent that monitors your app 24/7.
Why Manual Incident Response Does Not Scale
Three problems make manual incident response unsustainable. First, it is slow. A human takes 10 to 30 minutes to diagnose an incident, which means 10 to 30 minutes of downtime. An agent can diagnose in seconds. Second, it is error-prone. A woken-up human at 3 AM is not at their best, which means they might misdiagnose the issue or apply the wrong fix. An agent is consistent, which means it applies the same diagnostic process every time. Third, it is exhausting. Frequent 3 AM wakeups lead to burnout, which affects the human's performance during the day. An agent handles the routine incidents, which means the human is only woken up for novel incidents that require human judgment.
What the Incident Response Pipeline Does
The agentic incident response pipeline performs the following steps when an incident is detected:
- Detect the incident. The pipeline monitors your app via deployxa doctor on a schedule (e.g., every 1 minute). If the readiness grade drops below B, an incident is declared.
- Gather context. The pipeline calls deployxa_get_logs to get the recent logs, deployxa_get_metrics to get the resource usage, and deployxa_doctor to get the detailed check results. This context is fed to the LLM for diagnosis.
- Diagnose the root cause. The LLM analyzes the context and identifies the root cause. For example, "the database connectivity check failed because the database is out of connections, likely due to a connection pool that is too small for the current traffic."
- Apply a fix. Based on the diagnosis, the pipeline applies a fix. For known incident types, the fix is automated (e.g., "increase the connection pool size", "restart the container", "roll back to the previous release"). For novel incidents, the fix requires human intervention.
- Verify the fix. The pipeline calls deployxa doctor again to verify the fix worked. If the grade is back to A or B, the incident is resolved. If not, the pipeline tries the next fix (up to a limit).
- Notify the team. The pipeline sends a Slack notification with the incident summary, the diagnosis, the fix applied, and the verification result. If the fix worked, the notification is informational. If the fix did not work, the notification is an escalation that requests human intervention.
Step-by-Step: Building the Incident Response Pipeline
Here is how to build the incident response pipeline in Python, using the Deployxa MCP server and an LLM.
Step 1: Install dependencies
pip install openai requests schedule
npm install -g @deployxa/mcp-server
deployxa-mcp loginStep 2: Create the incident response script
# incident_response.py
import openai
import requests
import schedule
import time
import json
from datetime import datetime
# Configuration
APP_ID = "123"
APP_NAME = "my-app"
SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/..."
DEPLOYXA_MCP_URL = "http://localhost:3000/mcp"
CHECK_INTERVAL_SECONDS = 60
THRESHOLD_GRADE = "B"
# Known incident types and their fixes
KNOWN_INCIDENTS = {
"database_connection_failed": {
"diagnosis": "Database connection failed. Likely causes: database is down, connection pool is exhausted, or network issue.",
"fix": "restart the container to reset the connection pool",
"action": "restart",
},
"container_crash_loop": {
"diagnosis": "Container is in a crash loop. Likely causes: missing environment variable, invalid configuration, or code error.",
"fix": "roll back to the previous release",
"action": "rollback",
},
"high_error_rate": {
"diagnosis": "High error rate detected. Likely causes: bad deployment, external service failure, or resource exhaustion.",
"fix": "roll back to the previous release",
"action": "rollback",
},
}
def call_deployxa(tool, params):
"""Call a Deployxa MCP tool."""
response = requests.post(
DEPLOYXA_MCP_URL,
json={"tool": tool, "params": params},
)
return response.json()
def send_slack_notification(message, is_escalation=False):
"""Send a Slack notification."""
emoji = "🔴" if is_escalation else "🟢"
requests.post(SLACK_WEBHOOK_URL, json={
"text": f"{emoji} {message}",
})
def diagnose_with_llm(logs, metrics, doctor_report):
"""Use an LLM to diagnose the incident."""
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are an incident response agent. Diagnose the production incident based on the context provided. Return a JSON object with 'incident_type' (from the known types: database_connection_failed, container_crash_loop, high_error_rate, or 'unknown'), 'diagnosis' (a plain-English explanation), and 'recommended_fix' (the recommended action)."},
{"role": "user", "content": f"Logs:\n{logs[:3000]}\n\nMetrics:\n{json.dumps(metrics)}\n\nDoctor report:\n{json.dumps(doctor_report)}"},
],
)
return json.loads(response.choices[0].message.content)
def apply_fix(action):
"""Apply the fix based on the action."""
if action == "restart":
return call_deployxa("deployxa_restart_app", {"app_id": APP_ID})
elif action == "rollback":
return call_deployxa("deployxa_rollback_release", {
"app_id": APP_ID,
"confirmed": True,
})
return None
def handle_incident():
"""Handle a production incident."""
# Gather context
logs = call_deployxa("deployxa_get_logs", {"app_id": APP_ID, "count": 100})
metrics = call_deployxa("deployxa_get_metrics", {"app_id": APP_ID})
doctor = call_deployxa("deployxa_doctor", {"app_id": APP_ID})
# Diagnose
diagnosis = diagnose_with_llm(logs.get("logs", ""), metrics, doctor)
incident_type = diagnosis.get("incident_type", "unknown")
diagnosis_text = diagnosis.get("diagnosis", "Unknown issue")
recommended_fix = diagnosis.get("recommended_fix", "escalate to human")
# Apply fix for known incidents
if incident_type in KNOWN_INCIDENTS:
action = KNOWN_INCIDENTS[incident_type]["action"]
fix_result = apply_fix(action)
# Wait for the fix to take effect
time.sleep(30)
# Verify
new_doctor = call_deployxa("deployxa_doctor", {"app_id": APP_ID})
if new_doctor.get("grade") in ["A", "B"]:
send_slack_notification(
f"Incident resolved for {APP_NAME}.\n"
f"Diagnosis: {diagnosis_text}\n"
f"Fix applied: {recommended_fix}\n"
f"New grade: {new_doctor['grade']}"
)
return
else:
send_slack_notification(
f"Fix did not resolve the incident for {APP_NAME}.\n"
f"Diagnosis: {diagnosis_text}\n"
f"Fix attempted: {recommended_fix}\n"
f"Current grade: {new_doctor.get('grade', 'unknown')}\n"
f"Human intervention needed.",
is_escalation=True,
)
else:
# Unknown incident type, escalate
send_slack_notification(
f"Unknown incident for {APP_NAME}.\n"
f"Diagnosis: {diagnosis_text}\n"
f"Recommended fix: {recommended_fix}\n"
f"Human intervention needed.",
is_escalation=True,
)
def check_app():
"""Check the app and handle incidents."""
doctor = call_deployxa("deployxa_doctor", {"app_id": APP_ID})
grade = doctor.get("grade", "F")
if grade < THRESHOLD_GRADE:
print(f"[{datetime.utcnow().isoformat()}] Incident detected: grade {grade}")
handle_incident()
else:
print(f"[{datetime.utcnow().isoformat()}] OK: grade {grade}")
# Schedule the check
schedule.every(CHECK_INTERVAL_SECONDS).seconds.do(check_app)
# Run immediately, then on schedule
check_app()
while True:
schedule.run_pending()
time.sleep(1)Step 3: Run the incident response pipeline
python incident_response.pyThe pipeline runs immediately, then every 60 seconds. If the readiness grade drops below B, it diagnoses the incident, applies a fix (for known incidents), verifies the fix, and notifies the team via Slack.
Step 4: Deploy the pipeline
The pipeline should run on a server (not your laptop), so it runs 24/7. You can deploy it as a Deployxa app (a Python script that runs in a persistent container). For more on deploying Python apps, see our article on why Python, Streamlit, and Gradio belong on persistent containers.
Common Pitfalls and Troubleshooting
The first pitfall is false positives. The pipeline might declare an incident for a transient issue (e.g., a single failed health check) that resolves itself. The fix is to require multiple consecutive failures before declaring an incident (e.g., 3 failures in a row). The second pitfall is bad fixes. The LLM might recommend a fix that makes things worse (e.g., restarting a container that is already healthy). The fix is to verify the fix before considering it successful, and to roll back if the fix makes things worse. The third pitfall is escalation fatigue. If the pipeline escalates too many incidents, the team will start ignoring the alerts. The fix is to handle as many incidents as possible automatically and to only escalate truly novel incidents. The fourth pitfall is LLM hallucinations. The LLM might hallucinate a diagnosis that is not supported by the evidence, which leads to incorrect fixes. The fix is to ground the LLM in the actual logs and metrics, and to verify the diagnosis against known patterns. The fifth pitfall is cost. The pipeline makes LLM calls on every incident, which costs money. The fix is to use cheaper models for simple diagnoses (e.g., GPT-4o-mini) and to cache diagnoses for common incident types.
Advanced Pipeline Patterns
Beyond the basics, the incident response pipeline can be extended with several advanced patterns. The first is multi-incident handling. If multiple apps have incidents simultaneously, the pipeline should handle them in parallel, with separate diagnoses and fixes. The second is incident correlation. If multiple apps fail at the same time, the pipeline should correlate the incidents (e.g., "all three apps failed because the shared database is down") and apply a single fix (e.g., "restart the database"). The third is post-incident review. After an incident is resolved, the pipeline should generate a post-incident review (incident summary, timeline, root cause, fix applied, lessons learned) and post it to a designated Slack channel. The fourth is learning from incidents. The pipeline should learn from past incidents (e.g., "this incident type was fixed by restarting the container, so try that first next time") to improve its fix success rate. The fifth is integration with on-call systems. For escalations, the pipeline should integrate with on-call systems (e.g., PagerDuty, Opsgenie) to wake up the on-call engineer. For more on advanced patterns, see our articles on building a multi-agent deployment pipeline with LangGraph and auditing your AI agent's cloud actions.
Conclusion: Let the Agent Handle the 3 AM Wakeups
Manual incident response is slow, error-prone, and exhausting. An agentic incident response pipeline handles the routine incidents (diagnose, fix, verify, notify) automatically, and only escalates to a human for novel incidents that require human judgment. With the Deployxa MCP server and an LLM, you can build a pipeline that handles 80 percent of incidents automatically, which means you sleep through the night and only wake up for the incidents that truly need you. Stop waking up at 3 AM and start letting the agent handle it.
Ready to build your incident response pipeline? Install the Deployxa MCP server with npm i -g @deployxa/mcp-server, run deployxa-mcp login, and start building. For more on agentic workflows, see our articles on building an AI agent that monitors your app 24/7 and building a Slack bot that deploys apps. Learn about version controlling your infrastructure with Deployxa MCP and the agentic deployment checklist in our companion articles. Explore our free developer tools to speed up your workflow.