In large organizations across the banking, financial services, and insurance sectors in Vietnam, Incident Management processes are typically standardized to a certain degree. Service desks operate 24/7. SLAs are defined by priority level. On-call rotations and escalation matrices are already in place.
However, the reality of day-to-day operations reveals a different problem: many incidents are not detected early enough. In some cases, the IT team only becomes aware of an issue after being notified by business teams when end users start complaining.
Even when monitoring tools are in place, alerts are spread across multiple platforms: APM, infrastructure monitoring, log management, cloud monitoring, network monitoring, and database monitoring. Each tool sends its own type of alert in a different format, forcing the IT team to manually filter, assess, and decide which alerts are truly critical.
Another major challenge is what happens after an incident is resolved. Many organizations fail to log the resolution process properly. Timelines are incomplete. Root causes are only discussed in chat messages. Lessons learned are never formalized into runbooks or a knowledge base. As a result, when a similar incident occurs again, the IT team essentially has to start from scratch.
This article examines two real-world pain points in Incident Management: slow incident detection and the lack of post-incident documentation. It then proposes how to use Jira Service Management, monitoring tools, Opsgenie, and AI to detect alerts early, group them intelligently, determine which alerts should become incidents, notify the right people quickly, and automate the post-incident documentation process.
The Core Problem: Slow Incident Detection and No Learning Loop After Incidents
In banking and financial organizations, technology systems do not operate as standalone applications. A single fund transfer, a loan approval request, a QR payment, or a mobile banking login session all pass through multiple layers of systems.
A typical business flow may depend on:
Application layer: Mobile Banking, Internet Banking, Corporate Banking, Branch Teller, Contact Center
API layer: Internal APIs for authentication, transaction limit checks, transaction creation, and reconciliation
Database layer: Core banking DB, transaction DB, customer DB, reporting DB
Middleware: API Gateway, message queue, cache layer, integration bus
Infrastructure layer: Servers, Kubernetes clusters, cloud services, network, firewall
External partners: NAPAS, SWIFT, payment partners, eKYC vendors, credit bureaus
Business processes: Fund transfers, bill payments, loan approvals, end-of-day reconciliation and reporting
Because the system has so many interdependent layers, a small anomaly at one technical layer can quickly affect multiple business services. The problem is that the IT team does not always detect that anomaly early enough.
There are three common root causes:
First, alerts are scattered across multiple monitoring tools. One team uses Datadog or Dynatrace to track application performance. Another team uses Zabbix or Prometheus for infrastructure monitoring. Logs sit in ELK or Splunk. Databases have their own dashboards. Network has a separate alerting system. When an incident occurs, each tool may fire multiple alerts, but there is no central place to aggregate, correlate, and assess their severity.
Second, the IT team is overwhelmed by alert noise. Not every alert is an incident. A CPU spike lasting 2 minutes may just be normal fluctuation. But if it is accompanied by rising latency, increasing error rates, and growing transaction failures on a critical service, it could be a sign of a serious incident. Without proper correlation, deduplication, and clear classification rules, engineers have to read each alert individually and make judgment calls based on personal experience.
Third, organizations lack post-incident documentation. After an incident is resolved, much critical information is never properly recorded: when the first alert appeared, who was notified first, how long it took to identify the correct resolver group, what the workaround was, what the root cause was, which runbook was useful, and what needs to be improved. Without a proper post-incident review, the organization cannot learn from previous incidents.
The result is that Incident Management falls into a familiar cycle:
Late detection
Alerts are missed or not prioritized correctly
Incidents are created too late
The wrong team handles the issue initially
Escalation is delayed
Business learns about the outage before IT does
No proper documentation after resolution
Similar incidents recur but still take just as long to resolve
This is not just a tooling problem. It is a problem of how operational data is organized, how alerts are connected to incidents, how priority thresholds are defined, and how each incident is turned into a source of learning for the organization.
BFSI Use Case: One API Timeout, but IT Detected It Late Because Alerts Were Fragmented
Let us walk through a real-world scenario in a bank.
2:32 PM, Tuesday: The API check-transaction-limit starts showing signs of abnormal behavior. Average latency jumps from 80ms to 4,500ms. The timeout rate rises from 0.5% to 35%.
Datadog records a spike in application latency.
Zabbix flags an unusual CPU increase on one node.
The log management tool records numerous timeout errors from the transaction-validation service.
Database monitoring shows the connection pool is approaching its maximum limit.
However, these alerts sit in four different tools. Each alert is sent to a separate channel. Some arrive via email. Some go to Slack. Some are displayed on dashboards that no one is actively watching.
2:40 PM: The on-call engineer receives an email alert about rising latency. But because the system has a history of false positives, the engineer does not immediately create an incident. The alert is treated as a technical warning that needs further observation.
2:50 PM: The Contact Center starts receiving calls from customers: they cannot make fund transfers via Mobile Banking.
2:55 PM: The Mobile Banking team reports a sharp increase in 500 errors on the transfer feature.
3:00 PM: Branch Operations reports that tellers at branches cannot process certain corporate fund transfer requests.
3:10 PM: An incident is officially created in the ITSM system. By this point, the outage has been affecting multiple business channels for nearly 40 minutes.
After investigation, the IT team discovers that the check-transaction-limit API does not serve just one minor feature. It is a mandatory dependency for many transaction flows:
API check-transaction-limit timeout
↓
Domestic transfers on Mobile Banking fail
Interbank transfers fail
Corporate Banking is affected
Branch teller transactions are disrupted
QR Pay for merchants fails
Scheduled auto-transfers fail
↓
Contact Center is overwhelmed with customer calls
↓
End-of-day reconciliation is at risk of discrepancies if not resolved in time
The key takeaway is that monitoring did not lack data. The tools had detected multiple anomalies early on. But because the data was fragmented, alerts were not grouped, there was no correlation mechanism, and there were no clear rules to determine when an alert should become an incident, the IT team still responded late.
If the organization had Jira Service Management connected to its monitoring tools and Opsgenie, the process could have been very different:
2:32 PM: Datadog detects rising latency.
2:33 PM: Zabbix detects abnormal CPU and memory usage.
2:34 PM: The log tool detects multiple timeouts related to the same service.
2:35 PM: JSM/Opsgenie automatically groups alerts that share the same affected service, the same time window, and the same error pattern.
2:36 PM: The system determines this is no longer an isolated alert, but a cluster of alerts that is likely an incident.
2:37 PM: An incident is automatically created in JSM with full context: affected services, alert sources, log samples, abnormal metrics, suggested resolver group, and recommended severity.
2:38 PM: The on-call engineer receives notifications via app push, phone call, email, and Slack.
2:40 PM: If the engineer does not acknowledge, the system automatically escalates to the next person in the on-call schedule.
The value does not lie in having more alerts. The value lies in turning alerts into actionable signals.
Monitoring tells you something looks abnormal. Jira Service Management and Opsgenie help answer:
- Is this serious enough to create an incident?
- Who needs to handle it?
- Which notification channel should be used?
- If no one responds, who should it escalate to?
- Where is all this information recorded so we can learn from it later?
JSM + Monitoring: From Alert to Incident in Seconds
Jira Service Management is not a monitoring tool - but JSM is designed to be the central hub for receiving and acting on all alerts from every monitoring tool, turning them into meaningful actions. The value chain consists of 4 steps:
Step 1: Monitoring Integration - Collect Alerts from All Sources into One Place
JSM (through Opsgenie - built into JSM Cloud) supports 200+ integrations with popular monitoring tools:
| Monitoring Type |
Popular Tools |
Integration Method |
| Infrastructure |
Zabbix, Nagios, PRTG, SolarWinds |
Webhook / API integration |
| APM |
Datadog, Dynatrace, New Relic, AppDynamics |
Native integration |
| Cloud |
AWS CloudWatch, Azure Monitor, GCP Cloud Monitoring |
Native integration |
| Log |
Splunk, ELK Stack, Sumo Logic |
Webhook / API |
| Synthetic |
Catchpoint, ThousandEyes, Uptrends |
Webhook |
| Custom |
Internal monitoring, homegrown tools |
Email integration / REST API |
Result: Instead of engineers having to open 5--7 different monitoring dashboards, all alerts flow into JSM - a single pane of glass. Engineers only need to look at JSM to understand the full picture.
Example integration configuration - Datadog → JSM:
Datadog Webhook → Opsgenie Integration
├── Alert payload mapping:
├── Alert payload mapping:
│ ├── title → alert title
│ ├── tags → affected service (auto-mapped to JSM service)
│ ├── priority → alert priority
│ └── body → alert description + context
└── Auto-enrichment:
├── Link Datadog dashboard URL to alert
└── Attach monitoring graph snapshot
Step 2: Alert Grouping - Turning 50 Alerts into 1 Clear Incident
This is the most important capability and the biggest differentiator compared to using standalone monitoring tools. JSM/Opsgenie groups alerts using multiple criteria:
| Grouping Criteria |
Example |
Result |
| Time-based |
30 alerts appear within the same 5-minute window |
Grouped into 1 alert group, only 1 notification sent |
| Service-based |
10 alerts all related to transaction-service |
Grouped into 1 alert group with the context "transaction-service has an issue" |
| Content-based |
20 alerts with the same message "connection timeout to DB-TXN-01" |
Deduplicated, 1 representative alert kept + count |
| Correlation rule |
API timeout alert + DB CPU high alert + error rate increase → same root cause |
Grouped into 1 potential incident |
Practical example from the use case above:
BEFORE grouping:
47 individual alerts → 47 notifications → 3 engineers working in 3 different directions
AFTER grouping (JSM/Opsgenie):
Alert Group: "Transaction Flow Degradation"
├── 12 alerts: API check-transaction-limit timeout (Datadog)
├── 8 alerts: DB-TXN-01 high CPU (Zabbix)
├── 15 alerts: Mobile Banking 5xx errors (Dynatrace)
├── 7 alerts: API Gateway error rate (CloudWatch)
└── 5 alerts: Synthetic transaction failed (Catchpoint)
→ 1 notification → to the right team → focused response
When a new alert appears that matches an open group, JSM automatically adds it to the existing group instead of creating a new alert - engineers are not interrupted by constant notifications.
Step 3: Deciding Alert vs. Incident - When Does a Warning Become an Outage
Not every alert is an incident. JSM allows organizations to clearly define when an alert group should automatically become an incident that requires a response team:
Alert (warning) - monitor only:
CPU spike lasting less than 2 minutes that self-recovers
Disk usage gradually increasing but not yet at a critical threshold
Latency slightly elevated during peak hours but still within SLA
Single service degraded but a workaround exists
Incident - requires an immediate response team:
Error rate exceeds the allowed threshold for more than 3 minutes
Multiple related services are affected simultaneously (correlation)
A critical business flow is disrupted
The issue does not self-recover within the allowed timeframe
JSM Automation configuration:
RULE: Auto-create Incident from Alert Group
TRIGGER: Alert group created or updated
CONDITIONS (any match):
1. Alert count in group > 10 within 5 minutes
AND affected service tier = 1 or 2
2. Error rate > 5% sustained for > 3 minutes
AND business flow = critical
3. Multiple services affected (>= 3 services)
AND alert priority >= P2
4. Synthetic monitoring failed
AND service = customer-facing
ACTION:
1. Create JSM Incident ticket automatically
2. Link all alerts in the group to the incident
3. Auto-classify severity based on business impact
4. Trigger notification workflow (Step 4)
Example decision scenarios:
| Scenario |
Alert or Incident? |
Reason |
| DB server CPU spikes to 95% for 30 seconds then drops to 60% |
Alert - monitor, no action needed |
Self-recovered, no service impact |
| DB server CPU stays above 90% for 5 minutes + API timeouts increase |
Incident - action required |
Sustained, affecting the API layer |
| 1 node in a cluster goes down but traffic automatically shifts to another node |
Alert - monitor, schedule a fix |
Service is unaffected due to redundancy |
| 2 of 3 nodes in a cluster are down, service is degraded |
Incident - immediate action required |
No remaining redundancy, high risk |
| Log server disk usage reaches 80% |
Alert - schedule a cleanup |
No service impact yet, there is time to address it |
| Database disk usage reaches 95% and rising fast |
Incident - action required |
Risk of database full → all write operations will fail |
Step 4: Fast Notification - The Right Person, at the Right Time, Through the Right Channel
When an alert becomes an incident, the speed of notifying the right person determines everything. JSM/Opsgenie provides multi-channel notification to ensure no one misses it:
| Notification Channel |
When to Use |
Characteristics |
| Push notification (mobile app) |
All alerts/incidents |
Fast, non-intrusive, engineer checks on their own |
| SMS |
P1, P2 incidents |
Works without internet access |
| Phone call (voice call) |
P1 incident not acknowledged after 3 minutes |
Impossible to ignore - the phone keeps ringing |
| Email |
All incidents + stakeholder notification |
Official record, detailed |
| Slack/Microsoft Teams |
War room creation, team notification |
Real-time collaboration |
| Auto-escalation |
No acknowledgment after X minutes |
Automatically escalates to the next level |
Escalation policy - ensuring no incident is ever missed:
ESCALATION POLICY: P1 Critical Incident
Step 1 (0 minutes):
→ Push notification + SMS to primary on-call engineer
→ Slack message to #incident-response channel
Step 2 (3 minutes --- if not acknowledged):
→ Phone call to primary on-call engineer
→ Push notification to backup on-call engineer
Step 3 (5 minutes --- if still not acknowledged):
→ Phone call to backup on-call engineer
→ SMS to Engineering Manager
→ Email to IT Director
Step 4 (10 minutes --- if still not acknowledged):
→ Phone call to Engineering Manager
→ Email + SMS to CIO
→ Auto-create war room on Slack
Step 5 (15 minutes --- if still not acknowledged):
→ Phone call to IT Director
→ SMS blast to entire engineering team
On-call schedule - knowing exactly who to call and when:
JSM/Opsgenie manages on-call rotation automatically:
On-call Schedule: API Platform Team
Rotation: Weekly
Week 1: Engineer A (primary), Engineer B (backup)
Week 2: Engineer C (primary), Engineer A (backup)
Week 3: Engineer B (primary), Engineer C (backup)
Override rules:
- Public holidays: automatically switch to backup
- If primary is on leave: automatically switch to backup
Contact preferences (each engineer configures their own):
- Business hours (8AM-6PM): Push notification → wait 3 min → SMS
- After hours (6PM-8AM): SMS → wait 2 min → Phone call
- Weekend: Phone call immediately for P1
In summary, the 4-step value chain:
Monitoring tools (Datadog, Zabbix, Dynatrace...)
↓ [Integration --- 200+ integrations]
JSM/Opsgenie receives alerts from all sources
↓ [Grouping --- dedup, correlation, time-based]
47 alerts → 1 alert group with clear context
↓ [Classification --- rule-based + AI pattern matching]
Decision: Is this an alert (monitor only) or an incident (action required)?
↓ [Notification --- multi-channel, escalation]
The right person is notified within 1--2 minutes through the most effective channel
The Role of AI (Rovo) in Incident Detection and Post-Incident Learning
Within the Atlassian ecosystem, Rovo - the AI engine - adds an intelligent layer not only for detecting incidents faster, but also for documenting and learning from them after resolution.
4.1. Smart Correlation: Smarter Grouping Beyond Static Rules
Rule-based grouping (by time window or service name) works well for known patterns. But Rovo can identify new correlations that existing rules have not yet captured:
Service A and Service B were never grouped together before, but Rovo notices that over the past 3 months, whenever Service A times out, Service B also times out 5 minutes later → Rovo suggests creating a new correlation rule.
Two alerts with completely different messages, but Rovo recognizes from incident history that they share the same root cause → grouped together.
4.2. Context Enrichment: Adding Context to Alerts
When an incident is automatically created from an alert group, Rovo immediately:
Searches for similar incidents in JSM history (using semantic search, not just keyword matching)
Finds recent change requests on the affected component
Suggests the most relevant runbook from the Confluence knowledge base
Identifies the business processes affected using the Compass service catalog
The engineer opens the ticket and sees the full context right away, instead of having to manually search across 5--7 different tools.
4.3. Noise Reduction: Reducing Alert Fatigue
Rovo learns from engineer feedback: which alerts are commonly dismissed (noise) and which alerts typically lead to real incidents. Over time, Rovo suggests:
Adjusting monitoring thresholds for alerts with high false positive rates
Suppressing known noise patterns during expected peak periods
Highlighting new alert patterns never seen before (potential unknown issues)
4.4. Auto-Generated Post-Incident Review (PIR): Turning Resolution Experience into Organizational Knowledge
This addresses the second major pain point: after an incident is resolved, no one documents what happened.
Why is this pain point critical?
The reality in most BFSI organizations:
A P1 incident occurs. The entire team rushes to resolve it over 2-3 intense hours.
Once resolved, everyone breathes a sigh of relief and returns to their regular work.
No one writes a Post-Incident Review because they are "too busy" or plan to "write it tomorrow" - and tomorrow never comes.
3 months later, the same type of incident occurs again - a new engineer who has joined the team does not know how it was handled before and has to figure everything out from scratch.
The knowledge lives in the heads of 1-2 people. When they take leave or resign, that knowledge is lost.
Measurable consequences:
| Consequence |
How It Manifests |
| Recurring incidents |
The same root cause occurs 2–3 times because no one documented preventive actions |
| MTTR does not improve over time |
New engineers encounter a similar incident but have no reference material → resolution takes just as long as the first time |
| Knowledge loss from staff turnover |
A senior engineer resigns → all incident resolution experience is lost |
| No data for system improvement |
Leadership asks "How often do incidents recur? What are the main causes?" → there is no data to answer |
| Audit and compliance gaps |
The BFSI sector requires incident records to be retained - no PIR means compliance risk |
Solution: Rovo automatically generates a PIR draft
Instead of requiring engineers to write a PIR from scratch (which 80% of the time does not happen), Rovo automatically generates a PIR draft as soon as the incident is resolved:
TRIGGER: Incident status changes to "Resolved"
ROVO AUTOMATICALLY:
Collects data from the incident ticket:
- Timeline: when the first alert fired, when acknowledged, when resolved
- Who participated in resolution (from assignee, watchers, comments)
- Actions taken (from comment history, status changes)
Collects data from related sources:
- Slack/Teams war room messages (if applicable)
- Change requests related to the incident timeframe
- Alert data from Opsgenie
- Deployment logs from Bitbucket/CI-CD
Generates a PIR draft on Confluence with this structure:
├── Incident Summary (auto-generated)
├── Timeline of Events (auto-generated from system data)
├── Impact Assessment (auto-generated from service tier + duration)
├── Root Cause Analysis (draft --- requires engineer review)
├── What Went Well (draft from response time data)
├── What Could Be Improved (draft from timeline gaps)
├── Action Items (requires engineer input)
└── Lessons Learned (requires engineer input)
Sends a notification to the incident owner:
"A PIR draft has been automatically generated.
You only need to review and add Root Cause + Action Items.
Estimated time: 15 minutes instead of 1--2 hours writing from scratch."
Why this approach works:
| Traditional Approach |
New Approach with Rovo |
| Engineer has to write a PIR from a blank page → takes 1–2 hours → "I'll do it tomorrow" → never written |
Rovo creates an 80% draft → engineer reviews and adds the remaining 20% → 15 minutes → it actually gets written |
| PIR written 1–2 weeks later, details are forgotten |
PIR draft created immediately upon resolution, data is accurate from system logs |
| Everyone writes PIRs in different formats |
Standardized template, easy to search and compare across incidents |
| PIRs are scattered everywhere, no one can find them |
PIR is automatically linked to the incident ticket + stored in a structured Confluence knowledge base |
4.5. Knowledge Loop: Turning PIRs into a Weapon for the Next Incident
PIRs are not just for archival purposes - Rovo creates a continuous learning loop:
Incident occurs
↓
Rovo searches for PIRs from similar past incidents
↓ (context enrichment --- see section 4.2)
Engineer immediately sees: "Last time with the same symptoms, the root cause was X, fixed by doing Y"
↓
Faster resolution (MTTR decreases)
↓
Resolved → Rovo generates a new PIR
↓
New PIR is added to the knowledge base
↓
The next incident benefits from this PIR
↓ (cycle repeats)
Real-world example:
January: P1 Incident - Mobile Banking timeout. The team spends 3 hours finding the root cause (connection pool exhaustion on the database). Rovo creates a PIR documenting the fix in detail.
April: A similar incident occurs. Rovo immediately surfaces the January PIR: "A similar incident occurred on January 15. Root cause: connection pool exhaustion. Fix: restart connection pool + increase max_connections. Runbook: [link]." A new engineer (who has never encountered this issue before) resolves it in 30 minutes instead of 3 hours.
Practical Implementation Roadmap by Phase
Implementing an early incident detection system and post-incident documentation process is not a "buy a tool and start using it" project - it requires a clear roadmap. This section presents an 8-week implementation plan across 4 phases on the Atlassian platform.
Tools used throughout:
| Tool |
Role |
| Jira Service Management (JSM) |
Incident workflow, alert management, SLA tracking, automation rules |
| Opsgenie (built into JSM) |
Alert routing, on-call management, escalation, multi-channel notification |
| Compass |
Service catalog, dependency tracking, component ownership |
| Confluence |
Knowledge base, runbooks, Post-Incident Review repository |
| Rovo |
AI - smart correlation, context enrichment, noise reduction, auto-generated PIR |
Phase 1: Foundation - Connecting Monitoring to JSM (Weeks 1--2)
Objective: All alerts from every monitoring tool flow into JSM/Opsgenie. Basic on-call schedule is set up.
Phase 1 Checklist
List all monitoring tools currently in use
Configure integration: each monitoring tool → Opsgenie → JSM
Test alert flow: trigger a test alert from each tool, verify it is received in JSM
Set up on-call schedules for each team in Opsgenie
Configure contact preferences for each engineer (notification channels by time of day)
Set up a basic escalation policy (3 steps)
Run in parallel for 1 week: existing monitoring + JSM - verify no alerts are missed
Implementation Details
Week 1: Connecting monitoring
The most common mistake: trying to set up the entire complex system from day one. The first step is simply to ensure all alerts reach JSM - no grouping needed yet, no auto-incident creation yet.
For each monitoring tool:
Create an integration in Opsgenie (most popular tools have native integrations available)
Configure alert payload mapping: title, description, priority, tags
Test with real alerts: trigger an alert on the monitoring tool → verify JSM receives it
Configure basic deduplication: alerts from the same source with the same message → merge into one
Week 2: On-call & Escalation
Set up on-call rotation for each team:
Team: API Platform
Rotation: Weekly, starts Monday 9:00 AM
Members: 4 engineers
Primary + Backup
Team: Database
Rotation: Weekly
Members: 3 engineers
Team: Infrastructure
Rotation: Bi-weekly
Members: 5 engineers
Set up escalation policies by severity:
| Severity |
Step 1 (0 min) |
Step 2 (3 min) |
Step 3 (5 min) |
Step 4 (10 min) |
| P1 |
Push + SMS to on-call |
Phone call to on-call |
SMS to Manager |
Phone call to Director |
| P2 |
Push to on-call |
SMS to on-call |
Push to Manager |
SMS to Manager |
| P3 |
Push to on-call |
SMS after 15 min |
— |
— |
| P4 |
Email to on-call |
— |
— |
— |
Phase 2: Alert Grouping & Noise Reduction (Weeks 3--4)
Objective: Implement intelligent alert grouping. Reduce noise. Engineers only receive notifications that matter.
Phase 2 Checklist
Analyze 2 weeks of alert data: how many alerts per day, how many are noise
Configure time-based grouping: alerts within the same 5-minute window → 1 group
Configure service-based grouping: alerts for the same service → 1 group
Configure content-based deduplication: alerts with the same message → count instead of duplicating
Create correlation rules for known patterns
Configure suppression rules for known noise patterns
Measure results: how much did notifications per day decrease (in %)?
Verify: no real alerts are missed due to suppression
Implementation Details
Week 3: Grouping rules
Analyze alert data from the first 2 weeks:
Before grouping:
Total alerts/day: ~800
Notifications/engineer/day: ~120
→ Alert fatigue: HIGH
After grouping:
Alert groups/day: ~50-80
Notifications/engineer/day: ~10-15
→ Manageable, every notification is meaningful
Configure 3 layers of grouping in Opsgenie:
| Layer |
Rule |
Example |
| Deduplication |
Alerts from the same source with the same message within 5 minutes → 1 alert with count |
20 alerts "DB CPU high" → 1 alert, count=20 |
| Time-based |
Alerts from any source within the same 5-minute window → 1 group |
15 different alerts from 2:30–2:35 PM → 1 group |
| Correlation |
Alerts matching a defined pattern → 1 group |
API timeout + DB high CPU + Error rate increase → 1 group "Database Performance" |
Week 4: Noise suppression
Identify and suppress noise patterns:
Suppression rules:
1. CPU spike < 2 minutes then self-recovers → Suppress, log only
2. Known scheduled job causing spikes daily at 2--3 AM → Suppress during window
3. Alert that auto-resolves within 1 minute → Suppress notification, keep log
4. Duplicate alerts from redundant monitoring → Keep 1 source, suppress others
Important: All suppression rules must be reviewed weekly. Suppressed alerts are still logged - if patterns change (e.g., a spike that used to last 30 seconds now lasts 5 minutes), the rules need to be updated.
Phase 3: Auto Alert → Incident + AI Context + PIR Foundation (Weeks 5--6)
Objective: Automatically determine when an alert group should become an incident. Rovo automatically adds context to incidents. Establish the foundation for automated Post-Incident Reviews.
Phase 3 Checklist
Define alert → incident criteria for each service tier
Configure JSM Automation: auto-create incident from alert group
Configure auto-severity classification
Set up Compass service catalog (service → owner → tier)
Configure Rovo Action: auto-generate Incident Brief
Configure Rovo Action: find similar incidents
Configure Rovo Action: suggest runbook
Create a standardized PIR template on Confluence
Create a Confluence space "Incident Knowledge Base" with a clear structure
Configure JSM Automation: when an incident is resolved → automatically create a PIR draft on Confluence
Configure Rovo Action: auto-populate the PIR draft with data from the incident
Pilot for 2 weeks with 1--2 teams
Collect feedback, adjust rules
Implementation Details
Week 5: Auto-create incident + Severity classification
JSM Automation Rules:
Rule 1: Auto-create Incident
TRIGGER: Alert group matches incident criteria
CONDITIONS (any):
- Alert count > 10 AND service tier ≤ 2
- Error rate > 5% AND duration > 3 min
- Multiple services (≥ 3) affected simultaneously
- Customer-facing synthetic monitoring failed
ACTION:
1. Create JSM Incident
2. Auto-classify severity (Rule 2)
3. Link alert group → incident
4. Trigger Rovo: Generate Incident Brief
5. Trigger notification workflow
Rule 2: Auto-classify Severity
S1 - Critical:
Service tier = 1 AND (down OR error rate > 20%)
→ Phone call immediately + War room + Notify CIO
S2 - Major:
Service tier = 1 AND degraded
OR Service tier = 2 AND down
→ SMS + Slack channel + Notify Manager
S3 - Minor:
Service tier = 2 AND degraded
OR Service tier = 3 AND down
→ Push notification + Email
S4 - Low:
Service tier = 3 AND degraded
OR has workaround
→ Email only
Week 6: Rovo AI integration + PIR Foundation + Pilot
Configure Rovo Actions - now covering both early detection and post-incident documentation:
Rovo Actions for early detection (when an incident is created):
| Rovo Action |
Input |
Output |
| Generate Incident Brief |
Alert group data + Compass service info |
Summary: affected services, business impact, suggested resolver, runbook link |
| Find Similar Incidents |
Incident summary |
Top 5 similar incidents from JSM history, including resolution details and links to their PIRs |
| Suggest Runbook |
Affected service + error pattern |
Most relevant runbook from Confluence KB |
Rovo Actions for post-incident documentation (when an incident is resolved):
| Rovo Action |
Input |
Output |
| Auto-generate PIR Draft |
Incident ticket data + comment history + alert timeline + Slack war room messages |
Confluence page with the PIR template 80% pre-filled automatically |
| Extract Action Items |
PIR content + root cause analysis |
List of action items as Jira tickets, linked to the PIR |
| Link PIR to Knowledge Base |
PIR page + affected services |
PIR is automatically tagged, categorized, and linked into the Confluence KB by service/component |
Setting up the Confluence "Incident Knowledge Base":
Confluence Space: "Incident Knowledge Base"
├── 📁 By Service
│ ├── API Platform
│ │ ├── PIR-2026-001: Transaction timeout (Jan 15, 2026)
│ │ ├── PIR-2026-005: Connection pool exhaustion (Mar 12, 2026)
│ │ └── ...
│ ├── Database
│ ├── Mobile Banking
│ └── ...
├── 📁 By Root Cause Category
│ ├── Capacity / Performance
│ ├── Configuration Change
│ ├── Third-party Dependency
│ ├── Code Bug
│ └── Infrastructure Failure
├── 📁 By Quarter
│ ├── Q1-2026
│ ├── Q2-2026
│ └── ...
└── 📄 Incident Trends Dashboard (Rovo auto-updates monthly)
├── Top recurring root causes
├── MTTR trend by service
├── Action items completion rate
└── Services with most incidents
JSM Automation Rule: Auto-create PIR
TRIGGER: Incident status changes to "Resolved"
ACTION:
1. Rovo collects data:
- Incident timeline (created → acknowledged → investigating → resolved)
- All comments on the ticket (in chronological order)
- Alert group data (alert count, source, affected services)
- Participants (assignee, watchers, commenters)
- Change requests within ±24 hours on affected services
- Slack/Teams war room transcript (if applicable)
2. Rovo generates PIR draft on Confluence:
┌─────────────────────────────────────────────┐
│ POST-INCIDENT REVIEW │
│ Incident: [INC-XXX] --- [Title] │
│ Date: [Auto] │ Severity: [Auto] │
│ Duration: [Auto-calculated] │
├─────────────────────────────────────────────┤
│ 📋 SUMMARY (auto-generated) │
│ [Rovo summarizes the incident in 3-5 │
│ sentences] │
├─────────────────────────────────────────────┤
│ ⏱ TIMELINE (auto-generated from system │
│ data) │
│ 2:30 PM --- First alert: API timeout │
│ 2:31 PM --- Alert group created (47 alerts) │
│ 2:32 PM --- Incident auto-created, severity │
│ S1 │
│ 2:33 PM --- On-call acknowledged │
│ 2:35 PM --- War room opened │
│ 2:50 PM --- Root cause identified │
│ 3:10 PM --- Fix deployed │
│ 3:15 PM --- Service recovered │
│ 3:30 PM --- Incident resolved │
├─────────────────────────────────────────────┤
│ 💥 IMPACT (auto-generated) │
│ - Services affected: [from Compass] │
│ - Duration: [auto-calculated] │
│ - Users affected: [from monitoring data] │
│ - Business processes impacted: │
│ [from Compass] │
├─────────────────────────────────────────────┤
│ 🔍 ROOT CAUSE (⚠️ REQUIRES ENGINEER │
│ REVIEW) │
│ [Rovo draft based on comments + actions] │
│ → Engineer confirms or edits │
├─────────────────────────────────────────────┤
│ ✅ WHAT WENT WELL (auto-generated) │
│ - MTTI: 2 minutes (target: <3 min) │
│ - Acknowledge: 1 minute │
│ - Escalation: not needed (resolved quickly)│
├─────────────────────────────────────────────┤
│ ⚠️ WHAT COULD BE IMPROVED (draft) │
│ [Rovo suggests based on timeline gaps] │
│ → Engineer reviews and adds │
├─────────────────────────────────────────────┤
│ 📌 ACTION ITEMS (⚠️ REQUIRES ENGINEER │
│ INPUT) │
│ [ ] Preventive action 1: ___ │
│ [ ] Preventive action 2: ___ │
│ [ ] Monitoring improvement: ___ │
│ → Each action item creates a Jira ticket │
├─────────────────────────────────────────────┤
│ 📚 LESSONS LEARNED (⚠️ REQUIRES ENGINEER │
│ INPUT) │
│ [Rovo suggests guiding questions:] │
│ - Could this incident have been prevented? │
│ - Is the current monitoring sufficient? │
│ - Does the runbook need updating? │
└─────────────────────────────────────────────┘
3. Link PIR page ↔ JSM Incident ticket (bidirectional)
4. Auto-categorize PIR:
- Tag by service, root cause category, severity
- Place in the correct folder in the Confluence KB
5. Notify incident owner:
"A PIR draft is ready - you only need to review Root Cause,
fill in Action Items and Lessons Learned (~15 minutes).
Link: [Confluence page URL]
Deadline: 48 hours after resolution."
6. Auto-reminder if PIR has not been reviewed:
- 24 hours later: Reminder to incident owner
- 48 hours later: Notify Engineering Manager
- 72 hours later: Escalate to IT Director
Pilot with 1--2 teams for 2 weeks:
Run in parallel: existing system + JSM auto-detection + auto-PIR
Compare: MTTI with the new system vs. MTTI with the old system
Measure: PIR completion rate (what percentage of incidents have a PIR?)
Collect daily feedback via a Confluence "Pilot Feedback" page
Phase 4: Optimize, Knowledge Loop & Scale (Weeks 7--8+)
Objective: Fine-tune based on real data. Build a continuous learning loop. Expand coverage. Set up KPI dashboards.
Phase 4 Checklist
Review false positive rate: how many auto-created incidents are actually incidents?
Review false negative rate: were any incidents missed?
Adjust grouping rules and incident criteria based on data
Create a JSM Dashboard: "Incident Detection Analytics"
Review PIR completion rate: what percentage of incidents have a PIR?
Review PIR quality: are PIRs detailed enough? Are action items being followed up?
Configure Rovo: auto-generate a monthly Incident Trends Report from PIR data
Set up quarterly PIR review meetings - extract lessons at the organizational level
Configure Rovo Action: when a new incident is created → auto-search related PIRs → attach to the incident
Expand coverage: add more monitoring sources, add more services to Compass
Train all on-call engineers
Set up a quarterly review process
JSM Dashboard - "Incident Detection & Learning Analytics":
| # |
Widget |
Purpose |
|
Incident Detection |
|
| 1 |
MTTI trend (weekly/monthly) |
Is the time to detect incidents decreasing? |
| 2 |
Alert → Incident conversion rate |
What percentage of alert groups become real incidents? |
| 3 |
False positive rate |
How many auto-created incidents are closed immediately because they are not actual incidents? |
| 4 |
Alert noise reduction % |
What percentage of alerts are suppressed/deduped compared to total? |
| 5 |
Notification → Acknowledge time |
How quickly do engineers acknowledge? |
| 6 |
Escalation rate |
What percentage of incidents require escalation because the primary on-call did not respond? |
|
Top 5 noisy alert sources |
Which monitoring tool/rule generates the most noise? |
| 7 |
Channel effectiveness |
Which channel (push/SMS/call) has the highest acknowledge rate? |
|
Documentation & Learning |
|
| 8 |
PIR completion rate |
What percentage of incidents (P1, P2) have a PIR? Target: 100% |
| 9 |
PIR completion time |
Average time from resolution to PIR finalization? Target: <48h |
| 10 |
Action items completion rate |
What percentage of action items from PIRs are closed? |
| 11 |
Action items overdue |
How many action items are past due? |
| 12 |
Recurring incidents |
How many incidents with the same root cause category are recurring? |
| 13 |
Knowledge base coverage |
What percentage of services have at least 1 PIR + runbook? |
| 14 |
PIR-assisted resolution |
What percentage of incidents were resolved with reference to a previous PIR? |
Monthly Incident Trends Report (auto-generated by Rovo):
Each month, Rovo automatically generates a report on Confluence by analyzing all PIRs from that month:
📊 INCIDENT TRENDS REPORT --- May 2026
OVERVIEW:
- Total incidents: 23 (↓15% vs. April)
- P1: 2 | P2: 5 | P3: 11 | P4: 5
- Average MTTI: 2.3 minutes (target: <3 minutes) ✅
- Average MTTR: 45 minutes (↓20% vs. April) ✅
- PIR completion: 100% P1-P2, 82% P3 (target: 100% P1-P2) ✅
TOP ROOT CAUSE CATEGORIES:
1. Configuration Change (8 incidents) --- ⚠️ up 60%
→ Suggestion: Review change management process
2. Capacity/Performance (6 incidents) --- stable
3. Third-party Dependency (4 incidents) --- stable
4. Code Bug (3 incidents) --- ↓ 40%
5. Infrastructure (2 incidents) --- stable
RECURRING INCIDENTS:
- "Connection pool exhaustion" --- 3rd occurrence in 6 months
→ Action item from PIR-2026-005 has not been implemented!
→ ⚠️ ESCALATE: Needs to be prioritized
SERVICES REQUIRING ATTENTION:
- API Gateway: 5 incidents this month --- highest count
- Payment Service: 2 P1 incidents --- architectural review needed
ACTION ITEMS STATUS:
- Total from this month's PIRs: 34
- Completed: 21 (62%)
- In progress: 8 (23%)
- Overdue: 5 (15%) --- ⚠️ follow-up required
8-Week Timeline Summary
| Week |
Phase |
Focus |
Key Deliverable |
| 1 |
Foundation |
Connect monitoring → JSM |
All alerts reach JSM |
| 2 |
Foundation |
On-call + Escalation |
The right person is called at the right time |
| 3 |
Grouping |
Alert grouping rules |
Notifications reduced by 80–90% |
| 4 |
Grouping |
Noise suppression |
Engineers only see alerts that matter |
| 5 |
Auto-detection |
Alert → Incident automation |
Incidents created in 1–2 minutes instead of 30–45 minutes |
| 6 |
Auto-detection + PIR |
AI context + Auto-PIR + Pilot |
Engineers have full context during resolution + PIR auto-generated after resolution |
| 7 |
Optimize |
Fine-tune + Dashboard + Knowledge loop |
Data-driven improvement + PIRs feeding back into detection |
| 8+ |
Scale |
Expand coverage |
All services covered + Knowledge base continuously growing |
Common Mistakes to Avoid
| # |
Mistake |
Why It Causes Problems |
|
Incident Detection |
|
| 1 |
Connecting monitoring but not grouping alerts |
Alerts flow into JSM but there are still 500 alerts/day → alert fatigue just moves from monitoring to JSM |
| 2 |
Suppressing too aggressively |
Noise is reduced but real alerts are missed → trust in the system is lost |
| 3 |
Auto-creating incidents without clear criteria |
Too many false positives → engineers ignore auto-created incidents → back to alert fatigue |
| 4 |
Not setting up escalation |
An incident is created but the on-call engineer is not online → no one handles it → worse than having no system at all |
| 5 |
Using only email for notifications |
P1 incident at 2 AM → email sits in the inbox → engineer is asleep and unaware → late detection |
| 6 |
Not reviewing suppression rules |
Patterns change over time but rules are not updated → new incidents are missed |
|
Documentation & Learning |
|
| 7 |
Skipping Post-Incident Reviews |
The same type of incident occurs a 2nd and 3rd time, but no one learned anything from the previous occurrence |
| 8 |
Writing PIRs without action items |
The PIR becomes a "story" - it describes the incident but no one takes action to prevent it from recurring |
| 9 |
Having action items but not tracking them |
Action items are written in the PIR and then forgotten - no one follows up, no one is accountable, preventive fixes are never implemented |
| 10 |
PIRs are scattered and unorganized |
An engineer encounters a similar incident but cannot find the old PIR because it is buried in email, a Google Doc, or a random Confluence page |
| 11 |
Only writing PIRs for P1, ignoring P2/P3 |
Many important lessons come from P2/P3 incidents - and a poorly handled P3 can become a P1 next time |
| 12 |
Blame culture in PIRs |
PIRs turn into "finding who is at fault" → the team is afraid to write honestly → PIRs lose their value. PIRs must be blameless - focused on systems and processes, not individuals |
KPIs to Measure
| # |
KPI |
Baseline (Before Implementation) |
Target (After 3 Months) |
How to Measure |
| 1 |
MTTI (Mean Time to Identify) |
15–45 minutes |
< 3 minutes |
From when the incident actually occurs to when it is created in JSM |
| 2 |
Alert noise reduction |
0% (no filtering) |
85% of alert noise eliminated |
% of alerts deduped/suppressed/grouped vs. total |
| 3 |
Auto-detection rate |
0% (manual detection) |
80% of incidents auto-created |
% of incidents created by automation vs. manual |
| 4 |
False positive rate |
N/A |
< 10% |
% of auto-created incidents closed immediately because they are not real incidents |
| 5 |
Acknowledge time |
10–30 minutes |
< 3 minutes |
From notification to acknowledgment |
| 6 |
Escalation rate |
N/A |
< 15% |
% of incidents requiring escalation because the primary on-call did not respond |
| 7 |
MTTR (Mean Time to Resolve) |
Measure baseline |
Reduce by 30–50% |
From incident created to resolved |
| 8 |
Business-first detection rate |
50% (business detects before IT) |
< 10% |
% of incidents where business discovers the issue before IT |
| 9 |
PIR completion rate |
< 20% (most are never written) |
100% for P1-P2, > 80% for P3 |
% of incidents with a completed PIR |
| 10 |
PIR completion time |
1–2 weeks (if written at all) |
< 48 hours after resolution |
Time from resolution to PIR finalization |
| 11 |
Action items creation rate |
Near 0 (no PIR → no action items) |
90% of PIRs have at least 2 action items |
% of PIRs with specific action items |
| 12 |
Action items completion rate |
N/A |
80% completed within 30 days |
% of action items closed on time |
| 13 |
Recurring incident rate |
Not measurable (no data) |
Reduce by 50% after 6 months |
Number of incidents with the same root cause category recurring |
| 14 |
PIR-assisted resolution rate |
0% (no PIRs to reference) |
40% |
% of incidents where engineers referenced a previous PIR during resolution |
| 15 |
Knowledge base coverage |
0% |
70% of Tier 1-2 services have a PIR + runbook |
% of services with at least 1 PIR and 1 runbook |
The two most important KPIs:
#8. Business-first detection rate: If this number drops from >50% to <10%, it means the IT team has shifted from reactive to proactive.
#13. Recurring incident rate: If this number decreases by 50% after 6 months, it means the learning loop is working - the team is not just fixing incidents, but actually preventing them from happening again.
How to present this to leadership:
Before: "The system was down for 40 minutes before business reported it to IT. After resolution, no one documented anything. 3 months later, the same issue happened again - engineers had to start from scratch."
After: "The system showed anomalies for 2 minutes, IT already knew and was working on it. After resolution, a PIR was automatically generated. 3 months later, similar symptoms appeared - the engineer opened the ticket and immediately found the previous PIR, resolving the issue in 30 minutes instead of 3 hours. And this type of incident has decreased by 50% thanks to preventive actions from the PIR."
Conclusion
In many discussions about Incident Management, the focus is usually on resolving incidents faster - reducing MTTR, finding root causes sooner, rolling back quicker. But there is a simple truth:
You cannot resolve an incident quickly if you do not even know it is happening.
MTTI (Mean Time to Identify) is the "invisible" window - the incident has already occurred, customers are already affected, but the IT team does not know yet. Every minute in this window is a minute where the SLA is being consumed while no one is working on a fix.
JSM integrated with monitoring does not replace existing monitoring tools - it adds an intelligent layer between monitoring and people:
Collect alerts from all sources into one place
Group them to reduce noise and increase signal
Classify them to know what needs immediate action and what just needs to be noted
Notify the right person, at the right time, through a channel strong enough that it cannot be missed
And after every incident, the system does not just move on. It documents what happened, builds a knowledge base, and ensures the organization learns and improves -so the same incident does not take the same toll twice.
The core question is not "Do we have enough monitoring tools?" - it is: "When a monitoring tool fires an alert, how long does it take for the right person to know and start working on it?"
If the answer is "30--45 minutes," that is 30--45 minutes where customers are silently suffering.
Start by connecting your existing monitoring tools to JSM. In 8 weeks, you will have a working system. Then continuously fine-tune it based on real data.
See also: How AI Assesses Risk for Change Requests – Cutting CAB Preparation Time by 70%
