> ## Documentation Index
> Fetch the complete documentation index at: https://docs.upmetr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Alert Rules

> Configure custom alert rules with thresholds, cooldowns, and per-agent overrides

Alert rules define when Upmetr should create incidents and send notifications. You can set thresholds for infrastructure metrics, uptime monitors, SSL certificates, and cost budgets.

## Creating an Alert Rule

1. Navigate to **Alerts**
2. Click **Add Rule**
3. Configure the rule settings
4. Click **Save**

### Rule Settings

| Field         | Description                                          |
| ------------- | ---------------------------------------------------- |
| **Name**      | Descriptive name (e.g., "High CPU on prod servers")  |
| **Metric**    | The metric to evaluate (CPU, memory, disk, etc.)     |
| **Condition** | Threshold operator: above, below, equals             |
| **Threshold** | The value that triggers the alert                    |
| **Severity**  | Info, Warning, Error, or Critical                    |
| **Cooldown**  | Minimum time between repeated alerts (prevents spam) |

## Trigger Types

| Trigger               | Description                             |
| --------------------- | --------------------------------------- |
| **Monitor Down**      | Uptime monitor detected failure         |
| **Monitor Up**        | Uptime monitor recovered                |
| **Monitor Degraded**  | Slow response detected                  |
| **SSL Expiring**      | Certificate expiring within threshold   |
| **Budget Threshold**  | Budget limit approaching                |
| **Budget Exceeded**   | Budget limit exceeded                   |
| **Infra Metric**      | Host/container metric crossed threshold |
| **CloudWatch Metric** | AWS managed service metric alert        |

## Infrastructure Thresholds

For infrastructure agents, you can set metric-based thresholds:

### Common Threshold Examples

| Metric         | Condition | Threshold | Severity |
| -------------- | --------- | --------- | -------- |
| CPU Usage      | Above     | 90%       | Critical |
| CPU Usage      | Above     | 75%       | Warning  |
| Memory Usage   | Above     | 85%       | Warning  |
| Disk Usage     | Above     | 90%       | Critical |
| Network Errors | Above     | 100/min   | Warning  |

### Anti-Flapping

Alert rules use **breach thresholds** and **recovery thresholds** to prevent flapping:

<Steps>
  <Step title="Breach threshold">
    The metric must exceed the threshold for **N consecutive evaluations** before an incident is created. This prevents one-off spikes from triggering alerts.
  </Step>

  <Step title="Recovery threshold">
    The metric must stay below the threshold for **N consecutive evaluations** before the incident is auto-resolved. Default: 2 for host metrics, 3 for CloudWatch.
  </Step>
</Steps>

<Note>
  Evaluation frequency is every **2 minutes** for infrastructure metrics (via the `evaluate_infra_alerts` Celery task).
</Note>

## Per-Agent Overrides

If a specific server needs different thresholds (e.g., a database server that normally runs at 80% memory), you can create **per-agent overrides**:

1. Open the alert rule
2. Click **Add Override**
3. Select the agent
4. Set the custom threshold
5. Click **Save**

The override applies only to that agent — all other agents use the default threshold.

## Cooldown Period

The cooldown prevents notification spam:

* After an alert fires, the rule enters a cooldown period
* During cooldown, the same rule won't fire again even if conditions persist
* Use shorter cooldowns (5-15 min) for critical alerts
* Use longer cooldowns (30-60 min) for informational alerts

## Notification Routing

Alert rules work with **notification rules** to determine where alerts are sent:

1. **Alert Rule** triggers an incident
2. **Notification Rule** matches the trigger type and severity
3. **Notification Channel** delivers the message (Slack, email, SMS, etc.)

Configure notification rules at **Settings > Integrations**.

<Info>
  See [Notifications Guide](/guides/notifications) for channel configuration details.
</Info>

## Enabling/Disabling Rules

Toggle any rule on or off from the alert rules list. Disabled rules stop evaluating — no incidents will be created.

## CloudWatch Alert Thresholds

For AWS managed services monitored via CloudWatch, Upmetr includes **8 pre-configured critical rules**:

| Service    | Metric               | Threshold    |
| ---------- | -------------------- | ------------ |
| RDS        | CPU Utilization      | > 90%        |
| RDS        | Free Storage         | \< 1 GB      |
| RDS        | Database Connections | > 90% of max |
| ALB        | 5XX Error Rate       | > 5%         |
| ALB        | Target Response Time | > 5s         |
| CloudFront | 5XX Error Rate       | > 5%         |
| EC2        | Status Check Failed  | > 0          |
| EC2        | CPU Credit Balance   | \< 10        |

These are created automatically when you add a cloud account with CloudWatch access.

## Troubleshooting

| Issue                               | Solution                                                                                                        |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Rule not firing**                 | Check that the rule is enabled. Verify the metric is being collected. Check cooldown hasn't suppressed it.      |
| **Too many alerts**                 | Increase the breach threshold or cooldown period. Consider per-agent overrides for noisy servers.               |
| **Alert fires but no notification** | Check notification rules — trigger type and severity must match. Verify the notification channel is configured. |
| **Override not working**            | Ensure the override targets the correct agent ID. Check the override threshold is different from the default.   |