Skip to content

Commit 9d376e0

Browse files
authored
Merge pull request #924 from flashcatcloud/feat/alert-batch21-test
docs: sync Kener, Coroot, Dkron, Duplicati, Ofelia, Trigger.dev, Robotalp alert pages to test
2 parents 0a9c741 + feec1a6 commit 9d376e0

18 files changed

Lines changed: 1699 additions & 2 deletions

File tree

‎docs.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1705,6 +1705,9 @@
17051705
"zh/on-call/integration/alert-integration/alert-sources/aikido",
17061706
"zh/on-call/integration/alert-integration/alert-sources/cato",
17071707
"zh/on-call/integration/alert-integration/alert-sources/rollbar",
1708+
"zh/on-call/integration/alert-integration/alert-sources/trigger-dev",
1709+
"zh/on-call/integration/alert-integration/alert-sources/duplicati",
1710+
"zh/on-call/integration/alert-integration/alert-sources/robotalp",
17081711
"zh/on-call/integration/alert-integration/alert-sources/fivetran",
17091712
"zh/on-call/integration/alert-integration/alert-sources/coralogix",
17101713
"zh/on-call/integration/alert-integration/alert-sources/uptimeobserver",
@@ -1714,6 +1717,7 @@
17141717
"zh/on-call/integration/alert-integration/alert-sources/honeycomb",
17151718
"zh/on-call/integration/alert-integration/alert-sources/mongodb-atlas",
17161719
"zh/on-call/integration/alert-integration/alert-sources/honeybadger",
1720+
"zh/on-call/integration/alert-integration/alert-sources/ofelia",
17171721
"zh/on-call/integration/alert-integration/alert-sources/openobserve",
17181722
"zh/on-call/integration/alert-integration/alert-sources/postman",
17191723
"zh/on-call/integration/alert-integration/alert-sources/contrast",
@@ -1730,9 +1734,11 @@
17301734
"zh/on-call/integration/alert-integration/alert-sources/panther",
17311735
"zh/on-call/integration/alert-integration/alert-sources/axiom",
17321736
"zh/on-call/integration/alert-integration/alert-sources/checkmk",
1737+
"zh/on-call/integration/alert-integration/alert-sources/dkron",
17331738
"zh/on-call/integration/alert-integration/alert-sources/auvik",
17341739
"zh/on-call/integration/alert-integration/alert-sources/domotz",
17351740
"zh/on-call/integration/alert-integration/alert-sources/ntopng",
1741+
"zh/on-call/integration/alert-integration/alert-sources/kener",
17361742
"zh/on-call/integration/alert-integration/alert-sources/ohdear",
17371743
"zh/on-call/integration/alert-integration/alert-sources/dbmarlin",
17381744
"zh/on-call/integration/alert-integration/alert-sources/cloudamqp",
@@ -1864,6 +1870,7 @@
18641870
"zh/on-call/integration/alert-integration/alert-sources/monitive",
18651871
"zh/on-call/integration/alert-integration/alert-sources/hosted-graphite",
18661872
"zh/on-call/integration/alert-integration/alert-sources/netbeez",
1873+
"zh/on-call/integration/alert-integration/alert-sources/coroot",
18671874
"zh/on-call/integration/alert-integration/alert-sources/apimetrics",
18681875
"zh/on-call/integration/alert-integration/alert-sources/gatus",
18691876
"zh/on-call/integration/alert-integration/alert-sources/metoro",
@@ -3311,6 +3318,9 @@
33113318
"en/on-call/integration/alert-integration/alert-sources/aikido",
33123319
"en/on-call/integration/alert-integration/alert-sources/cato",
33133320
"en/on-call/integration/alert-integration/alert-sources/rollbar",
3321+
"en/on-call/integration/alert-integration/alert-sources/trigger-dev",
3322+
"en/on-call/integration/alert-integration/alert-sources/duplicati",
3323+
"en/on-call/integration/alert-integration/alert-sources/robotalp",
33143324
"en/on-call/integration/alert-integration/alert-sources/fivetran",
33153325
"en/on-call/integration/alert-integration/alert-sources/coralogix",
33163326
"en/on-call/integration/alert-integration/alert-sources/uptimeobserver",
@@ -3320,6 +3330,7 @@
33203330
"en/on-call/integration/alert-integration/alert-sources/honeycomb",
33213331
"en/on-call/integration/alert-integration/alert-sources/mongodb-atlas",
33223332
"en/on-call/integration/alert-integration/alert-sources/honeybadger",
3333+
"en/on-call/integration/alert-integration/alert-sources/ofelia",
33233334
"en/on-call/integration/alert-integration/alert-sources/openobserve",
33243335
"en/on-call/integration/alert-integration/alert-sources/postman",
33253336
"en/on-call/integration/alert-integration/alert-sources/contrast",
@@ -3336,9 +3347,11 @@
33363347
"en/on-call/integration/alert-integration/alert-sources/panther",
33373348
"en/on-call/integration/alert-integration/alert-sources/axiom",
33383349
"en/on-call/integration/alert-integration/alert-sources/checkmk",
3350+
"en/on-call/integration/alert-integration/alert-sources/dkron",
33393351
"en/on-call/integration/alert-integration/alert-sources/auvik",
33403352
"en/on-call/integration/alert-integration/alert-sources/domotz",
33413353
"en/on-call/integration/alert-integration/alert-sources/ntopng",
3354+
"en/on-call/integration/alert-integration/alert-sources/kener",
33423355
"en/on-call/integration/alert-integration/alert-sources/ohdear",
33433356
"en/on-call/integration/alert-integration/alert-sources/dbmarlin",
33443357
"en/on-call/integration/alert-integration/alert-sources/cloudamqp",
@@ -3470,6 +3483,7 @@
34703483
"en/on-call/integration/alert-integration/alert-sources/monitive",
34713484
"en/on-call/integration/alert-integration/alert-sources/hosted-graphite",
34723485
"en/on-call/integration/alert-integration/alert-sources/netbeez",
3486+
"en/on-call/integration/alert-integration/alert-sources/coroot",
34733487
"en/on-call/integration/alert-integration/alert-sources/apimetrics",
34743488
"en/on-call/integration/alert-integration/alert-sources/gatus",
34753489
"en/on-call/integration/alert-integration/alert-sources/metoro",

‎en/on-call/integration/alert-integration/alert-sources/cfengine.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ The script does not parse the parameter file. It sends every `KEY='VALUE'` line
5959

6060
<Step title="Upload the script">
6161

62-
1. Log in to Mission Portal, click **Settings** in the top right, and open **Custom notification scripts**
62+
1. Log in to Mission Portal, open the user menu at the top right (Hello, username), choose **Settings**, and open **Custom notification scripts**
6363
2. Click **Add script**, upload `flashduty_custom_action.sh`, and enter a name (for example `Flashduty`) and a description
6464
3. Click **Save**
6565

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
---
2+
title: "Coroot Alert Integration"
3+
description: "Sync Coroot alert and SLO incident open and resolve notifications to Flashduty On-call through a Webhook integration."
4+
keywords: ["alert integration", "Coroot", "Webhook", "Incident", "Alert", "observability"]
5+
---
6+
7+
Sync the alerts (alerting rules) and SLO incidents of a Coroot project to Flashduty On-call through Coroot's Webhook integration. Coroot sends one notification when an alert or incident opens and one when it resolves; both map to the same Flashduty alert, which closes automatically on resolution.
8+
9+
<div className="hide">
10+
11+
## In Flashduty On-call
12+
---
13+
14+
You can get the integration push URL in either of the following ways.
15+
16+
### Use a dedicated integration
17+
18+
1. In the Flashduty console, go to **Channels** and open a channel
19+
2. Select **Configuration** → **Integrations** → **Private integration**, then click **Add an integration**
20+
3. Select **Coroot** and click **Save**
21+
4. Open the new integration card and copy the **push URL**
22+
23+
### Use a shared integration
24+
25+
1. In the Flashduty console, go to **Integration Center → Alert Events**
26+
2. Select **Coroot** and enter an integration name
27+
3. Configure the default route and pick a channel; add more rules under **Routes** later
28+
4. Click **Save** and copy the generated **push URL**
29+
30+
</div>
31+
32+
## In Coroot
33+
---
34+
35+
<Steps>
36+
<Step title="Create a Webhook integration">
37+
38+
1. In Coroot, go to **Project Settings → Integrations** and create a **Webhook** integration
39+
2. Paste the full Flashduty push URL into **Webhook URL**
40+
3. Enable **Incidents** and **Alerts**. **Deployments** do not become alerts, so leave it off (if enabled, Flashduty receives and ignores deployment notifications)
41+
4. No HTTP basic authentication is needed; Coroot always sends `Content-Type: application/json`
42+
43+
</Step>
44+
45+
<Step title="Fill in the templates">
46+
47+
Use Coroot's built-in `json` function in both the **Incident template** and the **Alert template**:
48+
49+
```gotemplate
50+
{{ json . }}
51+
```
52+
53+
<Warning>
54+
Keep `{{ json . }}` and do not replace it with a custom text template. Flashduty reads the alert or incident ID from the `url` field of the JSON as the Alert Key, and rejects the request when it is missing.
55+
</Warning>
56+
57+
To carry fixed values such as the environment or team into Flashduty, add key-value pairs under **Custom fields** (for example `environment` = `production`). Coroot puts them at the top level of the JSON, and Flashduty turns them into labels.
58+
59+
</Step>
60+
61+
<Step title="Turn on notification routing">
62+
63+
1. Go to **Project Settings → Applications** and select an application category
64+
2. Turn on **Webhook** for **Incidents** and **Alerts**; repeat for every category you want to cover
65+
66+
Coroot only sends webhooks for categories that have notifications enabled.
67+
68+
</Step>
69+
70+
<Step title="Verify">
71+
72+
1. Back in the Webhook integration form, click **Test**. Flashduty returns success and opens an Info alert titled `Coroot test notification`, which you close manually. Each click opens a new test alert
73+
2. Let an alerting rule or SLO actually fire and confirm Flashduty shows an active alert; then let it resolve and confirm the same alert closes
74+
75+
</Step>
76+
</Steps>
77+
78+
## Alert Key
79+
---
80+
81+
Flashduty uses the ID carried in the `url` field of the Coroot notification as the Alert Key:
82+
83+
| Coroot notification | Source | Alert Key |
84+
| :--- | :--- | :--- |
85+
| Alert | `alert` parameter of `url`, such as `.../alerts?alert=abc123def456` | `alert:abc123def456` |
86+
| Incident | `incident` parameter of `url`, such as `.../incidents?incident=x1y2z3w4` | `incident:x1y2z3w4` |
87+
88+
Coroot generates a unique ID when it creates an alert or incident, and the open and resolve notifications use the same one. If the same rule fires again on the same application, Coroot creates a new ID, so it is a new Flashduty alert. Changes to severity, rule name, summary or the domain in `url` never change the Alert Key.
89+
90+
## Status and severity
91+
---
92+
93+
Coroot sends a notification only when an alert or incident opens or resolves; a severity change in between is not notified.
94+
95+
| Coroot `status` | Flashduty status or severity |
96+
| :--- | :--- |
97+
| `CRITICAL` | Critical |
98+
| `WARNING` | Warning |
99+
| `INFO` | Info |
100+
| `OK` | Recovery |
101+
102+
- When an alert resolves, the original severity comes from the `severity` field (`warning` or `critical`)
103+
- An incident notification carries only `status`, so its recovery event gets the Info severity
104+
- An unrecognized `status` is treated as Warning, so that Coroot is not made to pause later notifications by an error response
105+
106+
## Labels
107+
---
108+
109+
| Label | Meaning |
110+
| :--- | :--- |
111+
| `resource` | Application name |
112+
| `application` | The full Coroot application ID, `[cluster:]namespace:Kind:name` |
113+
| `namespace`, `kind` | Namespace and kind from the application ID |
114+
| `check` | Rule name for an alert; always `SLO` for an incident |
115+
| `rule_name`, `severity`, `project_name` | Rule, severity and project name of an alert |
116+
| `alert_id` or `incident_key` | The ID used for the Alert Key |
117+
| Custom fields | Custom fields configured on the integration |
118+
119+
## Troubleshooting
120+
---
121+
122+
<AccordionGroup>
123+
<Accordion title="Coroot logs show failed to send alert ... 400">
124+
Check that both templates are `{{ json . }}` and that the push URL is complete and includes `integration_key`. Flashduty returns 400 when it cannot read an ID from `url`; Coroot then keeps retrying for an hour and holds back later notifications to the same destination.
125+
</Accordion>
126+
127+
<Accordion title="An alert does not recover">
128+
Check that neither the Alert template nor the Incident template was changed to custom text, and that the application category has Webhook enabled for both **Alerts** and **Incidents**. The resolve notification is linked to the trigger only through the ID in `url`.
129+
</Accordion>
130+
131+
<Accordion title="No Alert notifications arrive">
132+
Coroot sends no Alert notification when the Alert template is empty, so make sure it is filled in. Also check that Webhook is enabled for **Alerts** on the category under **Project Settings → Applications**.
133+
</Accordion>
134+
135+
<Accordion title="The test succeeds but real alerts do not arrive">
136+
The Test button sends one fixed test incident and bypasses application category routing. Check the notification settings of the category the application belongs to, and whether the alerting rule really fires.
137+
</Accordion>
138+
139+
<Accordion title="Do deployment notifications become alerts?">
140+
No. Deployment notifications (`url` points to the application's Deployments page) are accepted and ignored by Flashduty, whatever their `status`.
141+
</Accordion>
142+
</AccordionGroup>
143+
144+
For the field reference, see Coroot's [Webhook](https://docs.coroot.com/alerting/webhook) and [Alerts](https://docs.coroot.com/alerting/alerts) documentation.
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
title: "Dkron alert integration"
3+
description: "Send Dkron job failure and recovery events to Flashduty On-call through a webhook notification."
4+
keywords: ["alert integration", "Dkron", "cron", "scheduled jobs", "webhook"]
5+
---
6+
7+
Dkron can send a webhook notification after every job execution. Point it at a Flashduty Push URL and a failed execution opens an alert in Flashduty; the next successful execution of the same job recovers that alert.
8+
9+
<div className="hide">
10+
11+
## In Flashduty On-call
12+
---
13+
14+
You can get the Push URL in either of two ways.
15+
16+
### Use a dedicated integration
17+
18+
1. In the Flashduty console, go to **Channels** and open a channel
19+
2. Go to **Settings** → **Integrations** → **Dedicated integrations** and click **Add integration**
20+
3. Select **Dkron** and click **Save**
21+
4. Open the integration card and copy the **Push URL**
22+
23+
### Use a shared integration
24+
25+
1. In the Flashduty console, go to **Integration Center → Alert events**
26+
2. Select **Dkron** and enter an integration name
27+
3. Configure the default route and choose a channel; you can add more rules under **Routes** after creation
28+
4. Click **Save** and copy the generated **Push URL**
29+
30+
</div>
31+
32+
## Configure Dkron
33+
---
34+
35+
Dkron has no fixed webhook body. It renders the Go template you set in `webhook-payload` and posts the result. Flashduty parses only the fields the template below produces, so use it as is. The webhook is a server-wide Dkron setting that applies to every job. It is sent once per finished job execution (after retries are used up), by the leader node, for both successes and failures.
36+
37+
<Steps>
38+
<Step title="Configure the webhook">
39+
40+
Add the following to the Dkron configuration file (for example `/etc/dkron/dkron.yml`) and replace `webhook-endpoint` with the complete Flashduty Push URL:
41+
42+
```yaml
43+
webhook-endpoint: "https://api.flashcat.cloud/event/push/alert/dkron?integration_key=YOUR_INTEGRATION_KEY"
44+
webhook-headers:
45+
- "Content-Type:application/json"
46+
webhook-payload: '{"job_name":{{printf "%q" .JobName}},"success":{{.Success}},"node":{{printf "%q" .NodeName}},"reporting_node":{{printf "%q" .ReportingNode}},"started_at":"{{.StartTime}}","finished_at":"{{.FinishedAt}}"}'
47+
```
48+
49+
You can also use the command-line flags `--webhook-endpoint`, `--webhook-headers` and `--webhook-payload`, or the environment variables `DKRON_WEBHOOK_ENDPOINT`, `DKRON_WEBHOOK_HEADERS` and `DKRON_WEBHOOK_PAYLOAD`.
50+
51+
<Warning>
52+
- Keep `job_name` and `success`. Flashduty rejects a request that lacks either one.
53+
- Do not add `{{.Output}}` to the template. Job output can contain quotes, control characters, or sensitive data, and Dkron does not escape it for JSON, so one unescaped character invalidates the whole body.
54+
- `{{printf "%q" ...}}` quotes and escapes text such as the job name. Do not replace it with `"{{.JobName}}"`.
55+
</Warning>
56+
57+
</Step>
58+
59+
<Step title="Restart Dkron">
60+
61+
Restart the Dkron servers so the configuration takes effect. In a cluster, give every server node the same configuration so notifications keep flowing after a leader change.
62+
63+
</Step>
64+
65+
<Step title="Verify the lifecycle">
66+
67+
Create a job that fails, for example with the command `false`, and run it manually. Confirm a Critical alert appears in Flashduty. Then change the job to a command that succeeds and run it again, and confirm the alert recovers. Dkron has no webhook test button, so a real execution is the only way to test.
68+
69+
</Step>
70+
</Steps>
71+
72+
## Alert Key
73+
---
74+
75+
Flashduty uses the job name, `job_name` (Dkron's `JobName`), as the Alert Key. The job name identifies a job in Dkron, and every execution result of that job, success or failure, carries the same name, so a failure and the success after it land on the same alert. Changes to the node, the start and end times, or the outcome do not change the Alert Key.
76+
77+
## Status and severity
78+
---
79+
80+
| Dkron execution result | Flashduty status or severity |
81+
| :--- | :--- |
82+
| `success` is `false` | Critical |
83+
| `success` is `true` | Recovered; original severity Critical |
84+
85+
Dkron has no severity, so every failed job is Critical. A request whose `success` is empty or is neither `true` nor `false` is rejected, so a request with an unknown state never lands in the wrong alert lifecycle.
86+
87+
## Alert labels
88+
---
89+
90+
| Label | Source |
91+
| :--- | :--- |
92+
| `job_name`, `check`, `resource` | Job name |
93+
| `node` | Node that ran the job |
94+
| `reporting_node` | Leader node that sent the notification |
95+
| `started_at`, `finished_at` | Start and end time of the execution |
96+
97+
## Troubleshooting
98+
---
99+
100+
- **Flashduty receives nothing**: Dkron only logs webhook send errors and does not retry. Confirm the server can reach `api.flashcat.cloud` and that the Push URL is complete and includes `integration_key`
101+
- **Dkron logs `notifier: error parsing template`**: the `webhook-payload` template has a syntax error. Compare the quotes and braces with the template above
102+
- **Flashduty returns a parameter error**: confirm the body is valid JSON with a non-empty `job_name` and a `success` of `true` or `false`
103+
- **The alert does not recover**: recovery comes from the next successful execution of the same job. If the job keeps failing or runs only once, the alert stays open; turn on [auto-close](/en/on-call/channel/create-edit) in the channel as a fallback
104+
- **Job retries**: Dkron notifies only after retries are used up, so failures during retries do not create alerts
105+
106+
For more settings, see [Dkron Configuration](https://dkron.io/docs/basics/configuration/).

0 commit comments

Comments
 (0)