Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 73 additions & 5 deletions .generator/schemas/v1/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9793,30 +9793,53 @@ components:
- data_source
- query
type: object
MonitorFormulaAndFunctionDataJobsMonitorOptions:
description: Monitor configuration options for data jobs queries.
properties:
alert_on_every_failure:
description: |-
Alert separately for each failed run instead of on the count of trailing failures.
Only supported for `failed_runs` monitors with the `aastra` query dialect. It requires
a `> 0` threshold with no warning threshold, an evaluation window of at most two days,
and renotification disabled.
example: true
type: boolean
type: object
MonitorFormulaAndFunctionDataJobsQueryDefinition:
description: A formula and functions data jobs query.
properties:
job_type:
description: |-
The type of job being monitored. Valid values include:
`databricks.job`, `spark.application`, `airflow.dag`,
`dbt.job`, `dbt.model`, `dbt.test`, `glue.job`.
`databricks.job`, `spark.application`, `airflow.dag`, `dbt.job`, `glue.job`, `adf.pipeline`.
The `aastra` dialect also accepts `airflow.job`, `dbt.model`, and `dbt.test`.
Custom job types are supported with the `custom.ol.` prefix.
example: "databricks.job"
type: string
jobs_query:
description: Filter expression used to select the jobs to monitor.
description: |-
Filter expression used to select the jobs to monitor. Its syntax depends on `query_dialect`:
a metric tag filter such as `job_name:smoke*` for `metric`, or an Aastra entity search
such as ``search for databricks_job where `@team:data-eng` `` for `aastra`.
example: "job_name:smoke*"
type: string
monitor_options:
$ref: "#/components/schemas/MonitorFormulaAndFunctionDataJobsMonitorOptions"
name:
description: Name of the query for use in formulas. Must be `run_query`.
example: "run_query"
type: string
query_dialect:
description: |-
Query dialect for data jobs queries. Currently only `metric` is supported.
Query dialect for data jobs queries. Valid values are `metric` and `aastra`.
example: "metric"
type: string
run_filter:
description: |-
Optional filter on individual runs, using metrics filter syntax.
Only supported with the `aastra` query dialect.
example: "env:prod"
type: string
required:
- name
- jobs_query
Expand All @@ -9831,6 +9854,18 @@ components:
type: string
x-enum-varnames:
- DATA_QUALITY_METRICS
MonitorFormulaAndFunctionDataQualityDetectionType:
description: |-
How a `data-jobs alert` job duration monitor detects a breach. Must match the
`.detection()` call in the monitor query. Required for job duration monitors.
enum:
- static
- anomaly
example: "anomaly"
type: string
x-enum-varnames:
- STATIC
- ANOMALY
MonitorFormulaAndFunctionDataQualityDiffFunction:
description: |-
Function applied to the measure before it is compared against the predicted bounds.
Expand Down Expand Up @@ -9891,7 +9926,8 @@ components:
The data quality measure to query. Common values include:
`bytes`, `cardinality`, `custom`, `freshness`, `max`, `mean`, `min`,
`nullness`, `percent_negative`, `percent_zero`, `row_count`, `stddev`,
`sum`, `uniqueness`. Additional values may be supported.
`sum`, `uniqueness`. `data-jobs alert` job duration monitors use `duration`.
Additional values may be supported.
example: "row_count"
type: string
MonitorFormulaAndFunctionDataQualityModelBoundsOverride:
Expand Down Expand Up @@ -9961,6 +9997,14 @@ components:
MonitorFormulaAndFunctionDataQualityMonitorOptions:
description: Monitor configuration options for data quality queries.
properties:
consecutive_runs:
description: |-
Number of consecutive runs that must breach before the monitor alerts.
Only supported by `data-jobs alert` job duration monitors. Defaults to `1`.
example: 1
format: int64
minimum: 1
type: integer
crontab_override:
description: Crontab expression to override the default schedule.
example: "* * * 10"
Expand All @@ -9973,6 +10017,8 @@ components:
description: Custom WHERE clause for the query.
example: "USER_ID = 123"
type: string
detection_type:
$ref: "#/components/schemas/MonitorFormulaAndFunctionDataQualityDetectionType"
group_by_columns:
description: Columns to group results by.
example: ["col1", "col2"]
Expand Down Expand Up @@ -10020,6 +10066,12 @@ components:
description: Name of the query for use in formulas.
example: "query1"
type: string
run_filter:
description: |-
Optional filter on individual runs, using metrics filter syntax.
Only supported by `data-jobs alert` job duration monitors.
example: "env:prod"
type: string
schema_version:
description: Schema version for the data quality query.
example: "0.0.1"
Expand Down Expand Up @@ -37275,6 +37327,22 @@ paths:
- `time_window` #m (between 1 and 2880), #h (between 1 and 48).
- `operator` `<`, `<=`, `>`, `>=`, `==`, or `!=`.
- `#` an integer or decimal number used to set the threshold.

**Data Jobs Alert Query**

Job status example: `formula("failed_runs(run_query)").by(group_keys).last(time_window) > #`

Job duration examples: `formula("job_duration(run_query)").detection(static).by(group_keys) > duration`
and `formula("job_duration(run_query)").detection(anomaly).by(group_keys) > 0`

- `run_query`: The name of the single query variable in `options.variables`. Job status monitors
use a data jobs query; job duration monitors use a data quality query with the `duration` measure.
- `group_keys`: The tags to group by, such as `entity_id,name,workspace_name`.
- `time_window`: Number of minutes (#m), hours (#h), or days (#d), up to `10d`.
- `#`: The number of trailing failed runs to alert above, as a non-negative integer.
- `duration`: The duration to alert above, such as `2h`, `30m`, or a number of seconds.
- `options.thresholds.critical`: Must equal the query threshold, in seconds for job duration.
Anomaly job duration monitors omit `options.thresholds`.
operationId: CreateMonitor
requestBody:
content:
Expand Down
62 changes: 62 additions & 0 deletions examples/v1/monitors/CreateMonitor_1267012748.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
"""
Create a Data Jobs duration monitor returns "OK" response
"""

from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.monitors_api import MonitorsApi
from datadog_api_client.v1.model.monitor import Monitor
from datadog_api_client.v1.model.monitor_formula_and_function_data_quality_data_source import (
MonitorFormulaAndFunctionDataQualityDataSource,
)
from datadog_api_client.v1.model.monitor_formula_and_function_data_quality_detection_type import (
MonitorFormulaAndFunctionDataQualityDetectionType,
)
from datadog_api_client.v1.model.monitor_formula_and_function_data_quality_monitor_options import (
MonitorFormulaAndFunctionDataQualityMonitorOptions,
)
from datadog_api_client.v1.model.monitor_formula_and_function_data_quality_query_definition import (
MonitorFormulaAndFunctionDataQualityQueryDefinition,
)
from datadog_api_client.v1.model.monitor_options import MonitorOptions
from datadog_api_client.v1.model.monitor_type import MonitorType

body = Monitor(
name="Example-Monitor",
type=MonitorType.DATA_JOBS_ALERT,
query='formula("job_duration(run_query)").detection(anomaly).by(name,workspace_name,entity_id) > 0',
message="Data jobs duration alert triggered",
tags=[
"test:examplemonitor",
"env:ci",
],
options=MonitorOptions(
variables=[
MonitorFormulaAndFunctionDataQualityQueryDefinition(
name="run_query",
data_source=MonitorFormulaAndFunctionDataQualityDataSource.DATA_QUALITY_METRICS,
schema_version="0.0.1",
measure="duration",
filter="search for databricks_job",
run_filter="env:prod",
group_by=[
"name",
"workspace_name",
"entity_id",
],
monitor_options=MonitorFormulaAndFunctionDataQualityMonitorOptions(
detection_type=MonitorFormulaAndFunctionDataQualityDetectionType.ANOMALY,
consecutive_runs=2,
),
),
],
),
)

configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = MonitorsApi(api_client)
response = api_instance.create_monitor(body=body)

print(response)
53 changes: 53 additions & 0 deletions examples/v1/monitors/CreateMonitor_1302731819.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
"""
Create a Data Jobs monitor with the aastra query dialect returns "OK" response
"""

from os import environ
from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.api.monitors_api import MonitorsApi
from datadog_api_client.v1.model.monitor import Monitor
from datadog_api_client.v1.model.monitor_formula_and_function_data_jobs_monitor_options import (
MonitorFormulaAndFunctionDataJobsMonitorOptions,
)
from datadog_api_client.v1.model.monitor_formula_and_function_data_jobs_query_definition import (
MonitorFormulaAndFunctionDataJobsQueryDefinition,
)
from datadog_api_client.v1.model.monitor_options import MonitorOptions
from datadog_api_client.v1.model.monitor_thresholds import MonitorThresholds
from datadog_api_client.v1.model.monitor_type import MonitorType

body = Monitor(
name="Example-Monitor",
type=MonitorType.DATA_JOBS_ALERT,
query='formula("failed_runs(run_query)").by(name,workspace_name,entity_id).last(1d) > 0',
message="Data jobs alert triggered",
tags=[
"test:examplemonitor",
"env:ci",
],
options=MonitorOptions(
thresholds=MonitorThresholds(
critical=0.0,
),
variables=[
MonitorFormulaAndFunctionDataJobsQueryDefinition(
name="run_query",
jobs_query="search for databricks_job",
job_type="databricks.job",
query_dialect="aastra",
run_filter="env:prod",
monitor_options=MonitorFormulaAndFunctionDataJobsMonitorOptions(
alert_on_every_failure=True,
),
),
],
),
)

configuration = Configuration()
configuration.access_token = environ["DD_BEARER_TOKEN"]
with ApiClient(configuration) as api_client:
api_instance = MonitorsApi(api_client)
response = api_instance.create_monitor(body=body)

print(response)
16 changes: 16 additions & 0 deletions src/datadog_api_client/v1/api/monitors_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -623,6 +623,22 @@ def create_monitor(
* ``operator`` ``<`` , ``<=`` , ``>`` , ``>=`` , ``==`` , or ``!=``.
* ``#`` an integer or decimal number used to set the threshold.

**Data Jobs Alert Query**

Job status example: ``formula("failed_runs(run_query)").by(group_keys).last(time_window) > #``

Job duration examples: ``formula("job_duration(run_query)").detection(static).by(group_keys) > duration``
and ``formula("job_duration(run_query)").detection(anomaly).by(group_keys) > 0``

* ``run_query`` : The name of the single query variable in ``options.variables``. Job status monitors
use a data jobs query; job duration monitors use a data quality query with the ``duration`` measure.
* ``group_keys`` : The tags to group by, such as ``entity_id,name,workspace_name``.
* ``time_window`` : Number of minutes (#m), hours (#h), or days (#d), up to ``10d``.
* ``#`` : The number of trailing failed runs to alert above, as a non-negative integer.
* ``duration`` : The duration to alert above, such as ``2h`` , ``30m`` , or a number of seconds.
* ``options.thresholds.critical`` : Must equal the query threshold, in seconds for job duration.
Anomaly job duration monitors omit ``options.thresholds``.

:param body: Create a monitor request body.
:type body: Monitor
:rtype: Monitor
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License.
# This product includes software developed at Datadog (https://www.datadoghq.com/).
# Copyright 2019-Present Datadog, Inc.
from __future__ import annotations

from typing import Union

from datadog_api_client.model_utils import (
ModelNormal,
cached_property,
unset,
UnsetType,
)


class MonitorFormulaAndFunctionDataJobsMonitorOptions(ModelNormal):
@cached_property
def openapi_types(_):
return {
"alert_on_every_failure": (bool,),
}

attribute_map = {
"alert_on_every_failure": "alert_on_every_failure",
}

def __init__(self_, alert_on_every_failure: Union[bool, UnsetType] = unset, **kwargs):
"""
Monitor configuration options for data jobs queries.

:param alert_on_every_failure: Alert separately for each failed run instead of on the count of trailing failures.
Only supported for ``failed_runs`` monitors with the ``aastra`` query dialect. It requires
a ``> 0`` threshold with no warning threshold, an evaluation window of at most two days,
and renotification disabled.
:type alert_on_every_failure: bool, optional
"""
if alert_on_every_failure is not unset:
kwargs["alert_on_every_failure"] = alert_on_every_failure
super().__init__(kwargs)
Loading
Loading