Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 34 additions & 3 deletions .generator/schemas/v2/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77999,6 +77999,13 @@ components:
LLMObsAnnotationItem:
description: A single annotation on an interaction.
properties:
annotator:
description: |-
Caller-supplied name or identifier for the external person or automated system that produced the labels.
The value is stored and returned as provided. It is not a Datadog user ID or login, and it is not
resolved against Datadog users.
example: "Alex Contractor"
type: string
created_at:
description: Timestamp when the annotation was created.
example: "2024-01-15T10:30:00Z"
Expand Down Expand Up @@ -78044,6 +78051,13 @@ components:
LLMObsAnnotationItemResponse:
description: A single annotation on an interaction, as returned by the API.
properties:
annotator:
description: |-
Caller-supplied name or identifier for the external person or automated system that produced the labels.
The value is stored and returned as provided. It is not a Datadog user ID or login, and it is not
resolved against Datadog users.
example: "Alex Contractor"
type: string
created_at:
description: Timestamp when the annotation was created.
example: "2024-01-15T10:30:00Z"
Expand Down Expand Up @@ -84929,6 +84943,18 @@ components:
A single annotation to create or update. The annotation is matched by
`interaction_id` and the requesting user's identity.
properties:
annotator:
description: |-
Caller-supplied name or identifier for the external person or automated system that produced the labels.
The value is an opaque string with no required format. Use a stable, recognizable value for the same
producer across annotations.
The value is stored and returned as provided. It is not a Datadog user ID or login, and it is not
resolved against Datadog users.
Omit this field or set it to `null` to preserve existing attribution. Set it to an empty string to clear
attribution.
example: "Alex Contractor"
nullable: true
type: string
interaction_id:
description: ID of the interaction to annotate.
example: "00000000-0000-0000-0000-000000000001"
Expand Down Expand Up @@ -208224,6 +208250,8 @@ paths:
description: |-
Create or update annotations on interactions in a queue. Each annotation is matched
by `interaction_id` and the requesting user's identity.
Use the optional `annotator` field to name an external person or automated system that produced the labels
without changing the authenticated creator, modifier, permissions, or per-user matching behavior.
Results and errors in the response are linked to request items by `interaction_id`.
Errors for individual items are returned in the `errors` field without blocking the rest of the batch.
Requests where every item is denied by queue access rules return `403`.
Expand All @@ -208239,7 +208267,8 @@ paths:
data:
attributes:
annotations:
- interaction_id: 00000000-0000-0000-0000-000000000001
- annotator: "Alex Contractor"
interaction_id: 00000000-0000-0000-0000-000000000001
label_values:
- label_schema_id: abc-123
value: good
Expand All @@ -208260,7 +208289,8 @@ paths:
data:
attributes:
annotations:
- created_at: "2024-01-15T10:30:00Z"
- annotator: "Alex Contractor"
created_at: "2024-01-15T10:30:00Z"
created_by: 00000000-0000-0000-0000-000000000002
id: 00000000-0000-0000-0000-000000000000
interaction_id: 00000000-0000-0000-0000-000000000001
Expand All @@ -208277,7 +208307,8 @@ paths:
data:
attributes:
annotations:
- created_at: "2024-01-15T10:30:00Z"
- annotator: "Alex Contractor"
created_at: "2024-01-15T10:30:00Z"
created_by: 00000000-0000-0000-0000-000000000002
id: 00000000-0000-0000-0000-000000000000
interaction_id: 00000000-0000-0000-0000-000000000001
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
attributes=LLMObsAnnotationsDataAttributesRequest(
annotations=[
LLMObsUpsertAnnotationItem(
annotator="Alex Contractor",
interaction_id="00000000-0000-0000-0000-000000000001",
label_values=[
LLMObsAnnotationLabelValue(
Expand Down
2 changes: 2 additions & 0 deletions src/datadog_api_client/v2/api/agent_observability_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -4090,6 +4090,8 @@ def upsert_llm_obs_annotations(

Create or update annotations on interactions in a queue. Each annotation is matched
by ``interaction_id`` and the requesting user's identity.
Use the optional ``annotator`` field to name an external person or automated system that produced the labels
without changing the authenticated creator, modifier, permissions, or per-user matching behavior.
Results and errors in the response are linked to request items by ``interaction_id``.
Errors for individual items are returned in the ``errors`` field without blocking the rest of the batch.
Requests where every item is denied by queue access rules return ``403``.
Expand Down
14 changes: 13 additions & 1 deletion src/datadog_api_client/v2/model/llm_obs_annotation_item.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,16 @@
# Copyright 2019-Present Datadog, Inc.
from __future__ import annotations

from typing import Any, Dict
from typing import Any, Dict, Union

from datadog_api_client.model_utils import (
ModelNormal,
cached_property,
date,
datetime,
none_type,
unset,
UnsetType,
UUID,
)

Expand All @@ -19,6 +21,7 @@ class LLMObsAnnotationItem(ModelNormal):
@cached_property
def openapi_types(_):
return {
"annotator": (str,),
"created_at": (datetime,),
"created_by": (str,),
"id": (str,),
Expand All @@ -44,6 +47,7 @@ def openapi_types(_):
}

attribute_map = {
"annotator": "annotator",
"created_at": "created_at",
"created_by": "created_by",
"id": "id",
Expand All @@ -62,11 +66,17 @@ def __init__(
label_values: Dict[str, Any],
modified_at: datetime,
modified_by: str,
annotator: Union[str, UnsetType] = unset,
**kwargs,
):
"""
A single annotation on an interaction.

:param annotator: Caller-supplied name or identifier for the external person or automated system that produced the labels.
The value is stored and returned as provided. It is not a Datadog user ID or login, and it is not
resolved against Datadog users.
:type annotator: str, optional

:param created_at: Timestamp when the annotation was created.
:type created_at: datetime

Expand All @@ -88,6 +98,8 @@ def __init__(
:param modified_by: Identifier of the user who last modified the annotation.
:type modified_by: str
"""
if annotator is not unset:
kwargs["annotator"] = annotator
super().__init__(kwargs)

self_.created_at = created_at
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,14 @@
# Copyright 2019-Present Datadog, Inc.
from __future__ import annotations

from typing import List, TYPE_CHECKING
from typing import List, Union, TYPE_CHECKING

from datadog_api_client.model_utils import (
ModelNormal,
cached_property,
datetime,
unset,
UnsetType,
)


Expand All @@ -24,6 +26,7 @@ def openapi_types(_):
)

return {
"annotator": (str,),
"created_at": (datetime,),
"created_by": (str,),
"id": (str,),
Expand All @@ -34,6 +37,7 @@ def openapi_types(_):
}

attribute_map = {
"annotator": "annotator",
"created_at": "created_at",
"created_by": "created_by",
"id": "id",
Expand All @@ -52,11 +56,17 @@ def __init__(
label_values: List[LLMObsAnnotationLabelValueResponse],
modified_at: datetime,
modified_by: str,
annotator: Union[str, UnsetType] = unset,
**kwargs,
):
"""
A single annotation on an interaction, as returned by the API.

:param annotator: Caller-supplied name or identifier for the external person or automated system that produced the labels.
The value is stored and returned as provided. It is not a Datadog user ID or login, and it is not
resolved against Datadog users.
:type annotator: str, optional

:param created_at: Timestamp when the annotation was created.
:type created_at: datetime

Expand All @@ -79,6 +89,8 @@ def __init__(
:param modified_by: Identifier of the user who last modified the annotation.
:type modified_by: str
"""
if annotator is not unset:
kwargs["annotator"] = annotator
super().__init__(kwargs)

self_.created_at = created_at
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,14 @@
# Copyright 2019-Present Datadog, Inc.
from __future__ import annotations

from typing import List, TYPE_CHECKING
from typing import List, Union, TYPE_CHECKING

from datadog_api_client.model_utils import (
ModelNormal,
cached_property,
none_type,
unset,
UnsetType,
)


Expand All @@ -27,27 +30,46 @@ def openapi_types(_):
from datadog_api_client.v2.model.llm_obs_annotation_label_value import LLMObsAnnotationLabelValue

return {
"annotator": (str, none_type),
"interaction_id": (str,),
"label_values": ([LLMObsAnnotationLabelValue],),
}

attribute_map = {
"annotator": "annotator",
"interaction_id": "interaction_id",
"label_values": "label_values",
}

def __init__(self_, interaction_id: str, label_values: List[LLMObsAnnotationLabelValue], **kwargs):
def __init__(
self_,
interaction_id: str,
label_values: List[LLMObsAnnotationLabelValue],
annotator: Union[str, none_type, UnsetType] = unset,
**kwargs,
):
"""
A single annotation to create or update. The annotation is matched by
``interaction_id`` and the requesting user's identity.

:param annotator: Caller-supplied name or identifier for the external person or automated system that produced the labels.
The value is an opaque string with no required format. Use a stable, recognizable value for the same
producer across annotations.
The value is stored and returned as provided. It is not a Datadog user ID or login, and it is not
resolved against Datadog users.
Omit this field or set it to ``null`` to preserve existing attribution. Set it to an empty string to clear
attribution.
:type annotator: str, none_type, optional

:param interaction_id: ID of the interaction to annotate.
:type interaction_id: str

:param label_values: Label values for this annotation. Each entry references a label schema by ID
and provides the corresponding value validated against the schema type constraints.
:type label_values: [LLMObsAnnotationLabelValue]
"""
if annotator is not unset:
kwargs["annotator"] = annotator
super().__init__(kwargs)

self_.interaction_id = interaction_id
Expand Down
6 changes: 3 additions & 3 deletions tests/v2/features/agent_observability.feature
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,7 @@ Feature: Agent Observability
Given operation "UpsertLLMObsAnnotations" enabled
And new "UpsertLLMObsAnnotations" request
And request contains "queue_id" parameter from "REPLACE.ME"
And body with value {"data": {"attributes": {"annotations": [{"interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [{"label_schema_id": "abc-123", "value": "good"}, {"label_schema_id": "ef56gh78", "value": "positive"}]}]}, "type": "annotations"}}
And body with value {"data": {"attributes": {"annotations": [{"annotator": "Alex Contractor", "interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [{"label_schema_id": "abc-123", "value": "good"}, {"label_schema_id": "ef56gh78", "value": "positive"}]}]}, "type": "annotations"}}
When the request is sent
Then the response status is 400 Bad Request

Expand All @@ -404,7 +404,7 @@ Feature: Agent Observability
Given operation "UpsertLLMObsAnnotations" enabled
And new "UpsertLLMObsAnnotations" request
And request contains "queue_id" parameter from "REPLACE.ME"
And body with value {"data": {"attributes": {"annotations": [{"interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [{"label_schema_id": "abc-123", "value": "good"}, {"label_schema_id": "ef56gh78", "value": "positive"}]}]}, "type": "annotations"}}
And body with value {"data": {"attributes": {"annotations": [{"annotator": "Alex Contractor", "interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [{"label_schema_id": "abc-123", "value": "good"}, {"label_schema_id": "ef56gh78", "value": "positive"}]}]}, "type": "annotations"}}
When the request is sent
Then the response status is 404 Not Found — the queue does not exist.

Expand All @@ -413,7 +413,7 @@ Feature: Agent Observability
Given operation "UpsertLLMObsAnnotations" enabled
And new "UpsertLLMObsAnnotations" request
And request contains "queue_id" parameter from "REPLACE.ME"
And body with value {"data": {"attributes": {"annotations": [{"interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [{"label_schema_id": "abc-123", "value": "good"}, {"label_schema_id": "ef56gh78", "value": "positive"}]}]}, "type": "annotations"}}
And body with value {"data": {"attributes": {"annotations": [{"annotator": "Alex Contractor", "interaction_id": "00000000-0000-0000-0000-000000000001", "label_values": [{"label_schema_id": "abc-123", "value": "good"}, {"label_schema_id": "ef56gh78", "value": "positive"}]}]}, "type": "annotations"}}
When the request is sent
Then the response status is 200 OK — annotations created or updated. For mixed batches, denied items and other per-item errors are listed in `errors`.

Expand Down
Loading