Skip to content

Sync LLM Obs annotation access docs #4212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
api-clients-generation-pipeline wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Sync LLM Obs annotation access docs #4212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 139 additions & 3 deletions .generator/schemas/v2/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55263,6 +55263,10 @@ components:
items:
$ref: "#/components/schemas/LLMObsAnnotationItem"
type: array
can_annotate:
description: Whether the current caller can annotate this interaction.
example: true
type: boolean
content_id:
description: Upstream entity identifier (trace ID, session ID, or deterministic display_block ID).
example: "trace-abc-123"
Expand Down Expand Up @@ -55301,6 +55305,7 @@ components:
- modified_at
- queue_id
- queue_name
- can_annotate
- annotations
type: object
LLMObsAnnotatedInteractionItem:
Expand Down Expand Up @@ -55365,6 +55370,7 @@ components:
description: List of interactions with their annotations.
example:
- annotations: []
can_annotate: true
content_id: trace-abc-123
id: "interaction-456"
type: trace
Expand Down Expand Up @@ -55423,6 +55429,8 @@ components:
description: ID of the annotation that failed, if applicable.
example: "00000000-0000-0000-0000-000000000000"
type: string
code:
$ref: "#/components/schemas/LLMObsAnnotationErrorCode"
error:
description: Error message.
example: "interaction not found"
Expand All @@ -55435,6 +55443,14 @@ components:
- interaction_id
- error
type: object
LLMObsAnnotationErrorCode:
description: Stable error code. `permission_denied` indicates the item was rejected by queue access rules.
enum:
- permission_denied
example: permission_denied
type: string
x-enum-varnames:
- PERMISSION_DENIED
LLMObsAnnotationItem:
description: A single annotation on an interaction.
properties:
Expand Down Expand Up @@ -55613,6 +55629,21 @@ components:
description: Identifier of the project this queue belongs to.
example: "00000000-0000-0000-0000-000000000002"
type: string
restrict_to_assignees:
description: Whether annotation access is restricted to assigned users.
example: false
type: boolean
restrict_to_reviewers:
description: Whether annotation access is restricted to queue reviewers.
example: true
type: boolean
reviewer_emails:
description: Email addresses of reviewers who can access the annotation queue.
example:
- "reviewer@example.com"
items:
type: string
type: array
required:
- name
- project_id
Expand All @@ -55622,6 +55653,11 @@ components:
properties:
annotation_schema:
$ref: "#/components/schemas/LLMObsAnnotationSchema"
can_manage_access:
description: Whether the current caller can manage access for the annotation queue.
example: true
readOnly: true
type: boolean
created_at:
description: Timestamp when the queue was created.
example: "2024-01-15T10:30:00Z"
Expand Down Expand Up @@ -55656,6 +55692,21 @@ components:
description: Identifier of the project this queue belongs to.
example: "00000000-0000-0000-0000-000000000002"
type: string
restrict_to_assignees:
description: Whether annotation access is restricted to assigned users.
example: false
type: boolean
restrict_to_reviewers:
description: Whether annotation access is restricted to queue reviewers.
example: true
type: boolean
reviewer_emails:
description: Email addresses of reviewers for the annotation queue. Returned only when the caller can manage queue access.
example:
- "reviewer@example.com"
items:
type: string
type: array
required:
- name
- project_id
Expand All @@ -55665,6 +55716,9 @@ components:
- modified_by
- modified_at
- owned_by
- restrict_to_reviewers
- restrict_to_assignees
- can_manage_access
type: object
LLMObsAnnotationQueueDataRequest:
description: Data object for creating an LLM Observability annotation queue.
Expand Down Expand Up @@ -55881,6 +55935,21 @@ components:
description: Updated name of the annotation queue.
example: "Updated queue name"
type: string
restrict_to_assignees:
description: Whether annotation access is restricted to assigned users.
example: false
type: boolean
restrict_to_reviewers:
description: Whether annotation access is restricted to queue reviewers.
example: true
type: boolean
reviewer_emails:
description: Updated email addresses of reviewers who can access the annotation queue.
example:
- "reviewer@example.com"
items:
type: string
type: array
type: object
LLMObsAnnotationQueueUpdateDataRequest:
description: Data object for updating an LLM Observability annotation queue.
Expand Down Expand Up @@ -57765,6 +57834,10 @@ components:
items:
$ref: "#/components/schemas/LLMObsAnnotationItem"
type: array
can_annotate:
description: Whether the current caller can annotate this interaction.
example: true
type: boolean
content_id:
description: Server-generated deterministic identifier derived from the block list.
example: "9a87f3e2b1d4c5a6f8b3e2d1c4a7b5f6e3d2a1c4b7e5f8a3d6c2e1b4a7d5f8c2"
Expand All @@ -57781,6 +57854,7 @@ components:
- id
- type
- content_id
- can_annotate
- annotations
- display_block
type: object
Expand Down Expand Up @@ -61771,6 +61845,10 @@ components:
items:
$ref: "#/components/schemas/LLMObsAnnotationItem"
type: array
can_annotate:
description: Whether the current caller can annotate this interaction.
example: true
type: boolean
content_id:
description: Upstream entity identifier supplied by the caller.
example: "trace-abc-123"
Expand All @@ -61797,6 +61875,7 @@ components:
- content_id
- created_at
- modified_at
- can_annotate
- annotations
type: object
LLMObsTraceInteractionItem:
Expand Down Expand Up @@ -159315,6 +159394,7 @@ paths:
value: good
modified_at: "0001-01-01T00:00:00Z"
modified_by: "00000000-0000-0000-0000-000000000002"
can_annotate: true
content_id: trace-abc-123
created_at: "2025-06-01T12:00:00Z"
id: interaction-456
Expand Down Expand Up @@ -159392,6 +159472,7 @@ paths:
value:
data:
- attributes:
can_manage_access: true
created_at: "2024-01-15T10:30:00Z"
created_by: 00000000-0000-0000-0000-000000000002
description: Queue for annotating customer support traces
Expand All @@ -159400,6 +159481,10 @@ paths:
name: My annotation queue
owned_by: 00000000-0000-0000-0000-000000000002
project_id: 00000000-0000-0000-0000-000000000002
restrict_to_assignees: false
restrict_to_reviewers: true
reviewer_emails:
- reviewer@example.com
id: 00000000-0000-0000-0000-000000000001
type: queues
schema:
Expand Down Expand Up @@ -159438,6 +159523,7 @@ paths:
description: |-
Create an annotation queue. The `name` and `project_id` fields are required.
An optional `annotation_schema` can be provided to define the labels for the queue.
Optional access-control fields can restrict annotation access to reviewers or assignees.
Fields such as `created_by`, `owned_by`, `created_at`, `modified_by`,
and `modified_at` are inferred by the backend.
operationId: CreateLLMObsAnnotationQueue
Expand All @@ -159453,6 +159539,19 @@ paths:
name: My annotation queue
project_id: 00000000-0000-0000-0000-000000000002
type: queues
with_access_controls:
summary: Create queue with access controls
value:
data:
attributes:
description: Queue for annotating customer support traces
name: Restricted annotation queue
project_id: 00000000-0000-0000-0000-000000000002
restrict_to_assignees: false
restrict_to_reviewers: true
reviewer_emails:
- reviewer@example.com
type: queues
with_schema:
summary: Create queue with annotation schema
value:
Expand Down Expand Up @@ -159488,6 +159587,7 @@ paths:
value:
data:
attributes:
can_manage_access: true
created_at: "2024-01-15T10:30:00Z"
created_by: 00000000-0000-0000-0000-000000000002
description: Queue for annotating customer support traces
Expand All @@ -159496,6 +159596,10 @@ paths:
name: My annotation queue
owned_by: 00000000-0000-0000-0000-000000000002
project_id: 00000000-0000-0000-0000-000000000002
restrict_to_assignees: false
restrict_to_reviewers: true
reviewer_emails:
- reviewer@example.com
id: 00000000-0000-0000-0000-000000000001
type: queues
schema:
Expand Down Expand Up @@ -159570,7 +159674,7 @@ paths:
If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/).
patch:
description: |-
Partially update an annotation queue. The `name`, `description`, and `annotation_schema` fields can be updated.
Partially update an annotation queue. The `name`, `description`, `annotation_schema`, `reviewer_emails`, `restrict_to_reviewers`, and `restrict_to_assignees` fields can be updated.
operationId: UpdateLLMObsAnnotationQueue
parameters:
- $ref: "#/components/parameters/LLMObsAnnotationQueueIDPathParameter"
Expand All @@ -159584,6 +159688,10 @@ paths:
attributes:
description: Updated description
name: Updated queue name
restrict_to_assignees: false
restrict_to_reviewers: true
reviewer_emails:
- reviewer@example.com
type: queues
schema:
$ref: "#/components/schemas/LLMObsAnnotationQueueUpdateRequest"
Expand All @@ -159598,6 +159706,7 @@ paths:
value:
data:
attributes:
can_manage_access: true
created_at: "2024-01-15T10:30:00Z"
created_by: 00000000-0000-0000-0000-000000000002
description: Queue for annotating customer support traces
Expand All @@ -159606,6 +159715,10 @@ paths:
name: My annotation queue
owned_by: 00000000-0000-0000-0000-000000000002
project_id: 00000000-0000-0000-0000-000000000002
restrict_to_assignees: false
restrict_to_reviewers: true
reviewer_emails:
- reviewer@example.com
id: 00000000-0000-0000-0000-000000000001
type: queues
schema:
Expand Down Expand Up @@ -159663,6 +159776,7 @@ paths:
attributes:
annotated_interactions:
- annotations: []
can_annotate: true
content_id: trace-abc-123
id: interaction-456
type: trace
Expand Down Expand Up @@ -159713,6 +159827,7 @@ paths:
by `interaction_id` and the requesting user's identity.
Results and errors in the response are linked to request items by `interaction_id`.
Errors for individual items are returned in the `errors` field without blocking the rest of the batch.
Requests where every item is denied by queue access rules return `403`.
operationId: UpsertLLMObsAnnotations
parameters:
- $ref: "#/components/parameters/LLMObsAnnotationQueueIDPathParameter"
Expand Down Expand Up @@ -159757,9 +159872,30 @@ paths:
modified_by: 00000000-0000-0000-0000-000000000002
id: 00000000-0000-0000-0000-000000000001
type: annotations
mixed_access_denied:
summary: Some annotations were denied by queue access rules
value:
data:
attributes:
annotations:
- created_at: "2024-01-15T10:30:00Z"
created_by: 00000000-0000-0000-0000-000000000002
id: 00000000-0000-0000-0000-000000000000
interaction_id: 00000000-0000-0000-0000-000000000001
label_values:
- label_schema_id: abc-123
value: good
modified_at: "2024-01-15T10:30:00Z"
modified_by: 00000000-0000-0000-0000-000000000002
errors:
- code: permission_denied
error: permission denied
interaction_id: 00000000-0000-0000-0000-000000000003
id: 00000000-0000-0000-0000-000000000001
type: annotations
schema:
$ref: "#/components/schemas/LLMObsAnnotationsResponse"
description: OK — annotations created or updated. Per-item errors are listed in `errors`.
description: OK — annotations created or updated. Mixed batches list denied items and other per-item errors in `errors`
"400":
content:
application/json:
Expand All @@ -159777,7 +159913,7 @@ paths:
application/json:
schema:
$ref: "#/components/schemas/JSONAPIErrorResponse"
description: Forbidden
description: Forbidden — all request items were denied by queue access rules
"404":
content:
application/json:
Expand Down
5 changes: 4 additions & 1 deletion examples/v2/llm-observability/CreateLLMObsAnnotationQueue.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,10 @@ public static void main(String[] args) {
.values(Arrays.asList("good", "bad", "neutral")))))
.description("Queue for annotating customer support traces")
.name("My annotation queue")
.projectId("00000000-0000-0000-0000-000000000002"))
.projectId("00000000-0000-0000-0000-000000000002")
.restrictToAssignees(false)
.restrictToReviewers(true)
.reviewerEmails(Collections.singletonList("reviewer@example.com")))
.type(LLMObsAnnotationQueueType.QUEUES));

try {
Expand Down
5 changes: 4 additions & 1 deletion examples/v2/llm-observability/UpdateLLMObsAnnotationQueue.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@ public static void main(String[] args) {
.type(LLMObsLabelSchemaType.SCORE)
.values(Arrays.asList("good", "bad", "neutral")))))
.description("Updated description")
.name("Updated queue name"))
.name("Updated queue name")
.restrictToAssignees(false)
.restrictToReviewers(true)
.reviewerEmails(Collections.singletonList("reviewer@example.com")))
.type(LLMObsAnnotationQueueType.QUEUES));

try {
Expand Down
6 changes: 3 additions & 3 deletions examples/v2/llm-observability/UpsertLLMObsAnnotations.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Create or update annotations returns "OK — annotations created or updated. Per-item errors are
// listed in `errors`."
// response
// Create or update annotations returns "OK — annotations created or updated. Mixed batches list
// denied items and other
// per-item errors in `errors`" response

import com.datadog.api.client.ApiClient;
import com.datadog.api.client.ApiException;
Expand Down
Loading
Loading