Skip to content

docs(agents): document evaluation spans in the Agents product - #19851

Merged
ArthurKnaus merged 2 commits into
masterfrom
arthurknaus/tet-3059-agent-evaluation-product-docs
Oct 8, 2026
Merged

ArthurKnaus merged 2 commits into
masterfrom
arthurknaus/tet-3059-agent-evaluation-product-docs

Conversation

@ArthurKnaus

@ArthurKnaus ArthurKnaus commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

DESCRIBE YOUR PR

Document evaluation spans (gen_ai.evaluate, such as TypeSafe Jev) on the Agents product surface.

  • Add an Evaluation Spans section to the Conversations page covering the transcript row, the timeline, and the span detail (Input questions, Output answers with probability bars and confidence), with screenshots
  • Add evaluations to the transcript row-type list
  • Add evaluations keywords to the Agents overview

Closes TET-3059

IS YOUR CHANGE URGENT?

Help us prioritize incoming PRs by letting us know when the change needs to go live.
Select exactly one option. For deadlines, replace YYYY-MM-DD with the due date. You can update this information later by editing the PR description.

  • Urgent deadline (GA date, etc.): YYYY-MM-DD
  • Other deadline: YYYY-MM-DD
  • No deadline: Not urgent, can wait up to 1 week+

SLA

  • Teamwork makes the dream work, so please add a reviewer to your PRs.
  • Please give the docs team up to 1 week to review your PR unless you've supplied a deadline.

Thanks in advance for your help!

PRE-MERGE CHECKLIST

Make sure you've checked the following before merging your changes:

  • Checked Vercel preview for correctness, including links
  • PR was reviewed and approved by any necessary SMEs (subject matter experts)
  • PR was reviewed and approved by a member of the Sentry docs team

🤖 Generated with Claude Code

Add an Evaluation Spans section to the Conversations product page
covering how gen_ai.evaluate spans (e.g. TypeSafe Jev) appear in the
transcript, timeline, and span detail, with screenshots. Add
evaluations keywords to the Agents overview.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
sentry-docs Ready Ready Preview Oct 8, 2026 1:04pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
develop-docs Ignored Ignored Preview Oct 8, 2026 1:04pm UTC

Request Review

@linear-code

linear-code Bot commented Oct 8, 2026

Copy link
Copy Markdown

TET-3059

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ArthurKnaus
ArthurKnaus requested a review from a team October 8, 2026 12:58
@ArthurKnaus
ArthurKnaus marked this pull request as ready for review October 8, 2026 12:59
@codeowner-assignment
codeowner-assignment Bot requested a review from a team October 8, 2026 12:59
@cursor

cursor Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

The conversations page now documents evaluation spans and adds screenshots. The plan checks client errors on that page. An issue escalates when those errors appear.

Rollout complete

Services: sentry-docs.

Plan

What changed

The conversations docs page now describes evaluation spans in the transcript, timeline, and span detail. Readers see evaluator results, question cards, and answer cards with probability bars.

This copy is live after a production deploy of sentry-docs. The URL is /product/agents/conversations/.
The Agents overview page only adds evaluations and LLM-as-judge keywords.
This change ships only from the docs/ tree, not from develop-docs/.

Risk

Bad MDX can fail the page render for every reader of that URL. Missing screenshots leave broken images in the new section. Wrong instrumentation links send readers to missing headings. This change does not alter the Agents product runtime.

Intended effect

This is a documentation change. No Sentry signal shows that the new Evaluation Spans section is live. Treat intended_effect as unknown from telemetry. absent would mean the live page still lacks that section and the three screenshots. Confirm on the live page after deploy.

Regression watch

New images and extra MDX can break render on this page. Client errors on that URL are the symptom to watch. Pageload volume for this URL was not visible, so the error count has no traffic denominator. Zero errors may mean no visits.

Signal Baseline Window Comparison Source
Client errors on /product/agents/conversations/ 0 events 2026-10-07T13:00:40Z to 2026-10-08T13:00:40Z Hold at 0. Any new event is a regression. Sentry errors, org sentry, project docs. Query: transaction:/product/agents/conversations/ AND environment:production. Dataset errors, field count().

If errors appear, inspect events for MDX render failures or missing image requests on that page.

Not observable

  • The new Evaluation Spans copy and screenshots. No content signal exists.
  • Pageload and navigation volume for this URL. A traces query for transaction:"/product/agents/conversations/" in production returned no spans in this window. A second query for transactions that contain this path also returned none. Next.js serves docs from app/[[...path]]/page.tsx, so the transaction name may be the catch-all route. Traces sample at 0.3.
  • LCP and CLS for this URL. A metrics query returned null percentiles.
  • HTTP 404s for the three new screenshot files.
  • Broken hash links to the JavaScript and Python evaluation-span instrumentation headings.

Get Slack DMs about this change: cursor.com/automations/rollouts

Comment thread docs/product/agents/conversations/index.mdx
@ArthurKnaus
ArthurKnaus merged commit ed04528 into master Oct 8, 2026
27 of 28 checks passed
@ArthurKnaus
ArthurKnaus deleted the arthurknaus/tet-3059-agent-evaluation-product-docs branch October 8, 2026 13:20
@cursor

cursor Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Verified in production

  • The live conversations page shows the Evaluation Spans section.

This branch was successfully deployed

1 active deployment
Preview – sentry-docs — 04d85498 Deployed Oct 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Priority: Normal Docs review has no urgent deadline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants