feat(gooddata-eval): support requesting a per-message reasoning effort - #1704
feat(gooddata-eval): support requesting a per-message reasoning effort#1704Tomkess wants to merge 2 commits into
Conversation
GoodData Cloud's chat-conversations endpoint accepts an experimental options.reasoningEffort (LOW/MEDIUM/HIGH) on each POST .../messages call. Thread it through ChatClient.send_message/ask, all 7 agentic evaluators, the CLI dispatcher, and RunConfig, exposed as --reasoning-effort / GD_EVAL_REASONING_EFFORT. Not persisted server-side, so every message the client sends must carry it (unlike agentId, which is set once at conversation creation).
|
Warning Review limit reached
Next review available in: 51 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe evaluation CLI adds ChangesReasoning effort support
Sequence Diagram(s)sequenceDiagram
participant CLI
participant RunConfig
participant AgenticRunner
participant AgenticEvaluator
participant ChatClient
CLI->>RunConfig: Resolve reasoning effort
RunConfig->>AgenticRunner: Pass configured value
AgenticRunner->>AgenticEvaluator: Dispatch evaluation
AgenticEvaluator->>ChatClient: Send messages with reasoning effort
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/gooddata-eval/README.md`:
- Around line 114-121: Update the Markdown example around the “One-off, via
flag” and “Session-wide, via env var” commands to use the repository’s
configured indented code-block style instead of fenced syntax. Preserve the
commands and their explanatory comments unchanged.
In `@packages/gooddata-eval/src/gooddata_eval/cli/main.py`:
- Line 451: Validate the value selected for reasoning_effort in the CLI
argument/configuration flow before passing it to ChatClient, applying the same
LOW/MEDIUM/HIGH constraint used for --reasoning-effort to values from
GD_EVAL_REASONING_EFFORT as well. Add a test covering an invalid environment
value and verify it is rejected.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f3af03b7-cefc-412e-9aff-9e79eb28f955
📒 Files selected for processing (16)
packages/gooddata-eval/README.mdpackages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.pypackages/gooddata-eval/src/gooddata_eval/cli/main.pypackages/gooddata-eval/src/gooddata_eval/core/agentic/alert_skill.pypackages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.pypackages/gooddata-eval/src/gooddata_eval/core/agentic/general_question.pypackages/gooddata-eval/src/gooddata_eval/core/agentic/guardrail.pypackages/gooddata-eval/src/gooddata_eval/core/agentic/metric_skill.pypackages/gooddata-eval/src/gooddata_eval/core/agentic/search_tool.pypackages/gooddata-eval/src/gooddata_eval/core/agentic/visualization.pypackages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.pypackages/gooddata-eval/src/gooddata_eval/core/config.pypackages/gooddata-eval/tests/test_agentic_runner.pypackages/gooddata-eval/tests/test_agentic_visualization.pypackages/gooddata-eval/tests/test_cli.pypackages/gooddata-eval/tests/test_sse_client.py
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1704 +/- ##
==========================================
+ Coverage 78.30% 78.46% +0.16%
==========================================
Files 271 271
Lines 18689 18702 +13
==========================================
+ Hits 14634 14675 +41
+ Misses 4055 4027 -28 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…easoning-effort The argparse choices constraint only covered the CLI flag; an invalid env var value slipped straight through to ChatClient/the API. Reject it in main() before building RunConfig. Addresses CodeRabbit review comment on PR #1704.
1 participant
Summary
GoodData Cloud's chat-conversations endpoint (
POST /api/v1/ai/workspaces/{workspaceId}/chat/conversations/{conversationId}/messages) has an experimentaloptions.reasoningEffortfield (LOW/MEDIUM/HIGH) that hints how much the LLM should reason before answering.gooddata-evalhad no way to set it.This PR adds a
--reasoning-effortflag (andGD_EVAL_REASONING_EFFORTenv var) that threads the value through every messagegd-evalsends.ChatClient.send_message/askgainreasoning_effort: str | None, sent as{"options": {"reasoningEffort": value}}when set, omitted entirely otherwise.metric_skill,alert_skill,visualization,search_tool,general_question,guardrail,conversation) gain the same parameter, threaded to everysend_messagecall — including every turn of multi-turn confirm-loops._dispatch_agentic/run_agentic_items),RunConfig, and the non-agentic_RoutingBackendpath are wired up.Key difference from
agentId(already supported):agentIdis set once at conversation creation and persists for the whole conversation.reasoningEffortis not persisted — it applies only to the message it's sent with, and every other message silently falls back toMEDIUM. So it has to be threaded as a per-send_messageparameter, not aChatClientconstructor parameter, and every call site in every multi-turn loop needs it.Also gated behind an org-level GoodData feature flag — when disabled, any value sent is ignored and
MEDIUMis used.Test plan
ChatClient.send_message(omit-by-default / send-when-given) intest_sse_client.pyreasoning_effortcorrectly (test_agentic_runner.py, new file)test_cli.py)test_agentic_visualization.pyassertions that asserted exactsend_messagecall argspytest -q→ 258 passed, 9 pre-existing failures (8 missingopenaipackage, 1 unrelated pre-existing bug intest_runner.py), no regressionsruff format --check .andruff check .(pinned0.15.20) both clean across the whole repoSummary by CodeRabbit
New Features
Documentation
Tests