Skip to content

Add agentest - #155

Open
srpadmanabhuni wants to merge 1 commit into
cleder:mainfrom
srpadmanabhuni:add-agentest-2026-11-21-b
Open

srpadmanabhuni wants to merge 1 commit into
cleder:mainfrom
srpadmanabhuni:add-agentest-2026-11-21-b

Conversation

@srpadmanabhuni

@srpadmanabhuni srpadmanabhuni commented Nov 21, 2026 •

Copy link
Copy Markdown

Adds agentest to the LLM & MCP Testing section, in alphabetical order, following the contributing format (one entry, description ends with a period).

I am the author of agentest. This adds a single listing for the open-source runner.

The PyPI package name is agentest-ai; the import and CLI are agentest. The project is Apache-2.0, offline, and deterministic (165 tests, no LLM judge).

Happy to adjust the wording or section if needed.

Summary by Sourcery

Add agentest to the LLM and MCP testing resources list.

New Features:

  • Add agentest to the README’s LLM & MCP Testing resources list.aa

Documentation:

  • Document agentest as an offline, deterministic testing tool for AI agents, covering OpenTelemetry GenAI traces, A2A AgentCards, and MCP tool lists.

@semanticdiff-com

Copy link
Copy Markdown

Review changes with  SemanticDiff

@sourcery-ai

sourcery-ai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Documentation

Adds agentest to the LLM & MCP Testing section as an offline, deterministic agent-evaluation tool.

  • README.md L85

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@qodo-code-review

qodo-code-review Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

PR Summary by Qodo

List agentest under LLM & MCP Testing

📝 Documentation 🕐 Less than 5 minutes

Grey Divider

AI Description

• Adds agentest to the LLM & MCP Testing section in alphabetical order.
• Describes its offline, deterministic evaluations of agent traces, A2A AgentCards, and MCP tool
 lists.
Merge Risk 🟢 Low

Blast radius 🟢 Low 💥 0 capabilities

  • No production behavior changes identified.
Behavioral complexity 🟢 Low 🧩 Assessed behavior

  • No production behavior changes identified.
Potential damage 🟢 Low 🩹 Assessed behavior

  • No production behavior changes identified.
Assessed commit 835d7cd
High-Level Assessment

A single entry in the existing section follows the repository's contribution format. No architectural alternative is warranted for this documentation-only addition.

Files changed (1) +1 / -0

Documentation (1) +1 / -0
README.mdAdd agentest to the LLM & MCP Testing list +1/-0

Add agentest to the LLM & MCP Testing list

• Adds a repository link and description for agentest ahead of agent-convergence-scorer. The entry highlights offline, deterministic evaluations of OpenTelemetry GenAI traces, A2A AgentCards, and MCP tool lists without an LLM judge.

README.md

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="README.md" line_range="85" />
<code_context>

 ## LLM & MCP Testing

+- [agentest](https://github.com/srpadmanabhuni/agentest) - Offline, deterministic agentic evals for AI agents: 165 tests of OpenTelemetry GenAI traces, A2A AgentCards and MCP tool lists, with no LLM judge.
 - [agent-convergence-scorer](https://github.com/hermes-labs-ai/agent-convergence-scorer) - A CLI and Python library that scores how lexically similar N agent or LLM outputs are: exact-match rate, Jaccard token overlap, divergence point, and a composite convergence score over any list of runs. An eval primitive for measuring reproducibility and fan-out collapse. Zero deps.
 - [mcp-server-fuzzer](https://github.com/Agent-Hellboy/mcp-server-fuzzer) - A comprehensive fuzzing tool designed specifically for testing Model Context Protocol (MCP) servers. It supports both tool argument fuzzing and protocol type fuzzing across multiple transport protocols.
</code_context>
<issue_to_address>
**Testing tools appear out of order**

When the section is expected to remain alphabetized, the `agentest` entry appears before `agent-convergence-scorer`, leaving the section out of alphabetical order for readers.

Move the `agentest` entry after `agent-convergence-scorer`.
</issue_to_address>

Sourcery assessment

Approved.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread README.md

## LLM & MCP Testing

- [agentest](https://github.com/srpadmanabhuni/agentest) - Offline, deterministic agentic evals for AI agents: 165 tests of OpenTelemetry GenAI traces, A2A AgentCards and MCP tool lists, with no LLM judge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Low · Testing tools appear out of order

When the section is expected to remain alphabetized, the agentest entry appears before agent-convergence-scorer, leaving the section out of alphabetical order for readers.

Move the agentest entry after agent-convergence-scorer.

Prompt for AI agents
In `README.md` at line 85:

**Testing tools appear out of order**

When the section is expected to remain alphabetized, the `agentest` entry appears before `agent-convergence-scorer`, leaving the section out of alphabetical order for readers.

Move the `agentest` entry after `agent-convergence-scorer`.

@llamapreview llamapreview Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LlamaPReview — Blocking issues found

The new listing breaks alphabetical order in the section.

Owner action: Move the agentest listing after agent-convergence-scorer.

Review details and evidence
Priority File Finding Evidence
P2 README.md The new listing breaks alphabetical order in the section. confirmed

LlamaPReview checks

  • Reviewed changed regions in README.md.
  • Read the complete PR-head file README.md.

LlamaPReview is open source. Explore the code behind this review.

Comment thread README.md

## LLM & MCP Testing

- [agentest](https://github.com/srpadmanabhuni/agentest) - Offline, deterministic agentic evals for AI agents: 165 tests of OpenTelemetry GenAI traces, A2A AgentCards and MCP tool lists, with no LLM judge.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 | Confidence: High

agent-convergence-scorer alphabetizes before agentest, so the addition breaks the requested ordering. The listing otherwise has the requested section and period-ending format.

Evidence: changed region in README.md; PR-head read of README.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant