diff --git a/CHANGELOG.md b/CHANGELOG.md index 4212614c..15998781 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,22 @@ All notable changes to the [Nucleus Python Client](https://github.com/scaleapi/n The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.19.0](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.19.0) - 2026-07-10 + +### Added +- **Benchmarks.** Full support for benchmark-paradigm evaluation: `NucleusClient.create_benchmark()` (members from `item_ids`, `(dataset_id, ref_id)` `items` pairs, a `slice_id`, or a `dataset_id`; membership frozen at creation), `list_benchmarks()`, `get_benchmark()`, `update_benchmark()`, `delete_benchmark()`, and `list_benchmark_items()`, plus the new `Benchmark` resource with `refresh()` / `update()` / `delete()` / `items()` / `create_evaluation_v2()`. +- **Benchmark evaluations.** `create_benchmark_evaluation_v2(benchmark_id, model_run_id, ...)` evaluates a model run against every benchmark item (uncovered items score as false negatives, keeping leaderboard scores comparable). Accepts `rollup_groups`, legacy `allowed_label_matches` / `allowed_label_matches_id`, `exclusion_rules`, and `preset`. Benchmark evaluations are the only creation surface — dataset/slice-scoped evaluation creation is deprecated platform-wide and was never shipped in this SDK. +- **Rollup groups.** The new `RollupGroup` type (`class_name` + `labels`) is the primary label configuration: each group evaluates a set of raw labels as one class. Presets support it end to end — `create_evaluation_v2_preset()` / `update_evaluation_v2_preset()` accept `rollup_groups` (mutually exclusive with `allowed_label_matches`), and `EvaluationV2Preset` exposes the field. +- **Exclusion rules.** `MetadataExclusionRule`, `LabelExclusionRule`, and `BoxAreaExclusionRule` (or equivalent dicts) drop items/annotations before metrics are computed, passed via `exclusion_rules` on benchmark evaluation create and presets. `EvaluationV2` exposes `benchmark_id`, `rollup_groups`, `slice_id`, `exclusion_rules`, and `exclusion_stats`. +- **Evaluation V2 presets.** Save and reuse evaluation configurations (`name` + label configuration + `exclusion_rules`) via `list_evaluation_v2_presets()`, `create_evaluation_v2_preset()`, `update_evaluation_v2_preset()`, and `delete_evaluation_v2_preset()`, plus the `EvaluationV2Preset` resource (with `update()` / `delete()`). Passing `preset=` to `create_benchmark_evaluation_v2` seeds the label configuration and rules (explicit arguments override the preset's values). +- **Results.** `EvaluationV2.charts()` (mAP summary, per-class AP, confusion matrix, PR/F1 curves, TIDE attribution, AP by size) and `EvaluationV2.examples()` (paginated TP/FP/FN match rows; `match_type` optional) with `EvaluationV2FilterArgs` filtering (confidence/IoU ranges, labels, metadata predicates, `gt_area_range`, `slice_ids`). +- **Cancel & retry.** `EvaluationV2.cancel()` stops a running evaluation; `EvaluationV2.retry()` re-runs a failed one, reusing its configuration. +- **Benchmark leaderboards.** `leaderboard_ranking(metric_type, benchmark_ids, ...)` ranks model runs on one or more benchmarks (metrics: `MAP_50`, `MAP_50_95`, `AP_SMALL`, `AP_MEDIUM`, `AP_LARGE`, `PRECISION`, `RECALL`, `F1`; `scope` / `collapse` controls), and `leaderboard_f1_curve(benchmark_ids, ...)` returns F1-vs-confidence curves for the top runs. Requires a scaleapi server with the REST leaderboard endpoints deployed. +- **Filter schema discovery.** `EvaluationV2.filter_schema()` / `NucleusClient.get_evaluation_v2_filter_schema()` return the evaluation's filter vocabulary (`gt_labels`, `pred_labels`, and item-metadata fields with inferred value types) — the valid inputs for `EvaluationV2FilterArgs`. Requires the same server deployment as the leaderboard endpoints. +- `Dataset.evaluation_label_schema()` returns the dataset's ground-truth and prediction label vocabularies (`gt_labels` / `prediction_labels`) for building rollup groups, label matches, and label exclusion rules. + +> Note: an unreleased 0.18.9 iteration of this branch carried dataset/slice-scoped creation (`create_evaluation_v2`, `create_evaluations_v2_batch`, `only_items_with_predictions`); that surface was removed before release as the platform moved to the benchmark paradigm. + ## [0.18.8](https://github.com/scaleapi/nucleus-python-client/releases/tag/v0.18.8) - 2026-06-17 ### Fixed diff --git a/docs/index.rst b/docs/index.rst index 88ec8c3d..cb310bab 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -12,36 +12,6 @@ Scale Nucleus helps you: Nucleus is a new way—the right way—to develop ML models, helping us move away from the concept of one dataset and towards a paradigm of collections of scenarios. -.. _evaluations-v2: - -Evaluations V2 --------------- - -Evaluation V2 measures how well a **model run** matches ground-truth annotations. -Create a run with :meth:`NucleusClient.create_evaluation_v2`, wait with -:meth:`nucleus.evaluation_v2.EvaluationV2.wait_for_completion`, then read summary metrics with -:meth:`nucleus.evaluation_v2.EvaluationV2.charts` or individual matches with -:meth:`nucleus.evaluation_v2.EvaluationV2.examples`. - -.. code-block:: python - - import nucleus - - client = nucleus.NucleusClient(api_key="YOUR_API_KEY") - evaluation = client.create_evaluation_v2( - model_run_id="run_xxx", - name="my-eval", - allowed_label_matches=[ - nucleus.AllowedLabelMatch( - ground_truth_label="car", - model_prediction_label="vehicle", - ), - ], - ) - evaluation.wait_for_completion() - charts = evaluation.charts(iou_threshold=0.5) - fps = evaluation.examples(match_type="FP", limit=20) - .. _installation: Installation @@ -56,8 +26,8 @@ To use Nucleus, first install it using `pip`: .. _api: -Sections --------- +API Reference +------------- .. toctree:: :maxdepth: 4 diff --git a/nucleus/__init__.py b/nucleus/__init__.py index 12e3c540..055439cc 100644 --- a/nucleus/__init__.py +++ b/nucleus/__init__.py @@ -3,6 +3,8 @@ __all__ = [ "AsyncJob", "AllowedLabelMatch", + "Benchmark", + "BenchmarkItemsPage", "EmbeddingsExportJob", "BoxAnnotation", "DeduplicationJob", @@ -23,8 +25,16 @@ "EvaluationV2Charts", "EvaluationV2ExamplesPage", "EvaluationV2FilterArgs", + "EvaluationV2FilterSchema", "EvaluationV2MatchExample", + "EvaluationV2Preset", "EvaluationV2Status", + "LeaderboardF1CurveEntry", + "LeaderboardRankingEntry", + "MetadataExclusionRule", + "LabelExclusionRule", + "BoxAreaExclusionRule", + "RollupGroup", "Frame", "Keypoint", "KeypointsAnnotation", @@ -93,6 +103,7 @@ ) from .async_job import AsyncJob, EmbeddingsExportJob from .async_utils import make_multiple_requests_concurrently +from .benchmark import Benchmark from .camera_params import CameraParams from .connection import Connection from .constants import ( @@ -139,10 +150,14 @@ from .data_transfer_object.dataset_details import DatasetDetails from .data_transfer_object.dataset_info import DatasetInfo from .data_transfer_object.evaluation_v2 import ( + BenchmarkItemsPage, EvaluationV2Charts, EvaluationV2ExamplesPage, EvaluationV2FilterArgs, + EvaluationV2FilterSchema, EvaluationV2MatchExample, + LeaderboardF1CurveEntry, + LeaderboardRankingEntry, ) from .data_transfer_object.job_status import JobInfoRequestPayload from .dataset import Dataset @@ -161,7 +176,19 @@ NotFoundError, NucleusAPIError, ) -from .evaluation_v2 import AllowedLabelMatch, EvaluationV2, EvaluationV2Status +from .evaluation_v2 import ( + AllowedLabelMatch, + EvaluationV2, + EvaluationV2Status, + RollupGroup, +) +from .evaluation_v2_exclusions import ( + BoxAreaExclusionRule, + EvaluationV2ExclusionRule, + LabelExclusionRule, + MetadataExclusionRule, +) +from .evaluation_v2_preset import _UNSET, EvaluationV2Preset from .job import CustomerJobTypes from .local_deduplication import ( LocalDeduplicationResult, @@ -895,75 +922,512 @@ def commit_model_run( payload = {} return self.make_request(payload, f"modelRun/{model_run_id}/commit") - def create_evaluation_v2( + def get_evaluation_v2(self, evaluation_id: str) -> EvaluationV2: + """Get an evaluation by id. + + Parameters: + evaluation_id: Evaluation id (``evalv2_*``). + + Returns: + :class:`EvaluationV2`. + """ + data = self.get(f"evaluationsV2/{evaluation_id}") + return EvaluationV2.from_json(data, self) + + def list_evaluations_v2(self, model_run_id: str) -> List[EvaluationV2]: + """List evaluations for a model run (newest first). + + Parameters: + model_run_id: Model run id (``run_*``). + + Returns: + List of :class:`EvaluationV2`. + """ + rows = self.get(f"modelRun/{model_run_id}/evaluationsV2") + if not isinstance(rows, list): + raise RuntimeError( + f"Unexpected list evaluations V2 response: {rows!r}" + ) + return [EvaluationV2.from_json(r, self) for r in rows] + + def list_evaluation_v2_presets(self) -> List[EvaluationV2Preset]: + """List the current user's saved Evaluation V2 presets. + + Returns: + List of :class:`EvaluationV2Preset` (presets are private per user). + """ + rows = self.get("evaluationV2Presets") + if not isinstance(rows, list): + raise RuntimeError( + f"Unexpected list evaluation V2 presets response: {rows!r}" + ) + return [EvaluationV2Preset.from_json(r, self) for r in rows] + + def create_evaluation_v2_preset( + self, + name: str, + *, + rollup_groups: Optional[List[RollupGroup]] = None, + allowed_label_matches: Optional[List[AllowedLabelMatch]] = None, + exclusion_rules: Optional[ + List[Union[EvaluationV2ExclusionRule, Dict[str, Any]]] + ] = None, + ) -> EvaluationV2Preset: + """Create a saved Evaluation V2 preset. + + Parameters: + name: Preset name. Must be non-empty and unique among the user's + presets. + rollup_groups: Optional rollup classes (the primary label + configuration); each :class:`RollupGroup` maps raw labels onto + one class name. Mutually exclusive with + ``allowed_label_matches``. + allowed_label_matches: Optional legacy label pairs to treat as + matches. Prefer ``rollup_groups``. + exclusion_rules: Optional rules that drop items/annotations (same + types accepted by :meth:`create_benchmark_evaluation_v2`). + + Returns: + :class:`EvaluationV2Preset`: The created preset. + """ + if rollup_groups is not None and allowed_label_matches is not None: + raise ValueError( + "rollup_groups and allowed_label_matches cannot both be set" + ) + payload: Dict[str, Any] = {"name": name} + if rollup_groups is not None: + payload["rollupGroups"] = [g.to_api_dict() for g in rollup_groups] + if allowed_label_matches is not None: + payload["allowedLabelMatches"] = [ + m.to_api_dict() for m in allowed_label_matches + ] + if exclusion_rules is not None: + payload["exclusionRules"] = [ + rule.to_api_dict() if hasattr(rule, "to_api_dict") else rule + for rule in exclusion_rules + ] + data = self.post(payload, "evaluationV2Presets") + return EvaluationV2Preset.from_json(data, self) + + def update_evaluation_v2_preset( + self, + preset_id: str, + *, + name: Any = _UNSET, + rollup_groups: Any = _UNSET, + allowed_label_matches: Any = _UNSET, + exclusion_rules: Any = _UNSET, + ) -> EvaluationV2Preset: + """Update a saved Evaluation V2 preset. + + Only the fields you pass are changed. Passing ``rollup_groups=None`` + or ``exclusion_rules=None`` clears that field; omitting an argument + leaves it unchanged. + + Parameters: + preset_id: Preset id (``prev_*``). Must be owned by the caller. + name: Optional new name. + rollup_groups: Optional new rollup classes, or ``None`` to clear. + Mutually exclusive with ``allowed_label_matches``. + allowed_label_matches: Optional new legacy label-match list. + exclusion_rules: Optional new exclusion rules, or ``None`` to clear. + + Returns: + :class:`EvaluationV2Preset`: The updated preset. + """ + if ( + rollup_groups is not _UNSET + and rollup_groups is not None + and allowed_label_matches is not _UNSET + and allowed_label_matches is not None + ): + raise ValueError( + "rollup_groups and allowed_label_matches cannot both be set" + ) + payload: Dict[str, Any] = {} + if name is not _UNSET: + payload["name"] = name + if rollup_groups is not _UNSET: + payload["rollupGroups"] = ( + None + if rollup_groups is None + else [g.to_api_dict() for g in rollup_groups] + ) + if allowed_label_matches is not _UNSET: + payload["allowedLabelMatches"] = ( + None + if allowed_label_matches is None + else [m.to_api_dict() for m in allowed_label_matches] + ) + if exclusion_rules is not _UNSET: + payload["exclusionRules"] = ( + None + if exclusion_rules is None + else [ + rule.to_api_dict() + if hasattr(rule, "to_api_dict") + else rule + for rule in exclusion_rules + ] + ) + data = self.patch(payload, f"evaluationV2Presets/{preset_id}") + return EvaluationV2Preset.from_json(data, self) + + def delete_evaluation_v2_preset(self, preset_id: str) -> None: + """Delete a saved Evaluation V2 preset. + + Parameters: + preset_id: Preset id (``prev_*``). Must be owned by the caller. + """ + self.make_request( + {}, + f"evaluationV2Presets/{preset_id}", + requests_command=requests.delete, + return_raw_response=True, + ) + + def create_benchmark( self, + name: str, + *, + description: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + item_ids: Optional[List[str]] = None, + items: Optional[List[Dict[str, str]]] = None, + slice_id: Optional[str] = None, + dataset_id: Optional[str] = None, + ) -> Benchmark: + """Create a benchmark from ground-truth items. + + Provide the members through exactly one source: explicit ``item_ids``, + ``(dataset_id, ref_id)`` pairs via ``items``, all items in a slice via + ``slice_id``, or all items in a dataset via ``dataset_id``. Items + without ground truth are skipped (reported on the returned benchmark + as ``skipped_items_without_ground_truth``). Membership is frozen at + creation. + + Parameters: + name: Benchmark display name. + description: Optional description. + metadata: Optional arbitrary metadata dict. + item_ids: Global dataset item ids (``di_*``). + items: ``{"dataset_id": ..., "ref_id": ...}`` pairs. + slice_id: Slice id (``slc_*``) whose items become members. + dataset_id: Dataset id (``ds_*``) whose items become members. + + Returns: + :class:`Benchmark`: The created benchmark. + """ + sources = [ + source + for source in (item_ids, items, slice_id, dataset_id) + if source is not None + ] + if len(sources) != 1: + raise ValueError( + "Provide exactly one of item_ids, items, slice_id, or " + "dataset_id to define benchmark membership" + ) + payload: Dict[str, Any] = {"name": name} + if description is not None: + payload["description"] = description + if metadata is not None: + payload["metadata"] = metadata + if item_ids is not None: + payload["item_ids"] = item_ids + if items is not None: + payload["items"] = items + if slice_id is not None: + payload["slice_id"] = slice_id + if dataset_id is not None: + payload["dataset_id"] = dataset_id + data = self.post(payload, "benchmarks") + return Benchmark.from_json(data, self) + + def list_benchmarks(self) -> List[Benchmark]: + """List benchmarks visible to the current user. + + Returns: + List of :class:`Benchmark`. + """ + rows = self.get("benchmarks") + if not isinstance(rows, list): + raise RuntimeError( + f"Unexpected list benchmarks response: {rows!r}" + ) + return [Benchmark.from_json(r, self) for r in rows] + + def get_benchmark(self, benchmark_id: str) -> Benchmark: + """Get a benchmark by id. + + Parameters: + benchmark_id: Benchmark id (``bm_*``). + + Returns: + :class:`Benchmark`. + """ + data = self.get(f"benchmarks/{benchmark_id}") + return Benchmark.from_json(data, self) + + def update_benchmark( + self, + benchmark_id: str, + *, + name: Optional[str] = None, + description: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> Benchmark: + """Update a benchmark's name, description, or metadata. + + Only the arguments you pass are changed. Benchmark membership is + frozen at creation and cannot be updated. + + Parameters: + benchmark_id: Benchmark id (``bm_*``). + name: Optional new display name. + description: Optional new description. + metadata: Optional new metadata dict. + + Returns: + :class:`Benchmark`: The updated benchmark. + """ + payload: Dict[str, Any] = {} + if name is not None: + payload["name"] = name + if description is not None: + payload["description"] = description + if metadata is not None: + payload["metadata"] = metadata + data = self.patch(payload, f"benchmarks/{benchmark_id}") + return Benchmark.from_json(data, self) + + def delete_benchmark(self, benchmark_id: str) -> None: + """Delete a benchmark. + + Parameters: + benchmark_id: Benchmark id (``bm_*``). + """ + self.make_request( + {}, + f"benchmarks/{benchmark_id}", + requests_command=requests.delete, + return_raw_response=True, + ) + + def list_benchmark_items( + self, + benchmark_id: str, + *, + limit: Optional[int] = None, + offset: Optional[int] = None, + ) -> BenchmarkItemsPage: + """Return one page of a benchmark's member item ids. + + Parameters: + benchmark_id: Benchmark id (``bm_*``). + limit: Optional page size. + offset: Optional row offset for pagination. + + Returns: + :class:`~nucleus.data_transfer_object.evaluation_v2.BenchmarkItemsPage`. + """ + route = f"benchmarks/{benchmark_id}/items" + params = [] + if limit is not None: + params.append(f"limit={limit}") + if offset is not None: + params.append(f"offset={offset}") + if params: + route = f"{route}?{'&'.join(params)}" + data = self.get(route) + return BenchmarkItemsPage.parse_obj(data) + + def create_benchmark_evaluation_v2( + self, + benchmark_id: str, model_run_id: str, *, name: Optional[str] = None, + rollup_groups: Optional[List[RollupGroup]] = None, allowed_label_matches: Optional[List[AllowedLabelMatch]] = None, allowed_label_matches_id: Optional[str] = None, + exclusion_rules: Optional[ + List[Union[EvaluationV2ExclusionRule, Dict[str, Any]]] + ] = None, + preset: Optional[EvaluationV2Preset] = None, ) -> EvaluationV2: - """Create an evaluation for a model run. + """Evaluate a model run against a benchmark. - The evaluation runs in the background. Call - :meth:`EvaluationV2.wait_for_completion`, then - :meth:`EvaluationV2.charts` or :meth:`EvaluationV2.examples` for results. + Every benchmark item is scored: items the model run has no + predictions for count as false negatives, keeping scores comparable + across runs with different coverage. The evaluation runs in the + background — call :meth:`EvaluationV2.wait_for_completion`, then + :meth:`EvaluationV2.charts` or :meth:`EvaluationV2.examples`. Parameters: - model_run_id: Model run id (``run_*``). + benchmark_id: Benchmark id (``bm_*``). + model_run_id: Model run id (``run_*``). Its predictions must + cover items from the benchmark's datasets. name: Optional display name. - allowed_label_matches: Optional label pairs to treat as matches. - allowed_label_matches_id: Optional id of a saved label-match configuration. + rollup_groups: Optional rollup classes (the primary label + configuration); each :class:`RollupGroup` maps raw labels + onto one class name. Mutually exclusive with the + ``allowed_label_matches*`` arguments. + allowed_label_matches: Optional legacy label pairs to treat as + matches. Prefer ``rollup_groups``. + allowed_label_matches_id: Optional id of a saved label-match + configuration. + exclusion_rules: Optional rules that drop items/annotations + before metrics are computed (see + :mod:`nucleus.evaluation_v2_exclusions`). + preset: Optional :class:`EvaluationV2Preset` whose label + configuration and ``exclusion_rules`` seed this evaluation. + Explicit arguments take precedence over the preset's values. Returns: :class:`EvaluationV2`: The created evaluation. """ - payload: Dict[str, Any] = {} + if preset is not None: + if ( + rollup_groups is None + and allowed_label_matches is None + and allowed_label_matches_id is None + ): + rollup_groups = preset.rollup_groups + if rollup_groups is None: + allowed_label_matches = preset.allowed_label_matches + if exclusion_rules is None and preset.exclusion_rules is not None: + exclusion_rules = list(preset.exclusion_rules) + label_configs = [ + config + for config in ( + rollup_groups, + allowed_label_matches, + allowed_label_matches_id, + ) + if config is not None + ] + if len(label_configs) > 1: + raise ValueError( + "Set at most one of rollup_groups, allowed_label_matches, " + "or allowed_label_matches_id" + ) + payload: Dict[str, Any] = {"model_run_id": model_run_id} if name is not None: payload["name"] = name + if rollup_groups is not None: + payload["rollupGroups"] = [g.to_api_dict() for g in rollup_groups] if allowed_label_matches is not None: payload["allowed_label_matches"] = [ m.to_api_dict() for m in allowed_label_matches ] if allowed_label_matches_id is not None: payload["allowed_label_matches_id"] = allowed_label_matches_id + if exclusion_rules is not None: + payload["exclusionRules"] = [ + rule.to_api_dict() if hasattr(rule, "to_api_dict") else rule + for rule in exclusion_rules + ] result = self.make_request( - payload, f"modelRun/{model_run_id}/evaluationsV2" + payload, f"benchmarks/{benchmark_id}/evaluationsV2" ) eval_id = result.get("evaluation_id") if not eval_id: raise RuntimeError( - f"Unexpected create evaluation V2 response: {result}" + f"Unexpected create benchmark evaluation V2 response: {result}" ) return self.get_evaluation_v2(str(eval_id)) - def get_evaluation_v2(self, evaluation_id: str) -> EvaluationV2: - """Get an evaluation by id. + def leaderboard_ranking( + self, + metric_type: str, + benchmark_ids: List[str], + *, + confidence_threshold: Optional[float] = None, + model_ids: Optional[List[str]] = None, + scope: Optional[str] = None, + collapse: Optional[str] = None, + ) -> List[LeaderboardRankingEntry]: + """Rank model runs on one or more benchmarks by a metric. Parameters: - evaluation_id: Evaluation id (``evalv2_*``). + metric_type: Metric to rank by — one of ``"MAP_50"``, + ``"MAP_50_95"``, ``"AP_SMALL"``, ``"AP_MEDIUM"``, + ``"AP_LARGE"``, ``"PRECISION"``, ``"RECALL"``, ``"F1"``. + benchmark_ids: Benchmark ids (``bm_*``) to rank across. + confidence_threshold: Confidence operating point for + ``PRECISION`` / ``RECALL`` / ``F1``. + model_ids: Optional model ids to restrict the ranking to. + scope: ``"mine"`` (only the caller's evaluations) or ``"all"`` + (default). + collapse: ``"bestPerModel"`` (default), ``"allRuns"``, or + ``"allEvaluations"``. Returns: - :class:`EvaluationV2`. + List of :class:`LeaderboardRankingEntry`, best score first. """ - data = self.get(f"evaluationsV2/{evaluation_id}") - return EvaluationV2.from_json(data, self) + payload: Dict[str, Any] = { + "metric_type": metric_type, + "benchmark_ids": benchmark_ids, + } + if confidence_threshold is not None: + payload["confidence_threshold"] = confidence_threshold + if model_ids is not None: + payload["model_ids"] = model_ids + if scope is not None: + payload["scope"] = scope + if collapse is not None: + payload["collapse"] = collapse + rows = self.post(payload, "leaderboard/ranking") + if not isinstance(rows, list): + raise RuntimeError( + f"Unexpected leaderboard ranking response: {rows!r}" + ) + return [LeaderboardRankingEntry.parse_obj(r) for r in rows] - def list_evaluations_v2(self, model_run_id: str) -> List[EvaluationV2]: - """List evaluations for a model run (newest first). + def leaderboard_f1_curve( + self, + benchmark_ids: List[str], + *, + model_ids: Optional[List[str]] = None, + top_n: int = 5, + ) -> List[LeaderboardF1CurveEntry]: + """Return F1-vs-confidence curves for the top runs on benchmarks. Parameters: - model_run_id: Model run id (``run_*``). + benchmark_ids: Benchmark ids (``bm_*``). + model_ids: Optional model ids to restrict the curves to. + top_n: Number of top-ranked runs to return curves for (default 5). Returns: - List of :class:`EvaluationV2`. + List of :class:`LeaderboardF1CurveEntry`, best F1 first. """ - rows = self.get(f"modelRun/{model_run_id}/evaluationsV2") + payload: Dict[str, Any] = { + "benchmark_ids": benchmark_ids, + "top_n": top_n, + } + if model_ids is not None: + payload["model_ids"] = model_ids + rows = self.post(payload, "leaderboard/f1Curve") if not isinstance(rows, list): raise RuntimeError( - f"Unexpected list evaluations V2 response: {rows!r}" + f"Unexpected leaderboard F1 curve response: {rows!r}" ) - return [EvaluationV2.from_json(r, self) for r in rows] + return [LeaderboardF1CurveEntry.parse_obj(r) for r in rows] + + def get_evaluation_v2_filter_schema( + self, evaluation_id: str + ) -> EvaluationV2FilterSchema: + """Return the filter vocabulary for an evaluation. + + Parameters: + evaluation_id: Evaluation id (``evalv2_*``). + + Returns: + :class:`~nucleus.data_transfer_object.evaluation_v2.EvaluationV2FilterSchema`. + """ + data = self.get(f"evaluationsV2/{evaluation_id}/filterSchema") + return EvaluationV2FilterSchema.parse_obj(data) @deprecated(msg="Prefer calling Dataset.info() directly.") def dataset_info(self, dataset_id: str): @@ -1316,6 +1780,9 @@ def delete(self, route: str): def get(self, route: str): return self.connection.get(route) + def patch(self, payload: dict, route: str): + return self.connection.patch(payload, route) + def post(self, payload: dict, route: str): return self.connection.post(payload, route) diff --git a/nucleus/benchmark.py b/nucleus/benchmark.py new file mode 100644 index 00000000..f7c92802 --- /dev/null +++ b/nucleus/benchmark.py @@ -0,0 +1,172 @@ +"""Benchmarks — frozen, cross-dataset ground-truth item sets for model evaluation. + +A benchmark is a named collection of dataset items (with ground truth) that +model runs are evaluated against. Benchmark evaluations score every benchmark +item: items a model run has no predictions for count as false negatives, so +leaderboard scores stay comparable across runs with different coverage. + +Create and manage benchmarks via :class:`~nucleus.NucleusClient`:: + + benchmark = client.create_benchmark("city-streets-v1", slice_id="slc_...") + evaluation = benchmark.create_evaluation_v2( + model_run_id, + rollup_groups=[RollupGroup("vehicle", ["car", "truck"])], + ) + evaluation.wait_for_completion() +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union + +from nucleus.data_transfer_object.evaluation_v2 import BenchmarkItemsPage +from nucleus.evaluation_v2 import ( + AllowedLabelMatch, + EvaluationV2, + RollupGroup, +) +from nucleus.evaluation_v2_exclusions import EvaluationV2ExclusionRule +from nucleus.evaluation_v2_preset import EvaluationV2Preset + +if TYPE_CHECKING: + from nucleus import NucleusClient + + +@dataclass +class Benchmark: + """A benchmark: a frozen set of ground-truth items models are scored against.""" + + id: str + name: str + description: Optional[str] = None + metadata: Optional[Dict[str, Any]] = None + created_by_user_id: Optional[str] = None + created_at: Optional[str] = None + item_count: Optional[int] = None + dataset_count: Optional[int] = None + skipped_items_without_ground_truth: Optional[int] = None + _client: Optional["NucleusClient"] = field(repr=False, default=None) + + @classmethod + def from_json( + cls, + payload: Dict[str, Any], + client: Optional["NucleusClient"] = None, + ) -> "Benchmark": + return cls( + id=str(payload["benchmark_id"]), + name=str(payload["name"]), + description=payload.get("description"), + metadata=payload.get("metadata"), + created_by_user_id=payload.get("created_by_user_id"), + created_at=payload.get("created_at"), + item_count=payload.get("item_count"), + dataset_count=payload.get("dataset_count"), + skipped_items_without_ground_truth=payload.get( + "skipped_items_without_ground_truth" + ), + _client=client, + ) + + def refresh(self) -> "Benchmark": + """Reload this benchmark from Nucleus. + + Returns: + self, with updated fields. + """ + if self._client is None: + raise RuntimeError( + "Benchmark has no client; use NucleusClient.get_benchmark." + ) + updated = self._client.get_benchmark(self.id) + self.__dict__.update(updated.__dict__) + return self + + def update( + self, + *, + name: Optional[str] = None, + description: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> "Benchmark": + """Update this benchmark's name, description, or metadata. + + Only the arguments you pass are changed. Benchmark membership is + frozen at creation and cannot be updated. + + Returns: + self, with updated fields. + """ + if self._client is None: + raise RuntimeError("Benchmark has no client.") + updated = self._client.update_benchmark( + self.id, + name=name, + description=description, + metadata=metadata, + ) + self.__dict__.update(updated.__dict__) + return self + + def delete(self) -> None: + """Delete this benchmark.""" + if self._client is None: + raise RuntimeError("Benchmark has no client.") + self._client.delete_benchmark(self.id) + + def items( + self, + *, + limit: Optional[int] = None, + offset: Optional[int] = None, + ) -> BenchmarkItemsPage: + """Return one page of this benchmark's member item ids. + + Parameters: + limit: Optional page size. + offset: Optional row offset for pagination. + + Returns: + :class:`~nucleus.data_transfer_object.evaluation_v2.BenchmarkItemsPage`: + The page of dataset item ids and the total member count. + """ + if self._client is None: + raise RuntimeError("Benchmark has no client.") + return self._client.list_benchmark_items( + self.id, limit=limit, offset=offset + ) + + def create_evaluation_v2( + self, + model_run_id: str, + *, + name: Optional[str] = None, + rollup_groups: Optional[List[RollupGroup]] = None, + allowed_label_matches: Optional[List[AllowedLabelMatch]] = None, + allowed_label_matches_id: Optional[str] = None, + exclusion_rules: Optional[ + List[Union[EvaluationV2ExclusionRule, Dict[str, Any]]] + ] = None, + preset: Optional[EvaluationV2Preset] = None, + ) -> EvaluationV2: + """Evaluate a model run against this benchmark. + + See :meth:`NucleusClient.create_benchmark_evaluation_v2` for parameter + details. + + Returns: + :class:`~nucleus.evaluation_v2.EvaluationV2`: The created evaluation. + """ + if self._client is None: + raise RuntimeError("Benchmark has no client.") + return self._client.create_benchmark_evaluation_v2( + self.id, + model_run_id, + name=name, + rollup_groups=rollup_groups, + allowed_label_matches=allowed_label_matches, + allowed_label_matches_id=allowed_label_matches_id, + exclusion_rules=exclusion_rules, + preset=preset, + ) diff --git a/nucleus/connection.py b/nucleus/connection.py index 467600a9..050f12ff 100644 --- a/nucleus/connection.py +++ b/nucleus/connection.py @@ -50,6 +50,11 @@ def delete(self, route: str): def get(self, route: str): return self.make_request({}, route, requests_command=requests.get) + def patch(self, payload: dict, route: str): + return self.make_request( + payload, route, requests_command=requests.patch + ) + def post(self, payload: dict, route: str): return self.make_request( payload, route, requests_command=requests.post diff --git a/nucleus/data_transfer_object/evaluation_v2.py b/nucleus/data_transfer_object/evaluation_v2.py index a1abb443..a7f6e7ab 100644 --- a/nucleus/data_transfer_object/evaluation_v2.py +++ b/nucleus/data_transfer_object/evaluation_v2.py @@ -43,9 +43,11 @@ class MetadataPredicate(DictCompatibleModel): "gt_labels": "gtLabels", "item_metadata": "itemMetadata", "prediction_metadata": "predictionMetadata", + "gt_area_range": "gtAreaRange", "label_equality": "labelEquality", "has_ground_truth": "hasGroundTruth", "tide_background": "tideBackground", + "slice_ids": "sliceIds", } @@ -58,9 +60,11 @@ class EvaluationV2FilterArgs(DictCompatibleModel): gt_labels: Optional[List[str]] = None item_metadata: Optional[List[MetadataPredicate]] = None prediction_metadata: Optional[List[MetadataPredicate]] = None + gt_area_range: Optional[RangeNum] = None label_equality: Optional[Literal["EQ", "NEQ"]] = None has_ground_truth: Optional[bool] = None tide_background: Optional[bool] = None + slice_ids: Optional[List[str]] = None def to_api_filters(self) -> Dict[str, Any]: """Return filters as a dict ready for API requests.""" @@ -160,3 +164,68 @@ class EvaluationV2MatchExample(DictCompatibleModel): class EvaluationV2ExamplesPage(DictCompatibleModel): rows: List[EvaluationV2MatchExample] total: int + + +class EvaluationV2FilterMetadataField(DictCompatibleModel): + key: str + value_type: str + values: List[str] + + +class EvaluationV2FilterSchema(DictCompatibleModel): + """Filter vocabulary for one evaluation's match rows. + + The valid label values and item-metadata predicates accepted by + :class:`EvaluationV2FilterArgs` for that evaluation. + """ + + gt_labels: List[str] + pred_labels: List[str] + metadata_fields: List[EvaluationV2FilterMetadataField] + + +class LeaderboardRankingEntry(DictCompatibleModel): + """One row of a benchmark leaderboard ranking.""" + + evaluation_id: str + evaluation_name: Optional[str] = None + model_run_id: str + model_run_name: Optional[str] = None + model_id: Optional[str] = None + model_name: Optional[str] = None + model_version_major: Optional[int] = None + model_version_minor: Optional[int] = None + model_version_label: Optional[str] = None + parent_model_project_id: Optional[str] = None + dataset_id: Optional[str] = None + dataset_name: Optional[str] = None + score: float + rank: int + + +class LeaderboardF1CurvePoint(DictCompatibleModel): + confidence_threshold: float + score: float + + +class LeaderboardF1CurveEntry(DictCompatibleModel): + """F1-vs-confidence curve for one evaluation on a benchmark leaderboard.""" + + evaluation_id: str + evaluation_name: Optional[str] = None + model_run_id: str + model_run_name: Optional[str] = None + model_id: Optional[str] = None + model_name: Optional[str] = None + dataset_id: Optional[str] = None + dataset_name: Optional[str] = None + best_f1: Optional[float] = None + points: List[LeaderboardF1CurvePoint] + rank: int + + +class BenchmarkItemsPage(DictCompatibleModel): + """One page of benchmark member item ids.""" + + item_ids: List[str] + total: int diff --git a/nucleus/dataset.py b/nucleus/dataset.py index f5efee54..7ece96ca 100644 --- a/nucleus/dataset.py +++ b/nucleus/dataset.py @@ -228,6 +228,21 @@ def slices(self) -> List[Slice]: ) return [Slice.from_request(info, self._client) for info in response] + def evaluation_label_schema(self) -> Dict[str, List[str]]: + """Ground-truth and prediction label vocabularies for this dataset. + + Useful for building :meth:`NucleusClient.create_benchmark_evaluation_v2` + rollup groups / label matches and label exclusion rules without + guessing label names. + + Returns: + A dict with ``"gt_labels"`` (ground-truth annotation labels) and + ``"prediction_labels"`` (model prediction labels). + """ + return self._client.make_request( + {}, f"dataset/{self.id}/labelSchema", requests.get + ) + @property def embedding_indexes(self) -> List[EmbeddingIndex]: """Gets all the embedding indexes belonging to this Dataset.""" diff --git a/nucleus/evaluation_v2.py b/nucleus/evaluation_v2.py index 43f8a03c..9e86aba9 100644 --- a/nucleus/evaluation_v2.py +++ b/nucleus/evaluation_v2.py @@ -7,7 +7,6 @@ from dataclasses import dataclass, field from enum import Enum from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set, Union -from urllib.parse import urlencode import requests @@ -15,6 +14,7 @@ EvaluationV2Charts, EvaluationV2ExamplesPage, EvaluationV2FilterArgs, + EvaluationV2FilterSchema, ) if TYPE_CHECKING: @@ -37,6 +37,23 @@ class EvaluationV2Status(str, Enum): } +def _parse_json_field(value: Any) -> Optional[Any]: + """Normalize a JSONB column that may arrive as a string or already parsed. + + The REST ``GET``/``LIST`` evaluation endpoints return raw DB rows, so the + ``exclusion_rules`` / ``exclusion_stats`` jsonb columns can come back either + already decoded (dict/list) or as a JSON string depending on the driver. + """ + if value is None or isinstance(value, (dict, list)): + return value + if isinstance(value, str): + try: + return json.loads(value) + except (ValueError, TypeError): + return None + return value + + @dataclass class AllowedLabelMatch: """Ground-truth and prediction label pair that counts as a match.""" @@ -51,6 +68,79 @@ def to_api_dict(self) -> Dict[str, str]: } +def parse_allowed_label_matches( + raw_matches: Any, +) -> Optional[List[AllowedLabelMatch]]: + """Parse an ``allowed_label_matches`` array from an API payload. + + Accepts both the camelCase (``groundTruthLabel`` / ``modelPredictionLabel``) + and snake_case shapes the backend may return, and drops malformed entries. + """ + if not isinstance(raw_matches, list): + return None + matches: List[AllowedLabelMatch] = [] + for m in raw_matches: + if not isinstance(m, dict): + continue + gt = m.get("groundTruthLabel") + if gt is None: + gt = m.get("ground_truth_label") + mp = m.get("modelPredictionLabel") + if mp is None: + mp = m.get("model_prediction_label") + if gt is not None and mp is not None: + matches.append( + AllowedLabelMatch( + ground_truth_label=str(gt), + model_prediction_label=str(mp), + ) + ) + return matches + + +@dataclass +class RollupGroup: + """A rollup class: raw labels evaluated together under one class name. + + Rollup groups are the primary label configuration for benchmark + evaluations — each group maps a set of raw ground-truth/prediction + labels onto a single canonical ``class_name``. A label may appear in + at most one group across the configuration. + """ + + class_name: str + labels: List[str] + + def to_api_dict(self) -> Dict[str, Any]: + return {"class_name": self.class_name, "labels": list(self.labels)} + + +def parse_rollup_groups(raw_groups: Any) -> Optional[List[RollupGroup]]: + """Parse a ``rollup_groups`` array from an API payload. + + Accepts both the camelCase (``className``) and snake_case shapes the + backend may return, and drops malformed entries. + """ + if not isinstance(raw_groups, list): + return None + groups: List[RollupGroup] = [] + for g in raw_groups: + if not isinstance(g, dict): + continue + class_name = g.get("className") + if class_name is None: + class_name = g.get("class_name") + labels = g.get("labels") + if class_name is not None and isinstance(labels, list): + groups.append( + RollupGroup( + class_name=str(class_name), + labels=[str(label) for label in labels], + ) + ) + return groups + + @dataclass class EvaluationV2: """An Evaluation V2 run for a model run.""" @@ -66,6 +156,11 @@ class EvaluationV2: allowed_label_matches_id: Optional[str] = None allowed_label_matches: Optional[List[AllowedLabelMatch]] = None allowed_label_matches_name: Optional[str] = None + rollup_groups: Optional[List[RollupGroup]] = None + benchmark_id: Optional[str] = None + slice_id: Optional[str] = None + exclusion_rules: Optional[List[Dict[str, Any]]] = None + exclusion_stats: Optional[Dict[str, Any]] = None _client: Optional["NucleusClient"] = field(repr=False, default=None) @classmethod @@ -74,26 +169,9 @@ def from_json( payload: Dict[str, Any], client: Optional["NucleusClient"] = None, ) -> "EvaluationV2": - raw_matches = payload.get("allowed_label_matches") - matches: Optional[List[AllowedLabelMatch]] = None - if isinstance(raw_matches, list): - matches = [] - for m in raw_matches: - if not isinstance(m, dict): - continue - gt = m.get("groundTruthLabel") - if gt is None: - gt = m.get("ground_truth_label") - mp = m.get("modelPredictionLabel") - if mp is None: - mp = m.get("model_prediction_label") - if gt is not None and mp is not None: - matches.append( - AllowedLabelMatch( - ground_truth_label=str(gt), - model_prediction_label=str(mp), - ) - ) + matches = parse_allowed_label_matches( + payload.get("allowed_label_matches") + ) return cls( id=str(payload["id"]), @@ -109,6 +187,13 @@ def from_json( allowed_label_matches_name=payload.get( "allowed_label_matches_name" ), + rollup_groups=parse_rollup_groups( + _parse_json_field(payload.get("rollup_groups")) + ), + benchmark_id=payload.get("benchmark_id"), + slice_id=payload.get("slice_id"), + exclusion_rules=_parse_json_field(payload.get("exclusion_rules")), + exclusion_stats=_parse_json_field(payload.get("exclusion_stats")), _client=client, ) @@ -170,6 +255,45 @@ def delete(self) -> None: return_raw_response=True, ) + def cancel(self) -> "EvaluationV2": + """Cancel this evaluation if it is still running. + + Stops the evaluation and sets its status to ``cancelled``. Finished + evaluations cannot be cancelled (use :meth:`delete` to archive them). + + Returns: + self, refreshed with the post-cancel status. + """ + if self._client is None: + raise RuntimeError("EvaluationV2 has no client.") + self._client.make_request( + {}, + f"evaluationsV2/{self.id}/cancel", + requests_command=requests.post, + return_raw_response=True, + ) + return self.refresh() + + def retry(self) -> "EvaluationV2": + """Retry this evaluation if it failed. + + Creates a new evaluation for the same model run, reusing this + evaluation's slice, allowed-label-matches, and exclusion rules. Only + ``failed`` evaluations can be retried. + + Returns: + :class:`EvaluationV2`: The newly created (retry) evaluation. + """ + if self._client is None: + raise RuntimeError("EvaluationV2 has no client.") + result = self._client.post({}, f"evaluationsV2/{self.id}/retry") + eval_id = result.get("evaluation_id") + if not eval_id: + raise RuntimeError( + f"Unexpected retry evaluation V2 response: {result}" + ) + return self._client.get_evaluation_v2(str(eval_id)) + def charts( self, iou_threshold: float = 0.5, @@ -190,24 +314,37 @@ def charts( """ if self._client is None: raise RuntimeError("EvaluationV2 has no client.") - params: Dict[str, str] = {} - params["iouThreshold"] = str(iou_threshold) + payload: Dict[str, Any] = {"iouThreshold": iou_threshold} if filters is not None: if isinstance(filters, EvaluationV2FilterArgs): - filt_dict = filters.to_api_filters() + payload["filters"] = filters.to_api_filters() else: - filt_dict = filters - params["filters"] = json.dumps(filt_dict) + payload["filters"] = filters if query: - params["query"] = query - qs = urlencode(params) - route = f"evaluationsV2/{self.id}/charts?{qs}" - data = self._client.get(route) + payload["query"] = query + data = self._client.post(payload, f"evaluationsV2/{self.id}/charts") return EvaluationV2Charts.parse_obj(data) + def filter_schema(self) -> EvaluationV2FilterSchema: + """Return the filter vocabulary for this evaluation. + + Lists the ground-truth labels, prediction labels, and item-metadata + fields (with inferred value types and distinct values) present in this + evaluation's match rows — the valid inputs for + :class:`~nucleus.data_transfer_object.evaluation_v2.EvaluationV2FilterArgs` + when calling :meth:`charts` or :meth:`examples`. + + Returns: + :class:`~nucleus.data_transfer_object.evaluation_v2.EvaluationV2FilterSchema`. + """ + if self._client is None: + raise RuntimeError("EvaluationV2 has no client.") + data = self._client.get(f"evaluationsV2/{self.id}/filterSchema") + return EvaluationV2FilterSchema.parse_obj(data) + def examples( self, - match_type: str, + match_type: Optional[str] = None, limit: int = 50, offset: int = 0, sort_by: Optional[str] = None, @@ -217,14 +354,16 @@ def examples( ] = None, query: Optional[str] = None, ) -> EvaluationV2ExamplesPage: - """Return paginated true-positive, false-positive, or false-negative examples. + """Return paginated match examples, optionally filtered by match type. Parameters: - match_type: ``"TP"``, ``"FP"``, or ``"FN"``. - limit: Page size (default 50). + match_type: ``"TP"``, ``"FP"``, or ``"FN"``. Omit (or ``None``) to + return examples of all match types. + limit: Page size (default 50, max 100). offset: Row offset for pagination. - sort_by: Optional field to sort by. - sort_order: Optional sort direction (e.g. ``"asc"`` or ``"desc"``). + sort_by: Optional field to sort by — one of ``"confidence"``, + ``"iou"``, ``"dataset_item_id"``, ``"gt_area"``. + sort_order: Optional sort direction (``"ASC"`` or ``"DESC"``). filters: Optional filters (:class:`EvaluationV2FilterArgs` or dict). query: Optional query string to narrow results. @@ -234,10 +373,11 @@ def examples( if self._client is None: raise RuntimeError("EvaluationV2 has no client.") payload: Dict[str, Any] = { - "match_type": match_type, "limit": limit, "offset": offset, } + if match_type is not None: + payload["match_type"] = match_type if sort_by is not None: payload["sort_by"] = sort_by if sort_order is not None: diff --git a/nucleus/evaluation_v2_exclusions.py b/nucleus/evaluation_v2_exclusions.py new file mode 100644 index 00000000..88207c01 --- /dev/null +++ b/nucleus/evaluation_v2_exclusions.py @@ -0,0 +1,112 @@ +"""Exclusion rules for Evaluation V2 creation. + +These rules drop items/annotations from an evaluation before metrics are computed. + +The per-rule shape is validated server-side at create time +(``parseEvaluationV2ExclusionRulesWithDiagnostics``), which reports exactly which +rules were rejected and why — so these classes only need to serialize correctly. + +Pass instances (or equivalent plain dicts) to +:meth:`nucleus.NucleusClient.create_benchmark_evaluation_v2` via ``exclusion_rules``:: + + client.create_benchmark_evaluation_v2( + benchmark_id, + model_run_id, + exclusion_rules=[ + BoxAreaExclusionRule(scope="annotation", target="groundTruth", min=1024), + LabelExclusionRule(scope="item", target="prediction", labels=["ignore"]), + MetadataExclusionRule(key="is_dark", op="EQ", value=True), + ], + ) +""" + +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Union + +# String literals are sent as values (not keys), so the server's request-body +# camelcaser preserves them verbatim — emit them exactly as the backend expects. +ExclusionScope = str # "item" | "annotation" +ExclusionTarget = str # "groundTruth" | "prediction" +MetadataOp = str # "EQ" | "IN" | "GT" | "LT" + + +@dataclass +class MetadataExclusionRule: + """Exclude whole items whose item-metadata ``key`` matches ``value`` under ``op``. + + ``scope`` is always ``"item"`` for metadata rules. + """ + + key: str + op: MetadataOp + value: Any + scope: ExclusionScope = "item" + + def to_api_dict(self) -> Dict[str, Any]: + return { + "type": "metadata", + "scope": self.scope, + "key": self.key, + "op": self.op, + "value": self.value, + } + + +@dataclass +class LabelExclusionRule: + """Exclude annotations/predictions (or whole items) carrying any of ``labels``. + + Parameters: + scope: ``"item"`` (drop the whole item if any annotation matches) or + ``"annotation"`` (drop only matching annotations). + target: ``"groundTruth"`` or ``"prediction"`` — which side to filter. + labels: Labels to exclude. + """ + + scope: ExclusionScope + target: ExclusionTarget + labels: List[str] = field(default_factory=list) + + def to_api_dict(self) -> Dict[str, Any]: + return { + "type": "labels", + "scope": self.scope, + "target": self.target, + "labels": list(self.labels), + } + + +@dataclass +class BoxAreaExclusionRule: + """Exclude boxes whose pixel area falls outside ``[min, max]`` (at least one bound required). + + Parameters: + scope: ``"item"`` or ``"annotation"``. + target: ``"groundTruth"`` or ``"prediction"``. + min: Minimum pixel area (inclusive lower bound), or ``None``. + max: Maximum pixel area (inclusive upper bound), or ``None``. + """ + + scope: ExclusionScope + target: ExclusionTarget + min: Optional[float] = None + max: Optional[float] = None + + def to_api_dict(self) -> Dict[str, Any]: + out: Dict[str, Any] = { + "type": "boxArea", + "scope": self.scope, + "target": self.target, + } + if self.min is not None: + out["min"] = self.min + if self.max is not None: + out["max"] = self.max + return out + + +EvaluationV2ExclusionRule = Union[ + MetadataExclusionRule, + LabelExclusionRule, + BoxAreaExclusionRule, +] diff --git a/nucleus/evaluation_v2_preset.py b/nucleus/evaluation_v2_preset.py new file mode 100644 index 00000000..576fc1b2 --- /dev/null +++ b/nucleus/evaluation_v2_preset.py @@ -0,0 +1,128 @@ +"""Evaluation V2 presets — saved, reusable evaluation configurations. + +A preset bundles a ``name`` with a label configuration (``rollup_groups``, +or legacy ``allowed_label_matches``) and ``exclusion_rules`` so the same +configuration can be applied across many evaluations. Presets are private to +the creating user. + +Create and manage presets via :class:`~nucleus.NucleusClient`:: + + preset = client.create_evaluation_v2_preset( + "vehicles", + rollup_groups=[RollupGroup("vehicle", ["car", "truck"])], + exclusion_rules=[LabelExclusionRule(scope="item", target="prediction", labels=["ignore"])], + ) + client.create_benchmark_evaluation_v2(benchmark_id, model_run_id, preset=preset) +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Dict, List, Optional + +from nucleus.evaluation_v2 import ( + AllowedLabelMatch, + RollupGroup, + _parse_json_field, + parse_allowed_label_matches, + parse_rollup_groups, +) + +if TYPE_CHECKING: + from nucleus import NucleusClient + + +# Sentinel distinguishing "argument omitted" from an explicit ``None`` (which, +# for ``exclusion_rules`` on update, means "clear the rules"). +class _Unset: + def __repr__(self) -> str: # pragma: no cover - cosmetic + return "" + + +_UNSET = _Unset() + + +@dataclass +class EvaluationV2Preset: + """A saved Evaluation V2 configuration owned by the current user.""" + + id: str + name: str + rollup_groups: Optional[List[RollupGroup]] = None + allowed_label_matches: Optional[List[AllowedLabelMatch]] = None + exclusion_rules: Optional[List[Dict[str, Any]]] = None + created_by_user_id: Optional[str] = None + created_at: Optional[str] = None + updated_at: Optional[str] = None + deleted_at: Optional[str] = None + _client: Optional["NucleusClient"] = field(repr=False, default=None) + + @classmethod + def from_json( + cls, + payload: Dict[str, Any], + client: Optional["NucleusClient"] = None, + ) -> "EvaluationV2Preset": + return cls( + id=str(payload["id"]), + name=str(payload["name"]), + rollup_groups=parse_rollup_groups( + _parse_json_field( + payload.get("rollup_groups") + if payload.get("rollup_groups") is not None + else payload.get("rollupGroups") + ) + ), + allowed_label_matches=parse_allowed_label_matches( + payload.get("allowed_label_matches") + or payload.get("allowedLabelMatches") + ), + exclusion_rules=_parse_json_field( + payload.get("exclusion_rules") + if payload.get("exclusion_rules") is not None + else payload.get("exclusionRules") + ), + created_by_user_id=payload.get("created_by_user_id"), + created_at=payload.get("created_at"), + updated_at=payload.get("updated_at"), + deleted_at=payload.get("deleted_at"), + _client=client, + ) + + def update( + self, + *, + name: Any = _UNSET, + rollup_groups: Any = _UNSET, + allowed_label_matches: Any = _UNSET, + exclusion_rules: Any = _UNSET, + ) -> "EvaluationV2Preset": + """Update this preset in place. + + Only the arguments you pass are changed. Passing + ``rollup_groups=None`` / ``exclusion_rules=None`` clears that field; + omitting an argument leaves it unchanged. + + Returns: + self, with updated fields. + """ + if self._client is None: + raise RuntimeError( + "EvaluationV2Preset has no client; fetch it via " + "NucleusClient.list_evaluation_v2_presets." + ) + updated = self._client.update_evaluation_v2_preset( + self.id, + name=name, + rollup_groups=rollup_groups, + allowed_label_matches=allowed_label_matches, + exclusion_rules=exclusion_rules, + ) + self.__dict__.update(updated.__dict__) + return self + + def delete(self) -> None: + """Delete this preset.""" + if self._client is None: + raise RuntimeError("EvaluationV2Preset has no client.") + self._client.delete_evaluation_v2_preset(self.id) diff --git a/pyproject.toml b/pyproject.toml index 794811da..e12a2c2f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,7 +25,7 @@ ignore = ["E501", "E741", "E731", "F401"] # Easy ignore for getting it running [tool.poetry] name = "scale-nucleus" -version = "0.18.8" +version = "0.19.0" description = "The official Python client library for Nucleus, the Data Platform for AI" license = "MIT" authors = ["Scale AI Nucleus Team "] diff --git a/tests/test_benchmarks.py b/tests/test_benchmarks.py new file mode 100644 index 00000000..49c5510e --- /dev/null +++ b/tests/test_benchmarks.py @@ -0,0 +1,294 @@ +"""Unit tests for benchmarks and benchmark evaluations (no live API).""" + +from unittest.mock import MagicMock + +import pytest +import requests + +from nucleus import ( + AllowedLabelMatch, + Benchmark, + EvaluationV2Preset, + LabelExclusionRule, + NucleusClient, + RollupGroup, +) + +_BENCHMARK_ROW = { + "benchmark_id": "bm_1", + "name": "city-streets", + "description": "desc", + "metadata": {"team": "av"}, + "created_by_user_id": "u_1", + "created_at": "2026-07-10T00:00:00.000Z", + "item_count": 10, + "dataset_count": 2, +} + +_EVAL_ROW = { + "id": "evalv2_1", + "model_run_id": "run_1", + "dataset_id": "ds_1", + "benchmark_id": "bm_1", + "status": "pending", +} + + +# --------------------------------------------------------------------------- # +# Benchmark CRUD +# --------------------------------------------------------------------------- # +def test_benchmark_from_json_maps_benchmark_id(): + b = Benchmark.from_json( + {**_BENCHMARK_ROW, "skipped_items_without_ground_truth": 3} + ) + assert b.id == "bm_1" + assert b.name == "city-streets" + assert b.item_count == 10 + assert b.dataset_count == 2 + assert b.skipped_items_without_ground_truth == 3 + + +def test_create_benchmark_from_slice(): + client = NucleusClient(api_key="test") + client.connection.post = MagicMock(return_value=dict(_BENCHMARK_ROW)) + benchmark = client.create_benchmark( + "city-streets", description="desc", slice_id="slc_1" + ) + payload, route = client.connection.post.call_args[0] + assert route == "benchmarks" + assert payload == { + "name": "city-streets", + "description": "desc", + "slice_id": "slc_1", + } + assert benchmark.id == "bm_1" + + +def test_create_benchmark_from_item_ids_and_metadata(): + client = NucleusClient(api_key="test") + client.connection.post = MagicMock(return_value=dict(_BENCHMARK_ROW)) + client.create_benchmark( + "city-streets", + metadata={"team": "av"}, + item_ids=["di_1", "di_2"], + ) + payload = client.connection.post.call_args[0][0] + assert payload["item_ids"] == ["di_1", "di_2"] + assert payload["metadata"] == {"team": "av"} + + +def test_create_benchmark_requires_exactly_one_member_source(): + client = NucleusClient(api_key="test") + with pytest.raises(ValueError, match="exactly one"): + client.create_benchmark("b") + with pytest.raises(ValueError, match="exactly one"): + client.create_benchmark("b", slice_id="slc_1", dataset_id="ds_1") + + +def test_list_benchmarks(): + client = NucleusClient(api_key="test") + client.connection.get = MagicMock(return_value=[dict(_BENCHMARK_ROW)]) + benchmarks = client.list_benchmarks() + client.connection.get.assert_called_once_with("benchmarks") + assert len(benchmarks) == 1 + assert benchmarks[0].id == "bm_1" + + +def test_get_benchmark(): + client = NucleusClient(api_key="test") + client.connection.get = MagicMock(return_value=dict(_BENCHMARK_ROW)) + benchmark = client.get_benchmark("bm_1") + client.connection.get.assert_called_once_with("benchmarks/bm_1") + assert benchmark.id == "bm_1" + + +def test_update_benchmark_sends_only_provided_fields(): + client = NucleusClient(api_key="test") + client.connection.patch = MagicMock( + return_value={**_BENCHMARK_ROW, "name": "renamed"} + ) + benchmark = client.update_benchmark("bm_1", name="renamed") + payload, route = client.connection.patch.call_args[0] + assert route == "benchmarks/bm_1" + assert payload == {"name": "renamed"} + assert benchmark.name == "renamed" + + +def test_delete_benchmark(): + client = NucleusClient(api_key="test") + client.connection.make_request = MagicMock(return_value=MagicMock()) + client.delete_benchmark("bm_1") + args = client.connection.make_request.call_args[0] + assert args[1] == "benchmarks/bm_1" + assert args[2] is requests.delete + + +def test_list_benchmark_items_paging_query_string(): + client = NucleusClient(api_key="test") + client.connection.get = MagicMock( + return_value={"item_ids": ["di_1", "di_2"], "total": 10} + ) + page = client.list_benchmark_items("bm_1", limit=2, offset=4) + client.connection.get.assert_called_once_with( + "benchmarks/bm_1/items?limit=2&offset=4" + ) + assert page.item_ids == ["di_1", "di_2"] + assert page.total == 10 + + +def test_list_benchmark_items_no_paging_params(): + client = NucleusClient(api_key="test") + client.connection.get = MagicMock( + return_value={"item_ids": [], "total": 0} + ) + client.list_benchmark_items("bm_1") + client.connection.get.assert_called_once_with("benchmarks/bm_1/items") + + +# --------------------------------------------------------------------------- # +# Benchmark instance methods +# --------------------------------------------------------------------------- # +def test_benchmark_instance_methods_delegate_to_client(): + client = MagicMock(spec=NucleusClient) + benchmark = Benchmark(id="bm_1", name="b", _client=client) + + client.update_benchmark.return_value = Benchmark( + id="bm_1", name="renamed", _client=client + ) + benchmark.update(name="renamed") + client.update_benchmark.assert_called_once_with( + "bm_1", name="renamed", description=None, metadata=None + ) + assert benchmark.name == "renamed" + + benchmark.delete() + client.delete_benchmark.assert_called_once_with("bm_1") + + benchmark.items(limit=5) + client.list_benchmark_items.assert_called_once_with( + "bm_1", limit=5, offset=None + ) + + benchmark.create_evaluation_v2("run_1", name="e") + _, kwargs = client.create_benchmark_evaluation_v2.call_args + assert client.create_benchmark_evaluation_v2.call_args[0] == ( + "bm_1", + "run_1", + ) + assert kwargs["name"] == "e" + + +def test_benchmark_without_client_raises(): + benchmark = Benchmark(id="bm_1", name="b") + with pytest.raises(RuntimeError, match="no client"): + benchmark.refresh() + + +# --------------------------------------------------------------------------- # +# Benchmark evaluation create +# --------------------------------------------------------------------------- # +def _mock_create_eval(client): + client.connection.make_request = MagicMock( + return_value={"evaluation_id": "evalv2_1"} + ) + client.connection.get = MagicMock(return_value=dict(_EVAL_ROW)) + + +def test_create_benchmark_evaluation_v2_with_rollup_groups(): + client = NucleusClient(api_key="test") + _mock_create_eval(client) + evaluation = client.create_benchmark_evaluation_v2( + "bm_1", + "run_1", + name="eval", + rollup_groups=[RollupGroup("vehicle", ["car", "truck"])], + exclusion_rules=[ + LabelExclusionRule( + scope="item", target="prediction", labels=["ignore"] + ) + ], + ) + args = client.connection.make_request.call_args[0] + payload, route = args[0], args[1] + assert route == "benchmarks/bm_1/evaluationsV2" + assert payload["model_run_id"] == "run_1" + assert payload["name"] == "eval" + assert payload["rollupGroups"] == [ + {"class_name": "vehicle", "labels": ["car", "truck"]} + ] + assert "onlyItemsWithPredictions" not in payload + client.connection.get.assert_called_once_with("evaluationsV2/evalv2_1") + assert evaluation.id == "evalv2_1" + assert evaluation.benchmark_id == "bm_1" + + +def test_create_benchmark_evaluation_v2_label_config_mutual_exclusion(): + client = NucleusClient(api_key="test") + with pytest.raises(ValueError, match="at most one"): + client.create_benchmark_evaluation_v2( + "bm_1", + "run_1", + rollup_groups=[RollupGroup("vehicle", ["car"])], + allowed_label_matches=[AllowedLabelMatch("car", "vehicle")], + ) + with pytest.raises(ValueError, match="at most one"): + client.create_benchmark_evaluation_v2( + "bm_1", + "run_1", + rollup_groups=[RollupGroup("vehicle", ["car"])], + allowed_label_matches_id="alm_1", + ) + + +def test_create_benchmark_evaluation_v2_preset_seeds_rollup_groups(): + client = NucleusClient(api_key="test") + _mock_create_eval(client) + preset = EvaluationV2Preset( + id="prev_1", + name="p", + rollup_groups=[RollupGroup("vehicle", ["car"])], + exclusion_rules=[{"type": "labels", "scope": "item"}], + ) + client.create_benchmark_evaluation_v2("bm_1", "run_1", preset=preset) + payload = client.connection.make_request.call_args[0][0] + assert payload["rollupGroups"] == [ + {"class_name": "vehicle", "labels": ["car"]} + ] + assert payload["exclusionRules"] == [{"type": "labels", "scope": "item"}] + assert "allowed_label_matches" not in payload + + +def test_create_benchmark_evaluation_v2_preset_seeds_legacy_matches(): + client = NucleusClient(api_key="test") + _mock_create_eval(client) + preset = EvaluationV2Preset( + id="prev_1", + name="p", + allowed_label_matches=[AllowedLabelMatch("car", "vehicle")], + ) + client.create_benchmark_evaluation_v2("bm_1", "run_1", preset=preset) + payload = client.connection.make_request.call_args[0][0] + assert payload["allowed_label_matches"] == [ + {"ground_truth_label": "car", "model_prediction_label": "vehicle"} + ] + assert "rollupGroups" not in payload + + +def test_create_benchmark_evaluation_v2_explicit_args_override_preset(): + client = NucleusClient(api_key="test") + _mock_create_eval(client) + preset = EvaluationV2Preset( + id="prev_1", + name="p", + rollup_groups=[RollupGroup("vehicle", ["car"])], + ) + client.create_benchmark_evaluation_v2( + "bm_1", + "run_1", + rollup_groups=[RollupGroup("person", ["ped"])], + preset=preset, + ) + payload = client.connection.make_request.call_args[0][0] + assert payload["rollupGroups"] == [ + {"class_name": "person", "labels": ["ped"]} + ] diff --git a/tests/test_evaluation_v2.py b/tests/test_evaluation_v2.py index 829e34a2..c4d91ebe 100644 --- a/tests/test_evaluation_v2.py +++ b/tests/test_evaluation_v2.py @@ -5,7 +5,14 @@ import pytest import requests -from nucleus import AllowedLabelMatch, EvaluationV2, NucleusClient +from nucleus import ( + AllowedLabelMatch, + BoxAreaExclusionRule, + EvaluationV2, + LabelExclusionRule, + MetadataExclusionRule, + NucleusClient, +) from nucleus.data_transfer_object.evaluation_v2 import ( EvaluationV2Charts, EvaluationV2FilterArgs, @@ -15,6 +22,28 @@ ) +def _charts_response(): + return { + "mapSummary": {"mapAt50": 0.1, "mapAt75": 0.2, "mapAt5095": 0.15}, + "perClassAp": [], + "confusionMatrix": [], + "scoreHistogram": [], + "computedIouRanges": [], + "totalCounts": {"tp": 0, "fp": 0, "fn": 0, "predsWithConfidence": 0}, + "apBySize": {"small": None, "medium": None, "large": None}, + "prCurve": [], + "tideAttribution": { + "truePositive": 0, + "localization": 0, + "classification": 0, + "both": 0, + "duplicate": 0, + "background": 0, + "missed": 0, + }, + } + + def test_evaluation_v2_filter_args_to_api_filters(): filters = EvaluationV2FilterArgs( confidence_range=RangeNum(min=0.1, max=0.9), @@ -104,57 +133,59 @@ def test_list_evaluations_v2_invalid_response(): client.list_evaluations_v2("run_1") -def test_create_evaluation_v2_then_get(): - client = NucleusClient(api_key="test") - client.connection.make_request = MagicMock( - return_value={ - "evaluation_id": "evalv2_new", - "status": "pending", - "workflow_id": "w", - } +def test_evaluation_v2_filter_args_gt_area_and_slices(): + filters = EvaluationV2FilterArgs( + gt_area_range=RangeNum(min=1024, max=9216), + slice_ids=["slc_a"], ) - client.connection.get = MagicMock( - return_value={ - "id": "evalv2_new", + assert filters.to_api_filters() == { + "gtAreaRange": {"min": 1024.0, "max": 9216.0}, + "sliceIds": ["slc_a"], + } + + +def test_evaluation_v2_from_json_slice_and_exclusions(): + # exclusion_rules as a JSON string (raw jsonb), exclusion_stats as a dict. + ev = EvaluationV2.from_json( + { + "id": "evalv2_1", "model_run_id": "run_1", "dataset_id": "ds_1", - "status": "pending", + "status": "succeeded", + "slice_id": "slc_x", + "exclusion_rules": '[{"type":"labels","scope":"item","target":"prediction","labels":["ignore"]}]', + "exclusion_stats": {"totals": {"itemsDropped": 3}}, } ) + assert ev.slice_id == "slc_x" + assert ev.exclusion_rules == [ + { + "type": "labels", + "scope": "item", + "target": "prediction", + "labels": ["ignore"], + } + ] + assert ev.exclusion_stats == {"totals": {"itemsDropped": 3}} - ev = client.create_evaluation_v2( - "run_1", - name="n1", - allowed_label_matches=[ - AllowedLabelMatch("gt", "pred"), - ], + +def test_evaluation_v2_from_json_exclusions_absent(): + ev = EvaluationV2.from_json( + { + "id": "evalv2_1", + "model_run_id": "run_1", + "dataset_id": "ds_1", + "status": "succeeded", + } ) - assert ev.id == "evalv2_new" - client.connection.make_request.assert_called_once() - client.connection.get.assert_called_once_with("evaluationsV2/evalv2_new") + assert ev.slice_id is None + assert ev.exclusion_rules is None + assert ev.exclusion_stats is None -def test_charts_get_query_string(): +def test_charts_post_body(): client = MagicMock(spec=NucleusClient) - client.get.return_value = { - "mapSummary": {"mapAt50": 0.1, "mapAt75": 0.2, "mapAt5095": 0.15}, - "perClassAp": [], - "confusionMatrix": [], - "scoreHistogram": [], - "computedIouRanges": [], - "totalCounts": {"tp": 0, "fp": 0, "fn": 0, "predsWithConfidence": 0}, - "apBySize": {"small": None, "medium": None, "large": None}, - "prCurve": [], - "tideAttribution": { - "truePositive": 0, - "localization": 0, - "classification": 0, - "both": 0, - "duplicate": 0, - "background": 0, - "missed": 0, - }, - } + client.post.return_value = _charts_response() ev = EvaluationV2( id="evalv2_1", model_run_id="run_1", @@ -164,9 +195,35 @@ def test_charts_get_query_string(): ) charts = ev.charts(iou_threshold=0.5) assert isinstance(charts, EvaluationV2Charts) - call_route = client.get.call_args[0][0] - assert "evaluationsV2/evalv2_1/charts" in call_route - assert "iouThreshold=0.5" in call_route + client.post.assert_called_once() + payload, route = client.post.call_args[0] + assert route == "evaluationsV2/evalv2_1/charts" + assert payload == {"iouThreshold": 0.5} + + +def test_charts_with_filter_args(): + client = MagicMock(spec=NucleusClient) + client.post.return_value = _charts_response() + ev = EvaluationV2( + id="evalv2_1", + model_run_id="run_1", + dataset_id="ds_1", + status="succeeded", + _client=client, + ) + filters = EvaluationV2FilterArgs( + gt_area_range=RangeNum(min=1024), + slice_ids=["slc_a", "slc_b"], + ) + ev.charts(iou_threshold=0.75, filters=filters, query="dog") + payload, route = client.post.call_args[0] + assert route == "evaluationsV2/evalv2_1/charts" + assert payload["iouThreshold"] == 0.75 + assert payload["query"] == "dog" + assert payload["filters"] == { + "gtAreaRange": {"min": 1024.0}, + "sliceIds": ["slc_a", "slc_b"], + } def test_examples_post_body(): @@ -263,3 +320,65 @@ def test_delete_success(status_code): assert cargs[0][1] == "evaluationsV2/evalv2_1" assert cargs[0][2] is requests.delete assert cargs[0][3] is True + + +# --------------------------------------------------------------------------- # +# Rollup groups and benchmark_id +# --------------------------------------------------------------------------- # +def test_rollup_group_to_api_dict(): + from nucleus import RollupGroup + + g = RollupGroup(class_name="vehicle", labels=["car", "truck"]) + assert g.to_api_dict() == { + "class_name": "vehicle", + "labels": ["car", "truck"], + } + + +def test_parse_rollup_groups_both_casings_and_malformed(): + from nucleus.evaluation_v2 import parse_rollup_groups + + groups = parse_rollup_groups( + [ + {"className": "vehicle", "labels": ["car"]}, + {"class_name": "person", "labels": ["ped"]}, + {"labels": ["orphan"]}, # missing class name → dropped + "not-a-dict", # malformed → dropped + ] + ) + assert groups is not None + assert [(g.class_name, g.labels) for g in groups] == [ + ("vehicle", ["car"]), + ("person", ["ped"]), + ] + assert parse_rollup_groups(None) is None + assert parse_rollup_groups("nope") is None + + +def test_evaluation_v2_from_json_benchmark_id_and_rollup_groups(): + ev = EvaluationV2.from_json( + { + "id": "evalv2_1", + "model_run_id": "run_1", + "dataset_id": "ds_1", + "status": "pending", + "benchmark_id": "bm_1", + "rollup_groups": [{"className": "vehicle", "labels": ["car"]}], + } + ) + assert ev.benchmark_id == "bm_1" + assert ev.rollup_groups is not None + assert ev.rollup_groups[0].class_name == "vehicle" + + +def test_evaluation_v2_from_json_benchmark_fields_absent(): + ev = EvaluationV2.from_json( + { + "id": "evalv2_1", + "model_run_id": "run_1", + "dataset_id": "ds_1", + "status": "pending", + } + ) + assert ev.benchmark_id is None + assert ev.rollup_groups is None diff --git a/tests/test_evaluation_v2_presets.py b/tests/test_evaluation_v2_presets.py new file mode 100644 index 00000000..35270a9d --- /dev/null +++ b/tests/test_evaluation_v2_presets.py @@ -0,0 +1,304 @@ +"""Unit tests for Evaluation V2 presets, batch create, cancel/retry, and +label-schema discovery (no live API).""" + +from unittest.mock import MagicMock + +import requests + +from nucleus import ( + AllowedLabelMatch, + EvaluationV2, + EvaluationV2Preset, + LabelExclusionRule, + NucleusClient, +) +from nucleus.dataset import Dataset + + +# --------------------------------------------------------------------------- # +# Preset CRUD +# --------------------------------------------------------------------------- # +def test_list_evaluation_v2_presets(): + client = NucleusClient(api_key="test") + client.connection.get = MagicMock( + return_value=[ + { + "id": "prev_1", + "name": "vehicles", + "allowed_label_matches": [ + { + "groundTruthLabel": "car", + "modelPredictionLabel": "vehicle", + } + ], + "exclusion_rules": None, + "created_by_user_id": "u_1", + } + ] + ) + presets = client.list_evaluation_v2_presets() + client.connection.get.assert_called_once_with("evaluationV2Presets") + assert len(presets) == 1 + assert presets[0].id == "prev_1" + assert presets[0].name == "vehicles" + assert presets[0].allowed_label_matches[0] == AllowedLabelMatch( + ground_truth_label="car", model_prediction_label="vehicle" + ) + + +def test_create_evaluation_v2_preset_payload(): + client = NucleusClient(api_key="test") + client.connection.post = MagicMock( + return_value={ + "id": "prev_1", + "name": "vehicles", + "allowed_label_matches": [], + "exclusion_rules": None, + } + ) + preset = client.create_evaluation_v2_preset( + "vehicles", + allowed_label_matches=[AllowedLabelMatch("car", "vehicle")], + exclusion_rules=[ + LabelExclusionRule( + scope="item", target="prediction", labels=["ignore"] + ) + ], + ) + payload, route = client.connection.post.call_args[0] + assert route == "evaluationV2Presets" + assert payload["name"] == "vehicles" + assert payload["allowedLabelMatches"] == [ + {"ground_truth_label": "car", "model_prediction_label": "vehicle"} + ] + assert payload["exclusionRules"] == [ + { + "type": "labels", + "scope": "item", + "target": "prediction", + "labels": ["ignore"], + } + ] + assert preset.id == "prev_1" + + +def test_update_evaluation_v2_preset_name_only_omits_other_fields(): + client = NucleusClient(api_key="test") + client.connection.patch = MagicMock( + return_value={"id": "prev_1", "name": "renamed"} + ) + client.update_evaluation_v2_preset("prev_1", name="renamed") + payload, route = client.connection.patch.call_args[0] + assert route == "evaluationV2Presets/prev_1" + # Only the provided field is sent; matches/rules untouched. + assert payload == {"name": "renamed"} + + +def test_update_evaluation_v2_preset_clear_rules_sends_null(): + client = NucleusClient(api_key="test") + client.connection.patch = MagicMock( + return_value={"id": "prev_1", "name": "p"} + ) + client.update_evaluation_v2_preset("prev_1", exclusion_rules=None) + payload = client.connection.patch.call_args[0][0] + # Explicit None clears the rules (distinct from "leave unchanged"). + assert payload == {"exclusionRules": None} + + +def test_delete_evaluation_v2_preset(): + client = NucleusClient(api_key="test") + client.connection.make_request = MagicMock(return_value=MagicMock()) + client.delete_evaluation_v2_preset("prev_1") + # NucleusClient.make_request forwards args positionally to the connection: + # (payload, route, requests_command, return_raw_response). + args = client.connection.make_request.call_args[0] + assert args[1] == "evaluationV2Presets/prev_1" + assert args[2] is requests.delete + + +def test_preset_instance_update_and_delete_delegate_to_client(): + client = MagicMock(spec=NucleusClient) + preset = EvaluationV2Preset(id="prev_1", name="p", _client=client) + client.update_evaluation_v2_preset.return_value = EvaluationV2Preset( + id="prev_1", name="renamed", _client=client + ) + preset.update(name="renamed") + assert preset.name == "renamed" + preset.delete() + client.delete_evaluation_v2_preset.assert_called_once_with("prev_1") + + +# --------------------------------------------------------------------------- # +# Apply preset + only_items_with_predictions on create +# --------------------------------------------------------------------------- # +def _stub_create(client): + client.connection.make_request = MagicMock( + return_value={"evaluation_id": "evalv2_new"} + ) + client.connection.get = MagicMock( + return_value={ + "id": "evalv2_new", + "model_run_id": "run_1", + "dataset_id": "ds_1", + "status": "pending", + } + ) + + +# --------------------------------------------------------------------------- # +# Batch create +# --------------------------------------------------------------------------- # +# --------------------------------------------------------------------------- # +# Cancel / retry +# --------------------------------------------------------------------------- # +def _eval(client, status="computing"): + return EvaluationV2( + id="evalv2_1", + model_run_id="run_1", + dataset_id="ds_1", + status=status, + _client=client, + ) + + +def test_evaluation_cancel_posts_and_refreshes(): + client = MagicMock(spec=NucleusClient) + client.get.return_value = { + "id": "evalv2_1", + "model_run_id": "run_1", + "dataset_id": "ds_1", + "status": "cancelled", + } + ev = _eval(client) + ev.cancel() + args, kwargs = client.make_request.call_args + assert args[1] == "evaluationsV2/evalv2_1/cancel" + assert kwargs["requests_command"] is requests.post + assert ev.status == "cancelled" + + +def test_evaluation_retry_resolves_new_evaluation(): + client = MagicMock(spec=NucleusClient) + client.post.return_value = {"evaluation_id": "evalv2_retry"} + client.get_evaluation_v2.return_value = EvaluationV2( + id="evalv2_retry", + model_run_id="run_1", + dataset_id="ds_1", + status="pending", + _client=client, + ) + ev = _eval(client, status="failed") + new_eval = ev.retry() + _, route = client.post.call_args[0] + assert route == "evaluationsV2/evalv2_1/retry" + assert new_eval.id == "evalv2_retry" + client.get_evaluation_v2.assert_called_once_with("evalv2_retry") + + +# --------------------------------------------------------------------------- # +# Examples optional match_type +# --------------------------------------------------------------------------- # +def test_examples_match_type_optional(): + client = MagicMock(spec=NucleusClient) + client.post.return_value = {"rows": [], "total": 0} + ev = _eval(client, status="succeeded") + + ev.examples() + payload = client.post.call_args[0][0] + assert "match_type" not in payload + + ev.examples(match_type="FP") + payload2 = client.post.call_args[0][0] + assert payload2["match_type"] == "FP" + + +# --------------------------------------------------------------------------- # +# Label schema discovery +# --------------------------------------------------------------------------- # +def test_dataset_evaluation_label_schema(): + client = NucleusClient(api_key="test") + client.connection.make_request = MagicMock( + return_value={"gt_labels": ["car"], "prediction_labels": ["vehicle"]} + ) + dataset = Dataset("ds_1", client) + out = dataset.evaluation_label_schema() + assert out == {"gt_labels": ["car"], "prediction_labels": ["vehicle"]} + args = client.connection.make_request.call_args[0] + assert args[1] == "dataset/ds_1/labelSchema" + assert args[2] is requests.get + + +# --------------------------------------------------------------------------- # +# Preset rollup groups +# --------------------------------------------------------------------------- # +def test_create_evaluation_v2_preset_with_rollup_groups(): + from nucleus import RollupGroup + + client = NucleusClient(api_key="test") + client.connection.post = MagicMock( + return_value={ + "id": "prev_1", + "name": "vehicles", + "rollup_groups": [{"className": "vehicle", "labels": ["car"]}], + } + ) + preset = client.create_evaluation_v2_preset( + "vehicles", + rollup_groups=[RollupGroup("vehicle", ["car"])], + ) + payload, route = client.connection.post.call_args[0] + assert route == "evaluationV2Presets" + assert payload["rollupGroups"] == [ + {"class_name": "vehicle", "labels": ["car"]} + ] + assert preset.rollup_groups is not None + assert preset.rollup_groups[0].class_name == "vehicle" + + +def test_create_evaluation_v2_preset_rollup_and_matches_mutually_exclusive(): + from nucleus import RollupGroup + + client = NucleusClient(api_key="test") + try: + client.create_evaluation_v2_preset( + "p", + rollup_groups=[RollupGroup("vehicle", ["car"])], + allowed_label_matches=[AllowedLabelMatch("car", "vehicle")], + ) + raise AssertionError("expected ValueError") + except ValueError as e: + assert "cannot both be set" in str(e) + + +def test_update_evaluation_v2_preset_rollup_groups_and_clear(): + from nucleus import RollupGroup + + client = NucleusClient(api_key="test") + client.connection.patch = MagicMock( + return_value={"id": "prev_1", "name": "p"} + ) + client.update_evaluation_v2_preset( + "prev_1", rollup_groups=[RollupGroup("vehicle", ["car"])] + ) + payload = client.connection.patch.call_args[0][0] + assert payload == { + "rollupGroups": [{"class_name": "vehicle", "labels": ["car"]}] + } + + client.update_evaluation_v2_preset("prev_1", rollup_groups=None) + payload2 = client.connection.patch.call_args[0][0] + # Explicit None clears the rollup groups (distinct from "leave unchanged"). + assert payload2 == {"rollupGroups": None} + + +def test_preset_from_json_parses_rollup_groups_both_casings(): + for key in ("rollup_groups", "rollupGroups"): + preset = EvaluationV2Preset.from_json( + { + "id": "prev_1", + "name": "p", + key: [{"className": "vehicle", "labels": ["car", "truck"]}], + } + ) + assert preset.rollup_groups is not None + assert preset.rollup_groups[0].labels == ["car", "truck"] diff --git a/tests/test_leaderboard.py b/tests/test_leaderboard.py new file mode 100644 index 00000000..538e6255 --- /dev/null +++ b/tests/test_leaderboard.py @@ -0,0 +1,138 @@ +"""Unit tests for benchmark leaderboard and filter-schema client methods +(no live API). + +The backing REST endpoints (``POST /nucleus/leaderboard/ranking``, +``POST /nucleus/leaderboard/f1Curve``, +``GET /nucleus/evaluationsV2/:id/filterSchema``) ship with the scaleapi +server; these tests only cover request/response wiring. +""" + +from unittest.mock import MagicMock + +import pytest + +from nucleus import EvaluationV2, NucleusClient +from nucleus.data_transfer_object.evaluation_v2 import ( + EvaluationV2FilterSchema, + LeaderboardF1CurveEntry, + LeaderboardRankingEntry, +) + +_RANKING_ROW = { + "evaluation_id": "evalv2_1", + "evaluation_name": "eval", + "model_run_id": "run_1", + "model_run_name": "run", + "model_id": "prj_1", + "model_name": "model", + "model_version_major": 1, + "model_version_minor": 2, + "model_version_label": "1.2", + "parent_model_project_id": None, + "dataset_id": "ds_1", + "dataset_name": "dataset", + "score": 0.42, + "rank": 1, +} + +_F1_ROW = { + "evaluation_id": "evalv2_1", + "model_run_id": "run_1", + "model_name": "model", + "best_f1": 0.8, + "points": [ + {"confidence_threshold": 0.25, "score": 0.7}, + {"confidence_threshold": 0.5, "score": 0.8}, + ], + "rank": 1, +} + +_FILTER_SCHEMA = { + "gt_labels": ["car", "truck"], + "pred_labels": ["car"], + "metadata_fields": [ + {"key": "weather", "value_type": "string", "values": ["rain", "sun"]} + ], +} + + +def test_leaderboard_ranking_payload_and_parsing(): + client = NucleusClient(api_key="test") + client.connection.post = MagicMock(return_value=[dict(_RANKING_ROW)]) + rows = client.leaderboard_ranking( + "MAP_50", + ["bm_1", "bm_2"], + confidence_threshold=0.5, + model_ids=["prj_1"], + scope="mine", + collapse="allRuns", + ) + payload, route = client.connection.post.call_args[0] + assert route == "leaderboard/ranking" + assert payload == { + "metric_type": "MAP_50", + "benchmark_ids": ["bm_1", "bm_2"], + "confidence_threshold": 0.5, + "model_ids": ["prj_1"], + "scope": "mine", + "collapse": "allRuns", + } + assert rows == [LeaderboardRankingEntry.parse_obj(_RANKING_ROW)] + assert rows[0].score == 0.42 + assert rows[0].rank == 1 + + +def test_leaderboard_ranking_minimal_payload(): + client = NucleusClient(api_key="test") + client.connection.post = MagicMock(return_value=[]) + client.leaderboard_ranking("F1", ["bm_1"]) + payload = client.connection.post.call_args[0][0] + assert payload == {"metric_type": "F1", "benchmark_ids": ["bm_1"]} + + +def test_leaderboard_ranking_invalid_response(): + client = NucleusClient(api_key="test") + client.connection.post = MagicMock(return_value={"error": "nope"}) + with pytest.raises(RuntimeError, match="Unexpected leaderboard ranking"): + client.leaderboard_ranking("MAP_50", ["bm_1"]) + + +def test_leaderboard_f1_curve_payload_and_parsing(): + client = NucleusClient(api_key="test") + client.connection.post = MagicMock(return_value=[dict(_F1_ROW)]) + rows = client.leaderboard_f1_curve(["bm_1"], top_n=3) + payload, route = client.connection.post.call_args[0] + assert route == "leaderboard/f1Curve" + assert payload == {"benchmark_ids": ["bm_1"], "top_n": 3} + assert rows == [LeaderboardF1CurveEntry.parse_obj(_F1_ROW)] + assert rows[0].best_f1 == 0.8 + assert rows[0].points[1].confidence_threshold == 0.5 + + +def test_get_evaluation_v2_filter_schema(): + client = NucleusClient(api_key="test") + client.connection.get = MagicMock(return_value=dict(_FILTER_SCHEMA)) + schema = client.get_evaluation_v2_filter_schema("evalv2_1") + client.connection.get.assert_called_once_with( + "evaluationsV2/evalv2_1/filterSchema" + ) + assert schema == EvaluationV2FilterSchema.parse_obj(_FILTER_SCHEMA) + assert schema.gt_labels == ["car", "truck"] + assert schema.metadata_fields[0].key == "weather" + + +def test_evaluation_filter_schema_instance_method(): + client = NucleusClient(api_key="test") + client.connection.get = MagicMock(return_value=dict(_FILTER_SCHEMA)) + evaluation = EvaluationV2( + id="evalv2_1", + model_run_id="run_1", + dataset_id="ds_1", + status="succeeded", + _client=client, + ) + schema = evaluation.filter_schema() + client.connection.get.assert_called_once_with( + "evaluationsV2/evalv2_1/filterSchema" + ) + assert schema.pred_labels == ["car"]