[DE-8678] Run-free ("model v2") predictions on Model - #479
Draft
luke-e-schaefer wants to merge 3 commits into
Draft
[DE-8678] Run-free ("model v2") predictions on Model#479luke-e-schaefer wants to merge 3 commits into
luke-e-schaefer wants to merge 3 commits into
Conversation
Introduce a run-free prediction concept where predictions are tied directly
to a Model as (model, dataset_item) -> prediction, with no ModelRun or
Dataset. Purely additive: all existing model-run prediction paths are
unchanged.
- Model.upload_predictions(...) — upsert predictions onto model/{id}/predictions
(sync + async), reusing the PredictionUploader batching machinery.
- Model.predictions_loc / predictions_refloc / predictions_iloc — model-scoped reads.
- Model.copy_predictions_from_run(model_run_id) — backfill from a v1 run (AsyncJob).
- create_benchmark_evaluation_v2 gains an optional model_id anchor (accepts a
prj_* id or a Model) as an alternative to model_run_id; exactly one required.
- EvaluationV2 gains an optional model_id field; model_run_id now optional.
- serialize_and_write_to_presigned_url gains route_prefix for the model route.
- Docs + CHANGELOG; minor version bump to 0.22.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nd [DE-8678]
The live scaleapi backend for DE-8678 is synchronous-only for these routes:
- Model.upload_predictions: the model route has no async/signed-URL endpoint
(?async=1 returns HTTP 400). Remove the assumed signed-URL async flow and
raise NotImplementedError when asynchronous=True; keep the sync path as-is.
Revert the now-unused route_prefix param added to
serialize_and_write_to_presigned_url in nucleus/utils.py.
- Model.copy_predictions_from_run: the backend runs synchronously and returns
{model_id, model_run_ids, predictions_copied, predictions_skipped_unsupported}.
Return that dict directly (drop the AsyncJob wrapping and the unused
asynchronous param); note it's synchronous in the docstring.
CHANGELOG updated to match; still additive, still v0.22.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…onse [DE-8678]
Model.predictions_loc / predictions_refloc / predictions_iloc were shipped
non-functional: the run-free read endpoints return a flat {"predictions": [...]}
list (each element carrying its own "type"), but format_prediction_response only
understood the legacy type-keyed {"annotations": {"box": [...]}} shape. It fell
through to the "an error occurred" branch and returned the raw payload unparsed,
so these reads yielded the raw dict instead of the documented
{"box": [...], "polygon": [...], "cuboid": [...]}.
Add a flat-list branch that groups predictions by their per-element "type" into
that same shape. Legacy type-keyed reads and the error/empty case are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a run-free "model v2" prediction concept where predictions are
(model, dataset_item) -> predictionwith noModelRunand noDataset. Every existing model-run prediction path keeps working unchanged.Linear: DE-8678
What's added
nucleus/model.py— newModelmethods:Model.upload_predictions(predictions, update=False, batch_size=5000, remote_files_per_upload_request=20, local_files_per_upload_request=10)— upserts predictions ontomodel/{id}/predictions, reusing the existingPredictionUploaderbatching machinery.box/polygon/cuboidonly (server-enforced). Synchronous only:asynchronous=TrueraisesNotImplementedError(the backend has no async/signed-URL route for this path).Model.predictions_loc(dataset_item_id)→model/{id}/predictions/loc/{datasetItemId}Model.predictions_refloc(reference_id)→model/{id}/predictions/refloc/{referenceId}Model.predictions_iloc(i)→model/{id}/predictions/iloc/{i}(all return the same
format_prediction_responseshape as the dataset-scoped equivalents)Model.copy_predictions_from_run(model_run_id)→POST model/{id}/predictions/copyFromRunwith{"model_run_id": ...}. Runs synchronously and returns the dict{model_id, model_run_ids, predictions_copied, predictions_skipped_unsupported}.nucleus/__init__.py— eval model anchor:create_benchmark_evaluation_v2(...)gains an optionalmodel_id(aprj_*id or aModel) as an alternative anchor tomodel_run_id.model_run_idis now optional but fully preserved. Exactly one of the two must be provided (elseValueError). Model-anchored payload sendsmodel_idinstead ofmodel_run_id.nucleus/evaluation_v2.py:EvaluationV2gains an optionalmodel_idfield;model_run_idis now optional and parsed defensively.Docs / version:
model_run.pymodule docstring to point at the run-free path and note the model-anchored eval ignores runs;CHANGELOG.mdentry; minor version bump 0.21.2 → 0.22.0.Notes
black/isort/ruff(E,F per repo config) clean on the added code, package imports and new API surface verified.🤖 Generated with Claude Code