Skip to content

[core][python] Add asynchronous prediction REST APIs - #11012

Open
XiaoHongbo-Hope wants to merge 41 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/prediction-job-api
Open

XiaoHongbo-Hope wants to merge 41 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/prediction-job-api

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Add provider-neutral asynchronous predictions for registered Models. The Java and Python REST clients are backed by an executable local reference provider that reads the recorded source snapshot, writes a managed Paimon result table, and handles cancellation.

API

  • table.predict() submits a prediction and returns PredictionWait.
  • result(timeout) polls until the deadline. The deadline is checked between REST requests and does not cancel the server-side prediction.
  • REST exposes Catalog-scoped submit, list, get, and cancel operations under /v1/{prefix}/predictions.
  • Submission records the source snapshot and resolves the registered Model. Execution reads that snapshot and fails if it is unavailable.
  • SDKs generate idempotencyToken by default; callers may supply one for retries. x-request-id remains a tracing identifier.
  • Time travel uses Paimon options scan.snapshot-id or scan.tag-name.
  • output is optional. The default managed table contains source identity columns, response, and error.

Strict request cancellation and an end-to-end HTTP deadline are generic client concerns and are out of scope.

Lifecycle

  • state: PENDING, RUNNING, TERMINATING, or TERMINATED.
  • terminationDetails contains the result code and optional message.
  • startTime and endTime describe execution time.
  • Cancellation is asynchronous and may return TERMINATING.

Industry references

Reference Adopted design
Databricks Runs Asynchronous resource lifecycle, execution timestamps, termination details, cancellation, and caller-reusable idempotency tokens
Databricks AI Function Task Task instructions and managed result columns such as response and error
Databricks Delta time travel Reproducible execution against a selected source version
BigQuery and Snowflake Idempotent request retry patterns
SageMaker Batch Transform Asynchronous batch prediction resource

Paimon-specific choices are the Catalog-scoped resource, structured Model identifier, table source, and snapshot/tag options.

Reference integration

The local REST Catalog reference provider executes a deterministic prediction against a real Paimon source table. Its integration test verifies:

  • submission returns the recorded snapshot;
  • data committed after submission is excluded from execution;
  • results are written to a managed Paimon table and read back through REST Catalog;
  • cancellation reaches TERMINATED/CANCELED without publishing an output table.

Tests

  • Executable snapshot-to-result readback and cancellation integration: passed.
  • Full MockRESTCatalogTest: 238 passed.
  • Java prediction, OpenAPI coverage, and request compatibility: 55 passed.
  • Java REST Catalog prediction: 3 passed.
  • Python 3.6 prediction and HTTP client: 20 passed.
  • Python 3.10 prediction and HTTP client: 20 passed, 20 subtests passed.
  • Java Checkstyle and Spotless, Python Flake8, and git diff --check passed.

@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/prediction-job-api branch 4 times, most recently from b1dd30d to 062f66b Compare November 16, 2026 08:38
Add a provider-neutral asynchronous table prediction contract for Java and PyPaimon REST catalogs.
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/prediction-job-api branch from 062f66b to ba1769d Compare November 16, 2026 08:54
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review November 18, 2026 09:12
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [core][python] Add asynchronous table prediction APIs [core][python] Add asynchronous prediction REST APIs Nov 18, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

Thanks for putting together the Java and Python prediction contracts. I ran the client and REST-contract tests from this head. They exercise request encoding and mocked lifecycle responses. The proposed workflow depends on a provider to submit and execute an actual job, read the pinned snapshot, publish the result table, and handle cancellation. This change leaves that provider work outside its scope, so the current patch does not give a Paimon user a deployable prediction workflow.

I'm closing this version. Please bring it back with a linked provider implementation or an executable reference integration that demonstrates submission through result readback, including the snapshot and cancellation behavior.

@JingsongLi JingsongLi closed this Nov 19, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft November 19, 2026 07:18
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review November 19, 2026 13:34
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft November 19, 2026 13:37
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review November 20, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants