Skip to content

[FEATURE] Add PPL asynchronous query public API - #11

Open
penghuo wants to merge 1 commit into
feat/asyncquery-corefrom
feat/asyncquery-public-api
Open

penghuo wants to merge 1 commit into
feat/asyncquery-corefrom
feat/asyncquery-public-api

Conversation

@penghuo

@penghuo penghuo commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Description

This stacked PR adds the public API for PPL asynchronous queries. Review it against feat/asyncquery-core, which is the source branch of opensearch-project/sql#5809.

The existing synchronous POST /_plugins/_ppl behavior remains unchanged. Supplying wait_for_completion_timeout or keep_alive selects asynchronous execution:

  • POST /_plugins/_ppl submits and waits for direct completion.
  • GET /_plugins/_ppl/jobs/{id} returns the complete current snapshot and optionally renews the lease.
  • DELETE /_plugins/_ppl/jobs/{id} cancels or removes the retained job.

Responses expose only id, status, schema, datarows, total, took, and error where applicable. This PR does not expose progress, partial results, paging, or delta delivery.

sequenceDiagram
    participant C as Client
    participant R as REST/Transport
    participant S as PPLAsyncQueryService
    participant E as AsyncQueryExecution

    C->>R: POST(query, wait, keep_alive)
    R->>S: start(...)
    S->>E: execute
    alt result before wait timeout
        E-->>S: completion
        S-->>R: SUCCEEDED, no ID
        R-->>C: final schema and rows
    else wait timeout before result
        S-->>R: RUNNING with opaque ID
        R-->>C: job ID
        C->>R: GET /jobs/{id} through any node
        R->>R: route to owner and preserve security context
        R->>S: authorize and get snapshot
        S-->>C: RUNNING, SUCCEEDED, or FAILED
    end
Loading

Submit, GET, and DELETE use independent transport permissions. Every GET and DELETE is routed to the owner node and reauthorizes the current caller against the retained owner identity. Owner loss and unknown, expired, or unauthorized jobs do not expose retained metadata or results.

The response formatter consumes the sealed JobSnapshot contract introduced by the core PR. Result materialization runs on the SQL worker pool; lifecycle transitions remain outside that work.

Validation

  • Async request and formatter unit tests.
  • Final-result equivalence and backward-compatibility REST IT.
  • Independent FGAC permission and owner-isolation IT.
  • Multi-node GET/DELETE forwarding, security-context preservation, and owner-departure IT.

Targeted ITs were run with -DignorePrometheus; Full IT was not run.

Related Issues

  • Implements opensearch-project/sql#5796.
  • Part of opensearch-project/sql#5765.
  • Stacked on opensearch-project/sql#5809.
  • Retained-result byte accounting remains in opensearch-project/sql#5804.

Check List

  • New functionality includes tests and Javadocs.
  • User and security documentation updated.
  • Commit is signed per DCO.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
penghuo requested a review from dai-chen as a code owner September 24, 2026 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant