[FEATURE] Add PPL asynchronous query lifecycle core by penghuo · Pull Request #5809 · opensearch-project/sql

penghuo · 2026-09-24T22:45:54Z

Description

This is the first implementation slice of #5796, under the PPL async-query meta issue #5765. It adds the execution contract and owner-node lifecycle core without registering REST routes or changing the existing synchronous PPL API.

The lifecycle module depends only on:

public interface AsyncQueryExecution extends AutoCloseable {
  Optional<QueryResponse> currentResult();
  CompletionStage<Void> completion();
  void close();
}

A successful final result must be visible through currentResult() before completion() completes normally.

Core components:

PPLAsyncQueryJob: synchronized state machine for one query.
PPLAsyncQueryService: job registry, timeout race, leases, capacity, task ownership, cancellation, expiration, and cleanup.
PPLAsyncQueryJobId: opaque owner-routable identifier.
PPLAsyncQueryUser: retained-job owner identity.
DefaultAsyncQueryExecution: final-result adapter for existing PPL execution.

Result before timeout

sequenceDiagram
    participant C as Submit caller
    participant S as PPLAsyncQueryService
    participant J as PPLAsyncQueryJob
    participant E as AsyncQueryExecution
    participant T as Wait timeout
    C->>S: start(wait)
    S->>J: create RUNNING
    S->>T: schedule
    S->>E: start and attach
    E-->>S: completion()
    S-->>T: cancel
    S->>J: complete()
    J-->>S: DirectResponse
    S->>E: currentResult()
    S-->>C: final snapshot without ID
    S->>E: close()

Timeout before result

sequenceDiagram
    participant C as Submit caller
    participant S as PPLAsyncQueryService
    participant J as PPLAsyncQueryJob
    participant E as AsyncQueryExecution
    participant T as Wait timeout
    C->>S: start(wait)
    S->>J: create RUNNING
    S->>T: schedule
    S->>E: start and attach
    T-->>S: wait expires
    S->>J: retain()
    J-->>S: Retained(RUNNING)
    S->>E: currentResult()
    S-->>C: RUNNING snapshot with ID
    E-->>S: completion()
    S->>J: complete()
    J-->>S: ExecutionFinished
    C->>S: get(ID)
    S->>J: get()
    J-->>S: succeeded snapshot source
    S->>E: currentResult()
    S-->>C: retained final snapshot

DELETE, expiration, startup abort, and shutdown produce typed removal transitions. The service removes registry state and releases tasks/executions after leaving the job lock.

The follow-up PR adds request parsing, REST/transport actions, owner-node forwarding, response formatting, and integration tests.

Related Issues

Part of #5796.
Parent meta issue: #5765.

Check List

New functionality includes testing.
New functionality has been documented.
- New functionality has javadoc added.
- User manual documentation is not applicable until the public API PR.
New PPL command checklist is not applicable.
API specification PR is not applicable because this PR exposes no endpoint.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR will accompany the public API PR.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Signed-off-by: Peng Huo <penghuo@gmail.com>

github-actions · 2026-09-24T22:46:58Z

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 7abe947.

Path	Line	Severity	Description
plugin/build.gradle	171	high	New dependency added: 'org.opensearch:common-utils:${opensearch_build}'. Per mandatory review policy, all dependency additions must be flagged regardless of whether the artifact name appears legitimate. Maintainers should verify the artifact resolves to the expected OpenSearch common-utils package from a trusted registry and that the version pinned via ${opensearch_build} is appropriate.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 1 | Medium: 0 | Low: 0

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

codecov · 2026-09-24T23:20:18Z

Codecov Report

❌ Patch coverage is 87.50000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.26%. Comparing base (f2f4893) to head (7abe947).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...c/main/java/org/opensearch/sql/ppl/PPLService.java	50.00%	4 Missing ⚠️
...opensearch/sql/ppl/DefaultAsyncQueryExecution.java	90.00%	1 Missing ⚠️

❌ Your project check has failed because the head coverage (63.26%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #5809      +/-   ##
============================================
+ Coverage     63.23%   63.26%   +0.02%     
- Complexity     8823     8829       +6     
============================================
  Files           938      939       +1     
  Lines         40236    40276      +40     
  Branches       4537     4537              
============================================
+ Hits          25445    25480      +35     
- Misses        13966    13971       +5     
  Partials        825      825

Flag	Coverage Δ
sql-engine	`63.26% <87.50%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dai-chen · 2026-09-25T19:07:59Z

+ * <p>This class is thread-safe. The registry is concurrent, capacity counters are guarded by {@code
+ * admissionLock}, and each job synchronizes its own lifecycle transitions.
+ */
+public final class PPLAsyncQueryService extends AbstractLifecycleComponent {


Can we make this new lifecycle layer language-neutral, similar to Livy’s control plane over Spark?

Probably the execution boundary can also be engine-neutral? SQL and PPL can both run on AE too, although AE does not currently support progress listeners.

dai-chen · 2026-09-25T19:17:44Z

+ * lock.
+ *
+ * <pre>
+ * Every job starts in RUNNING. It becomes retained when wait_for_completion_timeout expires.


Could we separate execution state from retention and treat wait_for_completion_timeout only as submit-response behavior? In a BigQuery-style model, the job remains RUNNING whether or not the request is still waiting.

dai-chen · 2026-09-25T20:21:28Z

+  private final Supplier<String> ownerNodeIdSupplier;
+  private final LongSupplier currentTimeMillis;
+  private final TimeoutScheduler timeoutScheduler;
+  private final IntSupplier maxRunningQueries;
+  private final IntSupplier maxRetainedJobs;
+  private final Supplier<TimeValue> maxWaitForCompletion;
+  private final Supplier<TimeValue> maxKeepAlive;
+  private final ThreadPool threadPool;
+  private final ConcurrentMap<String, PPLAsyncQueryJob> jobs = new ConcurrentHashMap<>();
+  private final Object admissionLock = new Object();


wondering shall we extract serveral internal components to offload the responsibilities of this service class, such as async-query-core’s JobStore, LeaseManager etc?

penghuo added 12 commits September 24, 2026 22:43

Add PPL asynchronous query lifecycle core

af73e70

Signed-off-by: Peng Huo <penghuo@gmail.com>

Keep async request parsing in API layer

419de01

Signed-off-by: Peng Huo <penghuo@gmail.com>

Clarify async query user tests

c2c5674

Signed-off-by: Peng Huo <penghuo@gmail.com>

Name the unsecured async query identity

7b73c9f

Signed-off-by: Peng Huo <penghuo@gmail.com>

Split async job state machine tests

eed8cb2

Signed-off-by: Peng Huo <penghuo@gmail.com>

Refine async query lifecycle state model

712d6e5

Signed-off-by: Peng Huo <penghuo@gmail.com>

Clarify async job state assertions

557108f

Signed-off-by: Peng Huo <penghuo@gmail.com>

Verify direct failure removes async job

9050a7f

Signed-off-by: Peng Huo <penghuo@gmail.com>

Clarify async job lease boundary test

83eaff4

Signed-off-by: Peng Huo <penghuo@gmail.com>

Use fluent scenarios for async query service tests

b2beea0

Signed-off-by: Peng Huo <penghuo@gmail.com>

Verify expired async jobs return not found

178b674

Signed-off-by: Peng Huo <penghuo@gmail.com>

Clarify deletion of completed async jobs

7abe947

Signed-off-by: Peng Huo <penghuo@gmail.com>

penghuo requested review from LantaoJin, RyanL1997, Swiddis, acarbonetto, ahkcs, anirudha, dai-chen, joshuali925, mengweieric, noCharger, ps48, qianheng-aws, songkant-aws, vamsimanohar, ykmr1224 and yuancu as code owners September 24, 2026 22:45

penghuo self-assigned this Sep 24, 2026

penghuo added PPL Piped processing language enhancement New feature or request labels Sep 24, 2026

dai-chen reviewed Sep 25, 2026

View reviewed changes

Navigation Menu

[FEATURE] Add PPL asynchronous query lifecycle core - #5809

[FEATURE] Add PPL asynchronous query lifecycle core#5809
penghuo wants to merge 12 commits into
opensearch-project:mainfrom
penghuo:feat/asyncquery-core

penghuo commented Sep 24, 2026

Uh oh!

github-actions Bot commented Sep 24, 2026

Uh oh!

codecov Bot commented Sep 24, 2026

Uh oh!

dai-chen Sep 25, 2026

Uh oh!

dai-chen Sep 25, 2026

Uh oh!

dai-chen Sep 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

penghuo commented Sep 24, 2026

Description

Result before timeout

Timeout before result

Related Issues

Check List

Uh oh!

github-actions Bot commented Sep 24, 2026

PR Code Analyzer ❗

Uh oh!

codecov Bot commented Sep 24, 2026

Codecov Report

Uh oh!

dai-chen Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

dai-chen Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

dai-chen Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants