Skip to content

[Improvement-18806][API] Remove cross-project workflow modifications - #18807

Open
ruanwenjun wants to merge 6 commits into
apache:devfrom
ruanwenjun:fix/18806-workflow-project-ownership
Open

ruanwenjun wants to merge 6 commits into
apache:devfrom
ruanwenjun:fix/18806-workflow-project-ownership

Conversation

@ruanwenjun

@ruanwenjun ruanwenjun commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Was this PR generated or assisted by AI?

YES. Implementation, regression tests, and the description were assisted by OpenAI Codex.

Purpose of the pull request

Fixes #18806.

Remove the legacy workflow batch-move API, which has no frontend caller, and keep existing workflow/task project ownership unchanged through definition edits, instance updates, and version switching. Reject inconsistent persisted or historical ownership before saving instead of moving records between projects implicitly.

Brief change log

  • Remove the move endpoint, service method/implementation, unused UI wrapper, and move-only wiring. Simplify the shared copy helpers after removing their move branches.
  • Validate workflow-owned task definitions and DAG relations against their project and workflow, including persisted task rows and historical task ownership when request fields are omitted or forged.
  • Use TaskProjectValidator and WorkflowRelationValidator, each implementing IValidator with a nested Context. Relation validation reuses task validation; simple project comparisons remain in the service entry points.
  • Check the instance, current definition, and recorded definition version before instance editing; preflight task/workflow version ownership before restoring a historical version.
  • Before restoring a workflow version, compare the distinct (taskCode, version) pairs referenced by the selected DAG with the returned task logs. Reject any missing pair with TASK_DEFINE_NOT_EXIST, including the missing task codes and versions, before any writes.
  • Restore history-only tasks into the current task table when switching workflow versions, clearing the version-log primary key before insertion. Require every task restoration to succeed and treat zero-row insert/update results as failures so the workflow restore transaction rolls back.
  • Validate copy source ownership before generating new codes and normalize copied relations to the destination project/workflow. Same-project and cross-project copying remain supported.
  • Preserve existing permission checks and legitimate cross-project DEPENDENT/SUB_PROCESS references. No data migration or changes to dependency targets are included.
  • Validate persisted current DAG relations before any writes during definition edits, version switching, and instance saves with syncDefine=true. Reject foreign-project relation rows before replacing the DAG.
  • Preserve historical workflow versions referenced by existing instances, including finished instances, using a database count and the existing WORKFLOW_DEFINITION_VERSION_IS_USED error.
  • Document API removal, rejection of inconsistent historical data, and behavior for instances whose recorded versions were already deleted in both upgrade guides.

Related: #18718 contains project-authorization work; this PR isolates the move removal and ownership rules. #18798 / #18799 continue to cover DEPENDENT self-reference remapping during ordinary copying.

Verify this pull request

  • Cover current-relation ownership during definition edits, instance synchronization, and version switching; instance/current/recorded-version project consistency; retention of workflow versions referenced by finished instances; and missing historical task versions.
  • Verify that a restore referencing a missing predecessor or successor task version is rejected before writes. Repeated task-version references are deduplicated, and complete task logs returned in a different order still permit the restore.
  • Add an H2-backed regression that saves an instance with one existing task and one new task using syncDefine=false, restores the saved version, and verifies both tasks through the task detail HTTP endpoint using MockMvc.
  • Inject zero-row task insert and update results and verify that the transaction restores the previous workflow version, DAG relations, and task rows, without changing schedules. Service tests also cover clearing version-log IDs before inserting current tasks and rejecting partial task restoration.
  • Full API verify and packaging passed with Java 8 and an isolated Maven local repository: 821 tests, 10 skipped, zero failures/errors. This includes 63 tests in WorkflowDefinitionServiceTest, 32 in WorkflowInstanceServiceTest, and three database-backed cases in WorkflowVersionRestoreTest.
  • Full service verify passed: 38 tests, zero failures/errors, using UTC as required by the existing business-parameter timestamp test.
  • Repository-wide Spotless (./mvnw -o spotless:check -B) and git diff --check passed, including the root Markdown formatter and all Maven modules.

Commands (offline, after preparing dependencies and installing the updated service module in the isolated Maven local repository):

./mvnw -o verify -B -pl dolphinscheduler-api -Dmaven.test.skip=false -Dspotless.skip=false -DskipUT=false -Djacoco.skip=true -Danalyze.skip=true
TZ=UTC ./mvnw -o verify -B -pl dolphinscheduler-service -Duser.timezone=UTC -Dmaven.test.skip=false -Dspotless.skip=false -DskipUT=false -Djacoco.skip=true -Danalyze.skip=true

The Docker-based dolphinscheduler-api-test suite, E2E tests, the full frontend build, and deployed-environment validation were not run. CI results are tracked separately by the PR checks.

Pull Request Notice

This removes a public API, rejects updates of inconsistent historical ownership, and prevents deleting workflow versions referenced by existing instances. Editing instances whose recorded versions were already deleted remains rejected, including with syncDefine=false. Compatibility notes are included in the English and Chinese upgrade guides; no automatic migration is performed.

Pull Request Notice

@github-actions github-actions Bot added UI ui and front end related backend test document labels Nov 18, 2026
@ruanwenjun ruanwenjun added this to the 3.5.0 milestone Nov 18, 2026
@ruanwenjun ruanwenjun changed the title [Improvement-18806][API] Remove workflow moves and enforce project ownership [Improvement-18806][API] Remove cross-project workflow modifications Nov 18, 2026
@ruanwenjun
ruanwenjun force-pushed the fix/18806-workflow-project-ownership branch from 1123bd5 to b1c378b Compare November 19, 2026 08:47

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validate the current DAG before replacing its relations

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkflowDefinitionServiceImpl.java:1686

Version switching validates the selected historical DAG but skips the persisted current relations. A workflow moved from project A to B and back through the legacy API can have a valid project-A snapshot and still retain current relations stamped with project B. Restoring the project-A snapshot deletes only project-A rows. Queries by workflow code then return the leftover project-B rows alongside the restored relations.

Validate the persisted current relations before replacing the DAG, including during definition edits and instance saves with syncDefine=true. Add a regression with a valid target snapshot and a current relation stamped with another project, and verify that the operation rejects it before any write.

Preserve workflow versions required for instance updates

  • dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkflowInstanceServiceImpl.java:423
  • dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkflowDefinitionServiceImpl.java:1796

Instance updates now require the recorded workflow definition version to exist, including with syncDefine=false. The version-delete endpoint still allows deleting a non-current version referenced by a finished instance. After that deletion, a direct API client submitting a complete, valid same-project DAG receives WORKFLOW_DEFINITION_NOT_EXIST before the payload is read. Previously, the update could save the submitted DAG as a new version.

Preserve the recorded version while an editable instance references it, and document the behavior for instances whose recorded versions were already deleted. Add a regression that attempts to delete a non-current version referenced by a finished instance and verifies that the instance remains editable.

@ruanwenjun

Copy link
Copy Markdown
Member Author

@SbloodyS Thanks for the review. Both points are addressed in 28f22b1.

  1. Current persisted DAG relations are now validated before any writes during definition edits, version switching, and instance saves with syncDefine=true. The regressions include a valid target snapshot with a foreign-project current relation and verify rejection before saving. Instance-only saves with syncDefine=false do not replace or validate the current DAG.
  2. Version deletion now checks references from all existing instances, including finished instances, using a database count and the existing WORKFLOW_DEFINITION_VERSION_IS_USED error. A regression attempts to delete a finished instance's non-current version, verifies that neither the definition log nor relation logs are deleted, and then successfully edits the instance with both syncDefine values. Unreferenced versions remain deletable.

For recorded versions deleted before upgrading, instance editing still returns WORKFLOW_DEFINITION_NOT_EXIST, even with syncDefine=false; a complete submitted DAG does not reconstruct the missing version. Both upgrade guides now describe this behavior and starting a new instance from a valid definition in the same project.

Validation: all nine new regression cases pass. Full API verify and packaging passed: 816 tests, 10 skipped, zero failures/errors. API/document Spotless and git diff --check also passed. CI results are tracked separately by the PR checks.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reject workflow version restores that reference missing task versions

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkflowDefinitionServiceImpl.java:1697

The restore preflight checks ownership only for task versions returned by the database. The task-version delete API allows deleting a non-current task version that an older workflow DAG still references. When restoring a two-task DAG with one of those versions missing, switchWorkflowTaskRelationVersion writes both relations and reports success if the other task switches successfully. The Master then fails to build the graph because an edge references a missing task.

Before writing, compare the distinct (taskCode, version) pairs referenced by the selected DAG with the task logs returned by the query. Report any missing pair and reject the restore. Add a regression with one available and one deleted task version, and verify that the current workflow version, DAG relations, and schedule remain unchanged.

@ruanwenjun

Copy link
Copy Markdown
Member Author

@SbloodyS Addressed this review comment in 10971c1.

The restore preflight now compares the distinct (taskCode, version) pairs referenced by the selected historical DAG with the task logs returned by the existing query. Any missing pair rejects the restore with TASK_DEFINE_NOT_EXIST, including the task code and version in the error, before calling processService.switchVersion or changing schedules and lineage.

Added regressions for a two-task DAG with either the predecessor or successor task version missing. They verify that the current definition version and relations remain unchanged and that version switching, relation writes, scheduler updates, and lineage updates are not invoked. The successful-restore test also covers repeated references and task logs returned in a different order.

Validation: WorkflowDefinitionServiceTest passed all 63 cases. Full API verify and packaging passed: 818 tests, 10 skipped, zero failures/errors. Spotless and git diff --check passed. CI results are tracked separately by the PR checks.

@SbloodyS

Copy link
Copy Markdown
Member

Restore tasks that exist only in version history

  • dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/WorkflowDefinitionServiceImpl.java:1719
  • dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java:401

Saving an instance with syncDefine=false can add a task whose version log exists but whose current task definition row does not. That workflow version appears in version history and passes the new restore preflight. During restore, switchTaskDefinitionVersion fails for the new task because its current row is missing. If another task switches successfully, the API still reports success and commits the DAG. The task detail API then returns TASK_DEFINE_NOT_EXIST for the new task in the restored definition.

Restore each current task row from its validated version log, inserting the row when needed, and roll back the workflow restore if any task cannot be restored. Add a regression that saves an instance with one existing task and one new task using syncDefine=false, restores that version, and verifies that both tasks are available through the task detail API.

@ruanwenjun

Copy link
Copy Markdown
Member Author

@SbloodyS Addressed this comment in 4af5d6b.

ProcessService now inserts a missing current task row from its version log, clearing the log primary key before insertion. Existing rows retain their current-table ID when updated. Instance-only saves with syncDefine=false remain valid.

Workflow restoration now requires every task restore to succeed. A missing task version or a zero-row insert/update returns failure, causing the API to throw and roll back its transaction instead of committing a partially restored DAG.

Added an H2-backed regression that saves a finished instance with one existing task and one new task using syncDefine=false, restores that version, and verifies both tasks through the task detail HTTP endpoint with MockMvc. Two failure-injection cases verify rollback of the workflow definition, DAG relations, and previously updated task rows when task insertion or updating returns zero; schedules are not changed. Service unit tests cover history-only restoration, log-ID clearing, failed updates, and partial restoration.

Validation: API verify/packaging passed 821 tests (10 skipped, no failures/errors); service verify passed 38 tests using UTC for its existing timestamp test. Repository-wide Spotless and git diff --check passed. CI results are tracked separately by the PR checks.

@ruanwenjun
ruanwenjun force-pushed the fix/18806-workflow-project-ownership branch from 4af5d6b to fb92ef5 Compare November 20, 2026 07:22

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend document test UI ui and front end related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement][API] Remove workflow batch-move and enforce immutable project ownership

2 participants