Repository navigation
[Doc-8628][Architecture] Update task-structure and metadata docs to the current schema - #18764
victorsheng wants to merge 14 commits into
Conversation
…he current schema - Rewrite task-structure.md (en/zh) based on t_ds_workflow_definition, t_ds_task_definition, t_ds_workflow_task_relation and their log tables, with field tables and task_params examples for 14 task types. - Update metadata.md (en/zh) to the current table and column names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md) |
SbloodyS
left a comment
There was a problem hiding this comment.
Document the sensitive property of parameters
[Fact] The common-fields table only lists prop, direct, type, and value for localParams. The Chinese document has the same omission.
[Fact] Property defines a boolean sensitive field, and the UI allows users to set it. The API uses this field to mask parameter values as ****** in query responses and to restore the original value during updates.
[Fact] The document also states that global_params has the same structure as localParams, so this omission affects both task-local and workflow-global parameters.
[Inference] An API user following this document may omit sensitive, causing the parameter to default to non-sensitive and its value to be returned without masking.
[Judgment] Please add sensitive as a Boolean field, document its default value and masking behavior in both the English and Chinese common-fields tables, and include "sensitive": true in at least one JSON example.
Correct the workflow-to-schedule cardinality
[Fact] metadata.md states that a workflow definition can have one or more schedules. The Chinese document makes the same statement.
[Fact] The current DDL defines the unique index uniq_workflow_definition_code on t_ds_schedules.workflow_definition_code, so the database permits at most one schedule for each workflow definition.
[Judgment] Please change the English text to:
A workflow definition can have zero or one schedule.
[Judgment] Please make the equivalent correction in the Chinese document.
- Document the sensitive property of parameters and its masking behavior. - Correct the workflow-to-schedule cardinality to zero or one. - Fix inaccuracies found by re-checking against the source, e.g. resourceList, PROCEDURE method, warning_group_id, DEPENDENT semantics, FLINK/DATAX/HTTP details, and the queue description in metadata.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review! Both points are addressed:
I also re-checked the rest of both docs against the source and fixed several more inaccuracies, e.g.:
|
SbloodyS
left a comment
There was a problem hiding this comment.
Do not describe warning_group_id as unused and always NULL
[Fact] task-structure.md states that t_ds_workflow_definition.warning_group_id is unused and always NULL.
[Fact] SchedulerServiceImpl writes the schedule's warningGroupId to the workflow definition when a schedule is created or updated, then persists the workflow definition through workflowDefinitionDao.updateById(...).
[Fact] Therefore, workflow-definition rows can contain a non-null alert-group ID.
[Judgment] Please document the column's actual behavior instead of describing it as always NULL. The metadata document should also include this reference if it intends to enumerate the tables that store warning_group_id.
Mark SQOOP resourceList as ignored
[Fact] The SQOOP section states that resourceList contains resource files used by the task.
[Fact] SqoopParameters does not define resourceList or override getResourceFilesList().
[Fact] The worker downloads resources only from the list returned by getResourceFilesList(), while the inherited implementation returns an empty list.
[Inference] A user may supply SQOOP resource files according to this document and expect them to be available in the task working directory, but the worker will not download them.
[Judgment] Please document resourceList as an unused UI-supplied field, as is already done for SQL, PROCEDURE, and HTTP, or implement actual SQOOP resource handling outside this documentation-only PR.
Do not claim that SWITCH validates the selected node as downstream
[Fact] The SWITCH section states that the chosen branch must be a downstream task in the DAG and that the task fails otherwise.
[Fact] SwitchLogicTask.checkIfBranchExist() only verifies that getTaskNodeByCode(branchNode) returns a node from the workflow graph. It does not verify an edge or downstream relationship from the SWITCH task.
[Fact] When editing an existing workflow, the UI builds the branch options from all task definitions except the SWITCH task itself, rather than only from downstream tasks.
[Inference] An upstream or unrelated task code can pass the implemented existence check, so the documented validation guarantee is stronger than the actual behavior.
[Judgment] Please state that the selected branch must exist in the workflow DAG, unless downstream-relation validation is implemented separately.
- Document warning_group_id of t_ds_workflow_definition as the schedule's alert group - Mark SQOOP resourceList as unused - SWITCH only checks that the chosen branch exists in the DAG Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the second round! All three points are addressed in both en/zh:
|
SbloodyS
left a comment
There was a problem hiding this comment.
Do not imply that warning_group_id is NULL whenever no schedule exists
[Fact] The document says that t_ds_workflow_definition.warning_group_id is set when a schedule is created or updated and is otherwise NULL.
[Fact] SchedulerServiceImpl.deleteSchedulesById() deletes only the schedule row. It does not clear warning_group_id in the corresponding workflow definition.
[Fact] Therefore, a workflow definition with no current schedule can retain the alert-group ID copied from a previously deleted schedule.
[Judgment] Please describe this column as the last schedule alert-group ID copied during schedule creation or update, and note that deleting the schedule does not currently clear it. Alternatively, the deletion behavior would need to be changed outside this documentation-only PR.
Do not say that DEPENDENT_FAILURE_WAITING keeps checking a failed dependency
[Fact] The DEPENDENT section says that DEPENDENT_FAILURE_WAITING keeps checking until failureWaitingTime has elapsed.
[Fact] Once a dependency item returns FAILED, DependentExecute.getModelDependResult() stores that result in dependResultMap.
[Fact] Subsequent evaluations return the cached result from getDependResultForItem() and do not query the dependency again.
[Fact] failureWaitingTime only delays completion of the already-failed DEPENDENT task until the deadline; it does not allow a later successful workflow or task instance to change the cached failure.
[Inference] The current wording may lead users to expect that a failed dependency can recover during the waiting period, which the implementation does not support.
[Judgment] Please replace “keep checking” with wording such as “remain waiting after the first failure until failureWaitingTime minutes after the base time, then fail,” and clarify that the failed dependency result is cached.
Identify the documented column types as MySQL-specific
[Fact] The document says that the schema is defined by the SQL files under dolphinscheduler-dao/src/main/resources/sql, but the column tables use MySQL-specific types such as int(11), tinyint(4), longtext, and datetime.
[Fact] The supported PostgreSQL schema uses different physical types, including int, bigint, text, and timestamp. Some H2 definitions also differ, such as the length of t_ds_task_definition.worker_group.
[Inference] PostgreSQL or H2 users may interpret these tables as the physical schema for their database even though the listed types come from dolphinscheduler_mysql.sql.
[Judgment] Please explicitly state that the displayed types follow the MySQL schema and direct users to the database-specific SQL files for PostgreSQL and H2. Alternatively, use database-neutral logical types.
- warning_group_id keeps the alert group of the last saved schedule and is not cleared on schedule deletion - DEPENDENT_FAILURE_WAITING caches the failed result and only delays the failure - State that column types follow the MySQL schema Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! All three points are addressed in both en/zh:
|
…tions - SQL: the worker runs sql when it is not empty, otherwise sqlResource; sqlSource is not read - MR: mainJar is always run by hadoop jar, the streaming jar for PYTHON - DEPENDENT: unscheduled runs use the time the DEPENDENT task started as the base time - DEPENDENT: thisWeek and thisMonth are split into days, each needing a successful instance - DEPENDENT: the first instance of a self dependency prefers the first scheduled instance Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
While going through the review comments, I also re-checked other runtime claims against the source and found a few more inaccuracies. They are fixed in both en/zh in 28ab61e:
|
SbloodyS
left a comment
There was a problem hiding this comment.
Use valid storage paths in the resource examples
[Fact] The SHELL, PYTHON, SPARK, MR, and FLINK examples use resource names such as file:/tmp/dolphinscheduler/default/resources/run.sh.
[Fact] The default storage type is LOCAL, with /tmp/dolphinscheduler as its default base path. LocalStorageOperator exposes ordinary absolute filesystem paths such as /tmp/dolphinscheduler/default/resources/run.sh.
[Fact] The UI stores the resource's returned fullName directly in resourceName.
[Fact] During download, LocalStorageOperator passes resourceName to Paths.get(...); it does not parse a file: URI. Therefore, file:/tmp/... is a different path from /tmp/... under the default LOCAL storage configuration.
[Inference] Users copying these examples with the default storage configuration will reference a resource path that the worker cannot download.
[Judgment] Please use /tmp/dolphinscheduler/default/resources/... for examples explicitly based on the default LOCAL storage, or use a clearly marked storage-specific placeholder and explain that the value must be the exact fullName returned by the resource center.
Document the ignored HTTP socketTimeout field
[Fact] The HTTP form initializes socketTimeout, and format-data.ts includes it in the submitted taskParams.
[Fact] HttpParameters has no socketTimeout field, so the worker ignores the submitted value. connectTimeout is used for the connection, read, and write timeouts instead.
[Fact] The HTTP field table and example omit socketTimeout, even though the page documents other UI-supplied but backend-ignored fields such as resourceList and the SQOOP nested type fields.
[Inference] Keeping this information only in the PR description leaves users with no durable documentation explaining why changing the HTTP socket-timeout value has no effect.
[Judgment] Please add socketTimeout to the HTTP table and example, explicitly stating that the UI submits it but the backend currently ignores it.
Distinguish the CONDITIONS UI options from backend status values
[Fact] The CONDITIONS table states that status is either SUCCESS or FAILURE.
[Fact] Those are the only options exposed by the UI.
[Fact] The backend field is TaskExecutionStatus, whose values also include FORCED_SUCCESS, PAUSE, KILL, NEED_FAULT_TOLERANCE, and other states.
[Fact] ConditionsParameters.checkParameters() does not restrict the status to SUCCESS or FAILURE, and ConditionLogicTask compares the configured enum value with the actual task state using exact equality.
[Fact] Therefore, FORCED_SUCCESS does not match SUCCESS, but a payload explicitly configured with "status": "FORCED_SUCCESS" can match a force-succeeded task.
[Judgment] Please state that the UI offers only SUCCESS and FAILURE, while the backend deserializes TaskExecutionStatus and compares it exactly. This preserves the useful warning about FORCED_SUCCESS without incorrectly presenting the UI choices as the complete backend schema.
- Use LOCAL storage paths without the file: prefix in resource examples - Document the HTTP socketTimeout field as submitted by the UI but ignored - Describe CONDITIONS status as an exactly compared TaskExecutionStatus Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! All three points are addressed in both en/zh:
|
SbloodyS
left a comment
There was a problem hiding this comment.
Document how non-global start parameters handle sensitive placeholders
[Fact] The sensitive-parameter section explains that a start or backfill parameter whose value is ****** inherits the value of the global parameter with the same name.
[Fact] SensitivePropertyUtils.restoreStartParams() has additional behavior when no global parameter has the same name.
[Fact] A non-global start parameter whose value is exactly ****** is silently removed from the start-parameter list.
[Fact] A non-global start parameter with any other value is retained but forcibly marked as non-sensitive, even if the submitted parameter has sensitive: true.
[Inference] A client following the current documentation may expect an unmatched placeholder to be preserved or rejected, while the backend actually drops the entire parameter.
[Judgment] Please document both non-global cases: bare ****** is dropped, and other non-global start parameters are treated as non-sensitive.
Mark SUB_WORKFLOW localParams as unused
[Fact] The SUB_WORKFLOW table presents localParams as ordinary task-local parameters.
[Fact] The UI does not expose a local-parameter editor for SUB_WORKFLOW tasks and normally submits an empty array.
[Fact] SubWorkflowLogicTask does not read SubWorkflowParameters.localParams when starting the child workflow.
[Fact] The child workflow's start parameters are built from the parent workflow's global parameters, command parameters, and varPool.
[Inference] An API user may populate SUB_WORKFLOW localParams expecting them to override or supplement the child workflow's parameters, but those values are ignored.
[Judgment] Please mark localParams as unused by SUB_WORKFLOW and document the actual sources from which the child workflow receives its start parameters.
Explain that hourly DEPENDENT ranges are split into hourly intervals
[Fact] The DEPENDENT section currently says that ranges are split into days.
[Fact] Day, week, and month ranges are split into daily intervals.
[Fact] Hour ranges use DependentDateUtils.getLastHoursInterval(), while last24Hours uses getSpecialLastDayInterval(); both produce separate hourly intervals.
[Fact] DependentExecute.calculateResultForTasks() evaluates every generated interval and stops when any interval is not successful.
[Inference] For last24Hours, users may interpret the current text as requiring one successful instance somewhere in the 24-hour range, while the implementation requires a successful candidate in every hourly interval.
[Judgment] Please state that hour ranges are split into hourly intervals, while day, week, and month ranges are split into daily intervals, and that every generated interval must succeed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! All three points are addressed in both en/zh:
|
|
Hi @SbloodyS, all the comments from the fifth round are addressed in 3dcf92a. Could you please take another look when you have time? Thanks! |
Do not document
|
…d as a legacy column Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, you are right. |
SbloodyS
left a comment
There was a problem hiding this comment.
Keep the metadata description consistent with the legacy column
[Fact] The English metadata.md still states that alert groups are referenced by t_ds_workflow_definition.warning_group_id; the Chinese document makes the same claim.
[Fact] The latest commit correctly describes this column in task-structure.md as a legacy column that is not written by the current code. WorkflowDefinition.warningGroupId has @TableField(exist = false), and no mapper update persists it.
[Inference] The metadata description still presents the column as an active alert-group relationship, contradicting both the implementation and the updated task-structure.md.
[Judgment] Please remove t_ds_workflow_definition from this list, or explicitly identify its warning_group_id as a legacy, currently unpersisted column, in both the English and Chinese documents.
…m alert group references Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! Removed |
SbloodyS
left a comment
There was a problem hiding this comment.
Document all parameter sources available to SWITCH expressions
[Fact] The SWITCH field table says ${name} is replaced from “global/startup parameters and the varPool.”
[Fact] paramParsingPreparation() actually builds prepareParamsMap from built-in parameters, project parameters, workflow-global parameters, task-local parameters, command/start parameters, varPool values, and business/scheduling parameters.
[Fact] SwitchLogicTask passes that complete map to generateContentWithTaskParams() and then overlays the task instance’s raw varPool.
[Inference] The current wording incorrectly implies that project parameters, task-local parameters, and built-in/business parameters cannot be referenced by SWITCH expressions.
[Judgment] Please describe the complete prepared-parameter context in both the English and Chinese documents, or link to a shared parameter-precedence description.
Document that DataX clamps both xms and xmx
[Fact] The DataX table states that xms values below 1 are treated as 1, but does not state the same behavior for xmx.
[Fact] DataxTask.loadJvmEnv() applies Math.max(value, 1) to both xms and xmx.
[Inference] A user may interpret xmx: 0 as an unset or special value, although the generated command uses -Xmx1g.
[Judgment] Please add “values below 1 are treated as 1” to the xmx description in both the English and Chinese documents.
…xmx clamping Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! Both points are addressed in both en/zh in 323992c:
|
SbloodyS
left a comment
There was a problem hiding this comment.
Document the ignored SQL connParams field
docs/docs/en/architecture/task-structure.md:245
docs/docs/zh/architecture/task-structure.md:227
The rewritten SQL field table omits connParams, although the UI still exposes this field for HIVE tasks. format-data.ts never copies it into taskParams, so a value entered in the UI is lost when the task is saved. SqlParameters still declares the field, but SqlTask only logs it and never applies it to the connection, so an API-supplied value also has no functional effect.
Please add connParams to both documents and state that the UI currently exposes but does not persist it, while the worker ignores it even if supplied. Fixing the UI/backend behavior can remain a separate non-documentation change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks! Added |
… doc-8628-architecture-schema
SbloodyS
left a comment
There was a problem hiding this comment.
Do not describe *_log tables as retaining every version
docs/docs/en/architecture/task-structure.md:5
docs/docs/zh/architecture/task-structure.md:5
The introduction states twice that the *_log tables keep every version. However, the public workflow-version deletion endpoint removes rows from both t_ds_workflow_definition_log and t_ds_workflow_task_relation_log for a non-current version once no non-terminal instance uses it. The task-version deletion endpoint similarly removes a row from t_ds_task_definition_log without checking historical task instances. Completed instances can therefore still reference a definition version whose log row has been deleted.
Please describe these tables as storing version records rather than retaining every version, and clarify in both the English and Chinese documents that non-current versions can be explicitly deleted, so complete historical retention is not guaranteed.
Was this PR generated or assisted by AI?
YES. AI assisted with comparing the docs against the current schema and task parameter classes, drafting the zh/en docs, and this description. The contributor reviewed the content.
Purpose of the pull request
Closes #8628.
docs/{en,zh}/architecture/task-structure.mdstill describes the 1.x storage model, where the whole DAG is stored as JSON int_ds_process_definition.process_definition_jsonwith fields such asreceivers,connectsandpreTasks.metadata.mdstill refers tot_ds_process_task_relation,t_ds_process_instanceandt_ds_relation_process_instance. Both no longer match the current schema.Brief change log
task-structure.md(en/zh): rewrite based on the current schema indolphinscheduler-dao/src/main/resources/sql:t_ds_workflow_definition,t_ds_task_definition,t_ds_workflow_task_relationand their*_logversion tables,code + version.pre_task_code = 0for root tasks).task_params: common fields (localParams,resourceList), the parameters class of each task type, and for each of SHELL, PYTHON, SQL, PROCEDURE, HTTP, SPARK, MR, FLINK, DATAX, SQOOP, SUB_WORKFLOW, DEPENDENT, CONDITIONS and SWITCH a field table plus a completetask_paramsexample. This keeps the per-type coverage of the old doc (and adds SWITCH), rewritten against the current*Parametersclasses and what the UI submits (format-data.ts).conditionResult.conditionSuccess,SwitchParameters.nextBranch) or never submitted by the UI (e.g.DataxParameters.batchSize) are left out or noted.metadata.md(en/zh): use current table and column names (t_ds_workflow_task_relation,pre_task_code,t_ds_workflow_instance,t_ds_relation_workflow_instance, ...), fix the tenant description (the tenant is chosen at start/schedule time, not stored in the workflow definition), fix thet_ds_projcettypo, link totask-structure.md, and note that the E-R diagrams still show the legacyprocesstable names.Not included: auto-generating schema docs from the SQL files (discussed in #8628) and redrawing the E-R diagrams, which can be follow-ups.
Inconsistencies found while writing (not changed in this PR)
socketTimeoutfor HTTP tasks, butHttpParametershas no such field, so it has no effect.guide/task/map-reduce.mdsays the program type is JAVA or SCALA, whileProgramTypeand the UI also support PYTHON.guide/task/flink.mddoes not mention thestandalonedeploy mode.guide/task/sqoop.mddescribes Form/SQL asmodelType, whilemodelTypeis import/export and Form/SQL issrcQueryType.Verify this pull request
This pull request is documentation only and is not covered by tests.
./mvnw spotless:checkpasses.@JsonPropertyname) in the parameters classes underdolphinscheduler-task-plugin, except the DataX job JSON embedded injsonand the SqoopsrcType/targetTypekeys that the UI writes and the backend ignores (noted in the doc).ProgramType,FlinkDeployMode,HttpCheckCondition, DEPENDENTcycle/dateValue) were checked against the source andDependentUtils.dolphinscheduler_mysql.sqland the related enums (TaskTimeoutStrategy,ConditionType,Priority, ...).Pull Request Notice
Pull Request Notice
🤖 Generated with Claude Code