Skip to content

[Doc-8628][Architecture] Update task-structure and metadata docs to the current schema - #18764

Open
victorsheng wants to merge 14 commits into
apache:devfrom
victorsheng:doc-8628-architecture-schema
Open

victorsheng wants to merge 14 commits into
apache:devfrom
victorsheng:doc-8628-architecture-schema

Conversation

@victorsheng

Copy link
Copy Markdown

Was this PR generated or assisted by AI?

YES. AI assisted with comparing the docs against the current schema and task parameter classes, drafting the zh/en docs, and this description. The contributor reviewed the content.

Purpose of the pull request

Closes #8628.

docs/{en,zh}/architecture/task-structure.md still describes the 1.x storage model, where the whole DAG is stored as JSON in t_ds_process_definition.process_definition_json with fields such as receivers, connects and preTasks. metadata.md still refers to t_ds_process_task_relation, t_ds_process_instance and t_ds_relation_process_instance. Both no longer match the current schema.

Brief change log

  • task-structure.md (en/zh): rewrite based on the current schema in dolphinscheduler-dao/src/main/resources/sql:
    • Overall model: t_ds_workflow_definition, t_ds_task_definition, t_ds_workflow_task_relation and their *_log version tables, code + version.
    • Column tables of the three tables, and how the DAG is stored as relation rows (pre_task_code = 0 for root tasks).
    • task_params: common fields (localParams, resourceList), the parameters class of each task type, and for each of SHELL, PYTHON, SQL, PROCEDURE, HTTP, SPARK, MR, FLINK, DATAX, SQOOP, SUB_WORKFLOW, DEPENDENT, CONDITIONS and SWITCH a field table plus a complete task_params example. This keeps the per-type coverage of the old doc (and adds SWITCH), rewritten against the current *Parameters classes and what the UI submits (format-data.ts).
    • Fields that are only written at runtime (e.g. conditionResult.conditionSuccess, SwitchParameters.nextBranch) or never submitted by the UI (e.g. DataxParameters.batchSize) are left out or noted.
  • metadata.md (en/zh): use current table and column names (t_ds_workflow_task_relation, pre_task_code, t_ds_workflow_instance, t_ds_relation_workflow_instance, ...), fix the tenant description (the tenant is chosen at start/schedule time, not stored in the workflow definition), fix the t_ds_projcet typo, link to task-structure.md, and note that the E-R diagrams still show the legacy process table names.

Not included: auto-generating schema docs from the SQL files (discussed in #8628) and redrawing the E-R diagrams, which can be follow-ups.

Inconsistencies found while writing (not changed in this PR)

  • The UI submits socketTimeout for HTTP tasks, but HttpParameters has no such field, so it has no effect.
  • guide/task/map-reduce.md says the program type is JAVA or SCALA, while ProgramType and the UI also support PYTHON.
  • guide/task/flink.md does not mention the standalone deploy mode.
  • guide/task/sqoop.md describes Form/SQL as modelType, while modelType is import/export and Form/SQL is srcQueryType.

Verify this pull request

This pull request is documentation only and is not covered by tests.

  • ./mvnw spotless:check passes.
  • All 16 JSON examples are valid JSON and identical between en and zh. Every key in them exists as a field (or @JsonProperty name) in the parameters classes under dolphinscheduler-task-plugin, except the DataX job JSON embedded in json and the Sqoop srcType/targetType keys that the UI writes and the backend ignores (noted in the doc).
  • Enum values (e.g. ProgramType, FlinkDeployMode, HttpCheckCondition, DEPENDENT cycle/dateValue) were checked against the source and DependentUtils.
  • Column names, types and enum values were checked against dolphinscheduler_mysql.sql and the related enums (TaskTimeoutStrategy, ConditionType, Priority, ...).

Pull Request Notice

Pull Request Notice

🤖 Generated with Claude Code

…he current schema

- Rewrite task-structure.md (en/zh) based on t_ds_workflow_definition,
  t_ds_task_definition, t_ds_workflow_task_relation and their log tables,
  with field tables and task_params examples for 14 task types.
- Update metadata.md (en/zh) to the current table and column names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng
victorsheng requested a review from SbloodyS as a code owner November 3, 2026 07:43
@boring-cyborg

boring-cyborg Bot commented Nov 3, 2026

Copy link
Copy Markdown

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document the sensitive property of parameters

[Fact] The common-fields table only lists prop, direct, type, and value for localParams. The Chinese document has the same omission.

[Fact] Property defines a boolean sensitive field, and the UI allows users to set it. The API uses this field to mask parameter values as ****** in query responses and to restore the original value during updates.

[Fact] The document also states that global_params has the same structure as localParams, so this omission affects both task-local and workflow-global parameters.

[Inference] An API user following this document may omit sensitive, causing the parameter to default to non-sensitive and its value to be returned without masking.

[Judgment] Please add sensitive as a Boolean field, document its default value and masking behavior in both the English and Chinese common-fields tables, and include "sensitive": true in at least one JSON example.

Correct the workflow-to-schedule cardinality

[Fact] metadata.md states that a workflow definition can have one or more schedules. The Chinese document makes the same statement.

[Fact] The current DDL defines the unique index uniq_workflow_definition_code on t_ds_schedules.workflow_definition_code, so the database permits at most one schedule for each workflow definition.

[Judgment] Please change the English text to:

A workflow definition can have zero or one schedule.

[Judgment] Please make the equivalent correction in the Chinese document.

- Document the sensitive property of parameters and its masking behavior.
- Correct the workflow-to-schedule cardinality to zero or one.
- Fix inaccuracies found by re-checking against the source, e.g.
  resourceList, PROCEDURE method, warning_group_id, DEPENDENT semantics,
  FLINK/DATAX/HTTP details, and the queue description in metadata.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks for the review! Both points are addressed:

  1. Added sensitive (Boolean, default false) to the common-fields table in both en/zh, with its masking behavior: stored in plain text, returned as ****** by query APIs, ****** rejected on create, kept on update only for an existing sensitive parameter, and restored from the global parameter on start/backfill. Added "sensitive": true to the SHELL example.
  2. Changed the schedule cardinality to "A workflow definition can have zero or one schedule." in both en/zh, matching the unique index on t_ds_schedules.workflow_definition_code.

I also re-checked the rest of both docs against the source and fixed several more inaccuracies, e.g.:

  • resourceList is not a field of AbstractParameters; it is now documented only where the parameters class defines it.
  • PROCEDURE method is the full call statement (call proc(${in}, ${out})), not the procedure name.
  • t_ds_workflow_definition.warning_group_id is not mapped by the entity (@TableField(exist = false)), so it is documented as unused.
  • DEPENDENT: meaning of depTaskCode -1, parameterPassing scope, the schedule-time base of dateValue, and the failure-waiting rule.
  • FLINK deploy mode / version combinations, DATAX batchSize and blank json, HTTP body/header/timeout behavior, SWITCH/CONDITIONS evaluation details.
  • metadata.md: the user/tenant queue fallback is only used for display; the task's YARN queue comes from its own parameters.

@SbloodyS SbloodyS added the improvement make more easy to user or prompt friendly label Nov 4, 2026
@SbloodyS SbloodyS added this to the 3.5.0 milestone Nov 4, 2026
@SbloodyS SbloodyS added the first time contributor First-time contributor label Nov 4, 2026

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not describe warning_group_id as unused and always NULL

[Fact] task-structure.md states that t_ds_workflow_definition.warning_group_id is unused and always NULL.

[Fact] SchedulerServiceImpl writes the schedule's warningGroupId to the workflow definition when a schedule is created or updated, then persists the workflow definition through workflowDefinitionDao.updateById(...).

[Fact] Therefore, workflow-definition rows can contain a non-null alert-group ID.

[Judgment] Please document the column's actual behavior instead of describing it as always NULL. The metadata document should also include this reference if it intends to enumerate the tables that store warning_group_id.

Mark SQOOP resourceList as ignored

[Fact] The SQOOP section states that resourceList contains resource files used by the task.

[Fact] SqoopParameters does not define resourceList or override getResourceFilesList().

[Fact] The worker downloads resources only from the list returned by getResourceFilesList(), while the inherited implementation returns an empty list.

[Inference] A user may supply SQOOP resource files according to this document and expect them to be available in the task working directory, but the worker will not download them.

[Judgment] Please document resourceList as an unused UI-supplied field, as is already done for SQL, PROCEDURE, and HTTP, or implement actual SQOOP resource handling outside this documentation-only PR.

Do not claim that SWITCH validates the selected node as downstream

[Fact] The SWITCH section states that the chosen branch must be a downstream task in the DAG and that the task fails otherwise.

[Fact] SwitchLogicTask.checkIfBranchExist() only verifies that getTaskNodeByCode(branchNode) returns a node from the workflow graph. It does not verify an edge or downstream relationship from the SWITCH task.

[Fact] When editing an existing workflow, the UI builds the branch options from all task definitions except the SWITCH task itself, rather than only from downstream tasks.

[Inference] An upstream or unrelated task code can pass the implemented existence check, so the documented validation guarantee is stronger than the actual behavior.

[Judgment] Please state that the selected branch must exist in the workflow DAG, unless downstream-relation validation is implemented separately.

- Document warning_group_id of t_ds_workflow_definition as the schedule's alert group
- Mark SQOOP resourceList as unused
- SWITCH only checks that the chosen branch exists in the DAG

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks for the second round! All three points are addressed in both en/zh:

  1. warning_group_id: no longer described as unused. It is now documented as the schedule's alert group, written by SchedulerServiceImpl when a schedule is created or updated (otherwise NULL). metadata.md now also lists t_ds_workflow_definition among the tables referencing warning_group_id.
  2. SQOOP resourceList: marked as a UI-supplied field that the task does not use, since SqoopParameters neither defines it nor overrides getResourceFilesList().
  3. SWITCH branch: the doc now says the selected branch must exist in the workflow DAG. SwitchLogicTask.checkIfBranchExist() only checks that the node exists, not that it is downstream of the SWITCH task, so the doc recommends configuring the branches as downstream tasks.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not imply that warning_group_id is NULL whenever no schedule exists

[Fact] The document says that t_ds_workflow_definition.warning_group_id is set when a schedule is created or updated and is otherwise NULL.

[Fact] SchedulerServiceImpl.deleteSchedulesById() deletes only the schedule row. It does not clear warning_group_id in the corresponding workflow definition.

[Fact] Therefore, a workflow definition with no current schedule can retain the alert-group ID copied from a previously deleted schedule.

[Judgment] Please describe this column as the last schedule alert-group ID copied during schedule creation or update, and note that deleting the schedule does not currently clear it. Alternatively, the deletion behavior would need to be changed outside this documentation-only PR.

Do not say that DEPENDENT_FAILURE_WAITING keeps checking a failed dependency

[Fact] The DEPENDENT section says that DEPENDENT_FAILURE_WAITING keeps checking until failureWaitingTime has elapsed.

[Fact] Once a dependency item returns FAILED, DependentExecute.getModelDependResult() stores that result in dependResultMap.

[Fact] Subsequent evaluations return the cached result from getDependResultForItem() and do not query the dependency again.

[Fact] failureWaitingTime only delays completion of the already-failed DEPENDENT task until the deadline; it does not allow a later successful workflow or task instance to change the cached failure.

[Inference] The current wording may lead users to expect that a failed dependency can recover during the waiting period, which the implementation does not support.

[Judgment] Please replace “keep checking” with wording such as “remain waiting after the first failure until failureWaitingTime minutes after the base time, then fail,” and clarify that the failed dependency result is cached.

Identify the documented column types as MySQL-specific

[Fact] The document says that the schema is defined by the SQL files under dolphinscheduler-dao/src/main/resources/sql, but the column tables use MySQL-specific types such as int(11), tinyint(4), longtext, and datetime.

[Fact] The supported PostgreSQL schema uses different physical types, including int, bigint, text, and timestamp. Some H2 definitions also differ, such as the length of t_ds_task_definition.worker_group.

[Inference] PostgreSQL or H2 users may interpret these tables as the physical schema for their database even though the listed types come from dolphinscheduler_mysql.sql.

[Judgment] Please explicitly state that the displayed types follow the MySQL schema and direct users to the database-specific SQL files for PostgreSQL and H2. Alternatively, use database-neutral logical types.

- warning_group_id keeps the alert group of the last saved schedule and is not cleared on schedule deletion
- DEPENDENT_FAILURE_WAITING caches the failed result and only delays the failure
- State that column types follow the MySQL schema

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks! All three points are addressed in both en/zh:

  1. warning_group_id: now described as the alert group copied from the last created or updated schedule, and noted that deleting the schedule does not clear it (deleteSchedulesById() only deletes the schedule row).
  2. DEPENDENT_FAILURE_WAITING: "keep checking" is replaced with "still waiting until failureWaitingTime minutes after the base time, then fail", and a note below the table says the failed result is cached in dependResultMap and not checked again, so a later successful instance does not change it.
  3. Column types: the overview now states that the column types follow dolphinscheduler_mysql.sql, and points PostgreSQL and H2 users to dolphinscheduler_postgresql.sql and dolphinscheduler_h2.sql.

…tions

- SQL: the worker runs sql when it is not empty, otherwise sqlResource; sqlSource is not read
- MR: mainJar is always run by hadoop jar, the streaming jar for PYTHON
- DEPENDENT: unscheduled runs use the time the DEPENDENT task started as the base time
- DEPENDENT: thisWeek and thisMonth are split into days, each needing a successful instance
- DEPENDENT: the first instance of a self dependency prefers the first scheduled instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

While going through the review comments, I also re-checked other runtime claims against the source and found a few more inaccuracies. They are fixed in both en/zh in 28ab61e:

  1. SQL: the worker runs sql when it is not empty, otherwise the file in sqlResource (SqlTask.ensureSqlContent()). sqlSource is only a UI choice and is not read by the worker.
  2. MR mainJar: it is always run with hadoop jar (MapReduceArgsUtils), so for PYTHON it is the Hadoop streaming jar, not the script.
  3. DEPENDENT base time: for an unscheduled run it is the time the DEPENDENT task started (DependentTaskTracker.calculateDependentDate()), not the workflow start time.
  4. DEPENDENT thisWeek / thisMonth: ranges are split into days (DependentDateUtils.getDateIntervalListBetweenTwoDates()), and every day must have a successful instance (DependentExecute.calculateResultForTasks()).
  5. DEPENDENT self dependency: the "first instance" is the first scheduled instance if the workflow has any, otherwise the first started instance (DependentExecute.isFirstWorkflowInstance()).

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use valid storage paths in the resource examples

[Fact] The SHELL, PYTHON, SPARK, MR, and FLINK examples use resource names such as file:/tmp/dolphinscheduler/default/resources/run.sh.

[Fact] The default storage type is LOCAL, with /tmp/dolphinscheduler as its default base path. LocalStorageOperator exposes ordinary absolute filesystem paths such as /tmp/dolphinscheduler/default/resources/run.sh.

[Fact] The UI stores the resource's returned fullName directly in resourceName.

[Fact] During download, LocalStorageOperator passes resourceName to Paths.get(...); it does not parse a file: URI. Therefore, file:/tmp/... is a different path from /tmp/... under the default LOCAL storage configuration.

[Inference] Users copying these examples with the default storage configuration will reference a resource path that the worker cannot download.

[Judgment] Please use /tmp/dolphinscheduler/default/resources/... for examples explicitly based on the default LOCAL storage, or use a clearly marked storage-specific placeholder and explain that the value must be the exact fullName returned by the resource center.

Document the ignored HTTP socketTimeout field

[Fact] The HTTP form initializes socketTimeout, and format-data.ts includes it in the submitted taskParams.

[Fact] HttpParameters has no socketTimeout field, so the worker ignores the submitted value. connectTimeout is used for the connection, read, and write timeouts instead.

[Fact] The HTTP field table and example omit socketTimeout, even though the page documents other UI-supplied but backend-ignored fields such as resourceList and the SQOOP nested type fields.

[Inference] Keeping this information only in the PR description leaves users with no durable documentation explaining why changing the HTTP socket-timeout value has no effect.

[Judgment] Please add socketTimeout to the HTTP table and example, explicitly stating that the UI submits it but the backend currently ignores it.

Distinguish the CONDITIONS UI options from backend status values

[Fact] The CONDITIONS table states that status is either SUCCESS or FAILURE.

[Fact] Those are the only options exposed by the UI.

[Fact] The backend field is TaskExecutionStatus, whose values also include FORCED_SUCCESS, PAUSE, KILL, NEED_FAULT_TOLERANCE, and other states.

[Fact] ConditionsParameters.checkParameters() does not restrict the status to SUCCESS or FAILURE, and ConditionLogicTask compares the configured enum value with the actual task state using exact equality.

[Fact] Therefore, FORCED_SUCCESS does not match SUCCESS, but a payload explicitly configured with "status": "FORCED_SUCCESS" can match a force-succeeded task.

[Judgment] Please state that the UI offers only SUCCESS and FAILURE, while the backend deserializes TaskExecutionStatus and compares it exactly. This preserves the useful warning about FORCED_SUCCESS without incorrectly presenting the UI choices as the complete backend schema.

- Use LOCAL storage paths without the file: prefix in resource examples
- Document the HTTP socketTimeout field as submitted by the UI but ignored
- Describe CONDITIONS status as an exactly compared TaskExecutionStatus

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks! All three points are addressed in both en/zh:

  1. Resource paths: the SHELL, PYTHON, SPARK, MR and FLINK examples now use /tmp/dolphinscheduler/default/resources/... instead of file:/tmp/.... The common-fields resourceName row now says the value must be the exact fullName returned by the resource center, with the default LOCAL storage path as an example.
  2. HTTP socketTimeout: added to the field table and the example, stating that the UI submits it but the backend ignores it, and connectTimeout is used for connect, read and write.
  3. CONDITIONS status: now described as a TaskExecutionStatus value compared exactly with the upstream task state, while the UI offers only SUCCESS and FAILURE. The FORCED_SUCCESS note is kept.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document how non-global start parameters handle sensitive placeholders

[Fact] The sensitive-parameter section explains that a start or backfill parameter whose value is ****** inherits the value of the global parameter with the same name.

[Fact] SensitivePropertyUtils.restoreStartParams() has additional behavior when no global parameter has the same name.

[Fact] A non-global start parameter whose value is exactly ****** is silently removed from the start-parameter list.

[Fact] A non-global start parameter with any other value is retained but forcibly marked as non-sensitive, even if the submitted parameter has sensitive: true.

[Inference] A client following the current documentation may expect an unmatched placeholder to be preserved or rejected, while the backend actually drops the entire parameter.

[Judgment] Please document both non-global cases: bare ****** is dropped, and other non-global start parameters are treated as non-sensitive.

Mark SUB_WORKFLOW localParams as unused

[Fact] The SUB_WORKFLOW table presents localParams as ordinary task-local parameters.

[Fact] The UI does not expose a local-parameter editor for SUB_WORKFLOW tasks and normally submits an empty array.

[Fact] SubWorkflowLogicTask does not read SubWorkflowParameters.localParams when starting the child workflow.

[Fact] The child workflow's start parameters are built from the parent workflow's global parameters, command parameters, and varPool.

[Inference] An API user may populate SUB_WORKFLOW localParams expecting them to override or supplement the child workflow's parameters, but those values are ignored.

[Judgment] Please mark localParams as unused by SUB_WORKFLOW and document the actual sources from which the child workflow receives its start parameters.

Explain that hourly DEPENDENT ranges are split into hourly intervals

[Fact] The DEPENDENT section currently says that ranges are split into days.

[Fact] Day, week, and month ranges are split into daily intervals.

[Fact] Hour ranges use DependentDateUtils.getLastHoursInterval(), while last24Hours uses getSpecialLastDayInterval(); both produce separate hourly intervals.

[Fact] DependentExecute.calculateResultForTasks() evaluates every generated interval and stops when any interval is not successful.

[Inference] For last24Hours, users may interpret the current text as requiring one successful instance somewhere in the 24-hour range, while the implementation requires a successful candidate in every hourly interval.

[Judgment] Please state that hour ranges are split into hourly intervals, while day, week, and month ranges are split into daily intervals, and that every generated interval must succeed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks! All three points are addressed in both en/zh:

  1. Non-global start parameters: the sensitive-parameter section now also says that a start parameter without a global parameter of the same name is dropped if its value is ******, and otherwise kept and treated as non-sensitive (SensitivePropertyUtils.restoreStartParams()).
  2. SUB_WORKFLOW localParams: marked as not used, and a note below the example explains that the child workflow's start parameters are merged from the parent workflow instance's global parameters, start parameters and varPool, with the later one winning for the same name (SubWorkflowLogicTask).
  3. DEPENDENT intervals: hour ranges are now described as split into hours, day/week/month ranges into days, and every hour or day must have a successful instance.

@victorsheng

Copy link
Copy Markdown
Author

Hi @SbloodyS, all the comments from the fifth round are addressed in 3dcf92a. Could you please take another look when you have time? Thanks!

@SbloodyS

Copy link
Copy Markdown
Member

Do not document warning_group_id as being persisted by schedule updates

[Fact] The document states that t_ds_workflow_definition.warning_group_id contains the alert group copied from the last created or updated schedule and is not cleared when that schedule is deleted. The Chinese document contains the same description.

[Fact] SchedulerServiceImpl assigns warningGroupId to the WorkflowDefinition entity and then calls workflowDefinitionDao.updateById(...).

[Fact] However, WorkflowDefinition.warningGroupId is annotated with @TableField(exist = false).

[Fact] WorkflowDefinitionDaoImpl delegates this operation to the MyBatis-Plus updateById(...) method, and WorkflowDefinitionMapper.xml does not define an explicit update statement that writes warning_group_id.

[Inference] Therefore, warningGroupId is excluded from the generated UPDATE, so creating or updating a schedule does not currently persist this value to t_ds_workflow_definition.warning_group_id. The claim that the column stores the last schedule alert group is inaccurate.

[Unknown] Existing databases may contain values written by an older version, manual SQL, or an external process, but the current schedule create/update path does not provide evidence for that behavior.

[Judgment] Please describe this as a legacy column that is not currently persisted by the WorkflowDefinition update path, in both the English and Chinese documents. If copying the alert group into the workflow definition is intended behavior, the persistence issue should be fixed and verified in a separate non-documentation PR.

…d as a legacy column

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks, you are right. WorkflowDefinition.warningGroupId is @TableField(exist = false) and no mapper statement writes warning_group_id, so the schedule create/update path never persists it. Fixed in both en/zh in 8c5a2e9: the column is now described as a legacy column that is not written by the current code, and the note points to t_ds_schedules.warning_group_id as where the schedule alert group is stored. The persistence issue itself is left for a separate non-documentation PR if needed.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep the metadata description consistent with the legacy column

[Fact] The English metadata.md still states that alert groups are referenced by t_ds_workflow_definition.warning_group_id; the Chinese document makes the same claim.

[Fact] The latest commit correctly describes this column in task-structure.md as a legacy column that is not written by the current code. WorkflowDefinition.warningGroupId has @TableField(exist = false), and no mapper update persists it.

[Inference] The metadata description still presents the column as an active alert-group relationship, contradicting both the implementation and the updated task-structure.md.

[Judgment] Please remove t_ds_workflow_definition from this list, or explicitly identify its warning_group_id as a legacy, currently unpersisted column, in both the English and Chinese documents.

…m alert group references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks! Removed t_ds_workflow_definition from the alert-group reference list in both en/zh metadata.md in ee7476c. The list now only contains t_ds_schedules and t_ds_workflow_instance, consistent with task-structure.md, which describes t_ds_workflow_definition.warning_group_id as a legacy column.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document all parameter sources available to SWITCH expressions

[Fact] The SWITCH field table says ${name} is replaced from “global/startup parameters and the varPool.”

[Fact] paramParsingPreparation() actually builds prepareParamsMap from built-in parameters, project parameters, workflow-global parameters, task-local parameters, command/start parameters, varPool values, and business/scheduling parameters.

[Fact] SwitchLogicTask passes that complete map to generateContentWithTaskParams() and then overlays the task instance’s raw varPool.

[Inference] The current wording incorrectly implies that project parameters, task-local parameters, and built-in/business parameters cannot be referenced by SWITCH expressions.

[Judgment] Please describe the complete prepared-parameter context in both the English and Chinese documents, or link to a shared parameter-precedence description.

Document that DataX clamps both xms and xmx

[Fact] The DataX table states that xms values below 1 are treated as 1, but does not state the same behavior for xmx.

[Fact] DataxTask.loadJvmEnv() applies Math.max(value, 1) to both xms and xmx.

[Inference] A user may interpret xmx: 0 as an unset or special value, although the generated command uses -Xmx1g.

[Judgment] Please add “values below 1 are treated as 1” to the xmx description in both the English and Chinese documents.

…xmx clamping

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

victorsheng commented Nov 13, 2026 •

Copy link
Copy Markdown
Author

Thanks! Both points are addressed in both en/zh in 323992c:

  1. SWITCH condition: ${name} is now described as replaced from the parameters prepared for the task, listing all sources built by paramParsingPreparation(): built-in, project, workflow global, task local and startup/backfill parameters, the varPool, and business parameters such as ${datetime}. It also links to docs/docs/{en,zh}/guide/parameter/priority.md for same-name precedence.
  2. DataX xmx: added "values below 1 are treated as 1", matching DataxTask.loadJvmEnv().

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document the ignored SQL connParams field

docs/docs/en/architecture/task-structure.md:245
docs/docs/zh/architecture/task-structure.md:227

The rewritten SQL field table omits connParams, although the UI still exposes this field for HIVE tasks. format-data.ts never copies it into taskParams, so a value entered in the UI is lost when the task is saved. SqlParameters still declares the field, but SqlTask only logs it and never applies it to the connection, so an API-supplied value also has no functional effect.

Please add connParams to both documents and state that the UI currently exposes but does not persist it, while the worker ignores it even if supplied. Fixing the UI/backend behavior can remain a separate non-documentation change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@victorsheng

Copy link
Copy Markdown
Author

Thanks! Added connParams to the SQL field table in both en/zh in 16adabb. It now says the UI shows the field only for HIVE but does not save it (format-data.ts never copies it into taskParams), and the worker ignores it even if supplied through the API (SqlTask only logs it). Fixing the UI/backend behavior is left for a separate non-documentation change.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not describe *_log tables as retaining every version

docs/docs/en/architecture/task-structure.md:5
docs/docs/zh/architecture/task-structure.md:5

The introduction states twice that the *_log tables keep every version. However, the public workflow-version deletion endpoint removes rows from both t_ds_workflow_definition_log and t_ds_workflow_task_relation_log for a non-current version once no non-terminal instance uses it. The task-version deletion endpoint similarly removes a row from t_ds_task_definition_log without checking historical task instances. Completed instances can therefore still reference a definition version whose log row has been deleted.

Please describe these tables as storing version records rather than retaining every version, and clarify in both the English and Chinese documents that non-current versions can be explicitly deleted, so complete historical retention is not guaranteed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

document first time contributor First-time contributor improvement make more easy to user or prompt friendly

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] [doc] Update table schema for task-structure and metadata

2 participants