Skip to content

[FLINK-41568][docs] Add pages for the comparison, boolean, hash, JSON and VARIANT functions - #30033

Open
raminqaf wants to merge 7 commits into
apache:masterfrom
raminqaf:FLINK-41568
Open

raminqaf wants to merge 7 commits into
apache:masterfrom
raminqaf:FLINK-41568

Conversation

@raminqaf

@raminqaf raminqaf commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Step 2 of FLINK-23804, continued after #29864: every function and operator of five more categories gets a page of its own, generated from YAML. Each page was rechecked against the implementation, and every example result comes from running its SQL in batch and streaming mode. The recheck changed what the docs say, for example:

Function Old docs Master
= "Returns TRUE if value1 is equal to value2" TRUE = 'true' is FALSE: the string isn't cast
SIMILAR TO "The escape character has not been supported yet." ESCAPE works. A . matches any character, as in a Java regular expression
JSON_OBJECT "keys must be non-NULL string literals" In SQL, a NOT NULL string column works too. The Table API needs literals
JSON_QUOTE Escapes ", \, / and the control characters \b, \f, \n, \r, \t An emoji comes out garbled, and CHR(1) isn't escaped
JSON_UNQUOTE A value that isn't a valid JSON string literal "is passed through unmodified" '"a"b"' returns a"b

Brief change log

  • The first commit names every argument that takes any expression expr, or expr1 and expr2, as Databricks, Snowflake and DuckDB do, also on the merged pages. It renames the arguments and aliases that the syntax highlighter colored like SQL keywords, such as type in CAST(value AS type) and offset in LAG. docs/README.md and AGENTS.md state both rules.
  • Add 40 pages: Comparison 9, Boolean 4, Hash 7, JSON 12, Variant 8. Every row of these categories now points at a page.
  • Comparison and Boolean are operators. A page covers an operator and its NOT form, such as IS NULL and IS NOT NULL. The operators =, <>, <, <=, > and >= share a "Comparison operators" page, and AND, OR and NOT a "Logical operators" page. The two VARIANT subscript rows share one page. The split follows Snowflake for the grouped operators and Databricks for the IS tests.
  • The Table API column shows the Java calls instead of the Scala ones, such as isEqual(...) for === and jsonQuote() for JsonQuote(). The Chinese table gets the JSON(value) row it lacked.
  • The XML mapping of VARIANT, about 220 lines with tables, moves from the category page to a page "XML and VARIANT", and the path syntax to a page "VARIANT Paths". The Variant category page keeps its intro and table, like the other categories. Links and the anchors of the old single functions page follow. Two sentences there are fixed: TRY_VARIANT_GET returns NULL for a malformed path that isn't a literal, and reading XML back with PARSE_XML needs the same force_array.
  • Chinese pages show the English sections. Where a row had a Chinese description, its first sentence becomes the summary, except on three pages where it described only one operator or mistranslated it.

Verifying this change

This change is documentation only.

  • Every SQL example ran in batch and streaming mode on master with a local runner, which isn't part of this PR. The results on the pages are pasted from those runs.
  • A second review per category checked each statement against the type strategies, the runtime code and the tests. A third pass removed what the reviews of [FLINK-41433][docs] Add pages for the regular expression, cast, conditional, window and utility functions #29864 asked to leave out.
  • Hugo 0.126.3 builds both languages without warnings. A link check finds no new broken internal links. Every commit builds on its own.

The recheck found bugs in the code, which this PR doesn't fix. The pages describe the current behavior, also where it's a bug that users would plausibly hit, such as JSON_STRING writing a FLOAT with the digits of a DOUBLE. They name no tickets. I'll file the bugs separately.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)

@flinkbot

flinkbot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

…id SQL keywords

The syntax highlighter colors a lower-case word like a keyword when it is one, such as `type` in CAST(value AS type) or `offset` and `default` in LAG, while `value1` stays plain. Rename those arguments and example aliases on the pages of the cast, conditional, utility and window functions, such as CAST(expr AS data_type) and LAG(expr [, n [, default_value]]).

Name an argument that takes any expression expr, or expr1 and expr2, as Databricks, Snowflake and DuckDB do. The window pages used expression, and IFNULL and TYPEOF input.

The contributor guide in docs/README.md and AGENTS.md state both rules.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each operator was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. A page covers an operator and its NOT form, and the six operators =, <>, <, <=, > and >= share one page. Compared with the old table rows, the pages add:

- <, <=, > and >= cast a string to a number, while = and <> reject a string and a number. TRUE = 'true' is FALSE.
- SIMILAR TO supports ESCAPE, and a . matches any character.
- NOT IN with a NULL in the list or the sub-query keeps no row. Sub-queries run in streaming mode too.
- The Java and Python calls, such as isEqual(...) instead of the Scala ===.

The IS_NAN page documents the name IS_NAN, which SQL doesn't resolve yet: built-in functions are registered under their name, here isNan, not their SQL name. Its example results come from running ISNAN, and the fix gets its own ticket.

The Chinese pages keep the first sentence of the existing Chinese description as their summary. The comparison operators and IS DISTINCT FROM pages show the English summary, since their Chinese sentence described only = or mistranslated the operator.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each operator was rechecked against Calcite's operators, the generated code and the tests, and every example result comes from running its SQL. AND, OR and NOT share one page, and IS TRUE, IS FALSE and IS UNKNOWN each get a page with their NOT form. Compared with the old table rows, the pages add:

- FALSE AND NULL is FALSE and TRUE OR NULL is TRUE, while the other combinations with NULL are NULL.
- The IS tests never return NULL. IS UNKNOWN only takes a BOOLEAN and has no Table API method.
- The Java and Python calls. The old rows showed only the Scala ones.

The IS NULL page links to IS UNKNOWN from this commit on.

The Chinese pages keep the first sentence of the existing Chinese description as their summary. The logical operators page shows the English summary, since its Chinese sentence described only OR.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, the generated code and the tests, and every example result comes from running its SQL and matches the digest of another tool. Compared with the old table rows, the pages add:

- The digest is lower-case hexadecimal, and a string is hashed as its UTF-8 bytes. In SQL, a binary string works too.
- The result types: CHAR(32) to CHAR(128) for the fixed-length functions, VARCHAR(128) for SHA2.
- SHA2 fails for a length other than 224, 256, 384 or 512.
- The Python calls. md5 to sha512 are properties there.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, Calcite's JSON functions, the generated code and the tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add:

- Strict mode is the default, so a missing key goes to ON ERROR, and ON EMPTY applies only in lax mode. JSON_LENGTH and JSON_TYPE take no mode at all.
- How JSON_STRING writes each type. A FLOAT gets the digits of a DOUBLE.
- JSON_QUOTE escapes a character above U+FFFF wrongly and leaves most control characters as they are.
- Whether JSON_OBJECT and JSON_ARRAY insert a nested call as JSON depends on the query. Wrapping a value in JSON always works.
- The Table API calls, such as jsonQuote() instead of JsonQuote() and jsonLength(...) as a method.

The Chinese table gets the JSON(value) row it lacked. The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. The two subscript rows, variant[INT] and variant[STRING], share one page. The pages link the sections of the category page on XML and on paths instead of repeating them. Compared with the old table rows, the pages add:

- The result types and NULL cases of each function. A JSON null is a variant null, not SQL NULL.
- A subscript on the wrong kind of value fails the job, also for a JSON null in a chain, while VARIANT_GET returns NULL.
- XML_STRING fails for a VARIANT that isn't an object with one field, or that has no XML form.

The category page now says that TRY_VARIANT_GET returns NULL for a malformed path that isn't a literal, and that reading XML back with PARSE_XML needs the same force_array. The JSON pages link to the VARIANT pages from this commit on.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
…their own pages

The XML mapping, about 220 lines with tables, filled most of the Variant category page. It moves to a page "XML and VARIANT" with both directions, since they share the conventions for attributes, text, order and xsi:type. The paths that VARIANT_GET and TRY_VARIANT_GET take move to a page "VARIANT Paths". The category page keeps its intro and the function table, like the other categories, and links both pages.

The function pages, the VARIANT section of the data types page and the anchors of the old single function page point at the new pages. Apart from one intro sentence, the moved text doesn't change.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
@raminqaf
raminqaf marked this pull request as ready for review November 20, 2026 08:00
Comment on lines +69 to +73
SELECT id,
qty > 10 AND price > 10 AS both_hold,
qty > 10 OR price > 10 AS either_holds,
NOT qty > 10 AS small_order
FROM (VALUES (1, 5, 20.0), (2, 50, 8.0), (3, 12, 15.0)) AS t(id, qty, price);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
SELECT id,
qty > 10 AND price > 10 AS both_hold,
qty > 10 OR price > 10 AS either_holds,
NOT qty > 10 AS small_order
FROM (VALUES (1, 5, 20.0), (2, 50, 8.0), (3, 12, 15.0)) AS t(id, qty, price);
SELECT qty,
price,
qty > 10 AND price > 10 AS `qty > 10 AND price > 10`,
qty > 10 OR price > 10 AS `qty > 10 OR price > 10`,
NOT qty > 10 AS `NOT qty > 10`
FROM (VALUES (1, 5, 20.0), (2, 50, 8.0), (3, 12, 15.0)) AS t(id, qty, price);

how about this?
no need to go from table to query each time to compare values by eyes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for others

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants