Skip to content

fix: parse_url handles relative references like java.net.URI - #7392

Open
valentinsoe wants to merge 2 commits into
apache:mainfrom
valentinsoe:fix/parse-url-relative-urls
Open

valentinsoe wants to merge 2 commits into
apache:mainfrom
valentinsoe:fix/parse-url-relative-urls

Conversation

@valentinsoe

@valentinsoe valentinsoe commented Nov 1, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Closes #7385.

Rationale for this change

Native parse_url / try_parse_url return NULL for every part of a URL without a scheme (/p?id=1, ?id=7, //host/p, notaurl). Spark's ParseUrl uses java.net.URI, which parses these as relative references, so with Comet enabled parse_url(page, 'QUERY', 'gclid') on logged page paths silently returns NULL instead of the value.

What changes are included in this PR?

In parse_url_component, the early return Ok(None) for scheme-less values is replaced by the two cases where java.net.URI actually rejects such a value:

  • a leading : (empty scheme name, "Expected scheme name at index 0") → INVALID_URL;
  • an empty authority with nothing after it (//, http://, "Expected authority") → INVALID_URL.

Everything else goes through the existing decompose_uri / part extraction, which already splits relative references per RFC 3986 Appendix B.

Side effects, both matching Spark:

  • :// inside a relative URL's query (/r?to=http://x.com&id=1) no longer raises INVALID_URL.
  • http:// (scheme, empty authority, nothing else) used to return PROTOCOL http; it now raises INVALID_URL (ANSI) / returns NULL, as Spark does.

Not in scope: Comet's character validation (has_invalid_uri_chars) is narrower than java.net.URI's (e.g. |, a malformed % escape, a second #), and extract_host doesn't model registry-based authorities (//ex_ample.com/p). Those gaps already exist for absolute URLs and are tracked in #7330; with this change scheme-less inputs reach them too (Spark returns NULL for such inputs, Comet now returns parsed values instead of NULL).

How are these changes tested?

  • SQL file tests (main coverage, compared against Spark): a table of 17 relative and edge-case URLs queried for every part and a QUERY key in parse_url.sql, plus relative and error cases in parse_url_ansi.sql. CometSqlFileTestSuite parse_url passes locally (Spark 4.1 profile) and fails on main at the new queries.
  • Rust unit tests in parse_url.rs for the changed branch: test_relative_reference (a few relative references per part, expectations from Spark 4.1.1) and test_invalid_relative_reference (:, :foo, ://missing-scheme, //, http://). Both fail on main; all existing spark-expr tests pass, cargo fmt and clippy -D warnings are clean.
  • Differential check (local only, not committed) of the native UDFs against Spark 4.1.1 try_parse_url + java.net.URI validity on ~110k generated URLs and ~40k real-world relative page paths (each also tried with a host prepended): no change in results for absolute URLs except the http://-style inputs above; relative references now match Spark except the inputs covered by Native parse_url accepts URLs and registry-based hosts that java.net.URI treats differently #7330.
  • cargo bench --bench parse_url (main vs this branch): existing absolute-URL cases unchanged (all within ±4%). Scheme-less rows now do the normal extraction instead of returning early.

🤖 Generated with Claude Code

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 1, 2026
@valentinsoe
valentinsoe force-pushed the fix/parse-url-relative-urls branch from 5e6ef04 to 1b6f970 Compare November 2, 2026 10:17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@valentinsoe
valentinsoe force-pushed the fix/parse-url-relative-urls branch from 1b6f970 to 778984c Compare November 2, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native parse_url returns NULL for relative URLs (no scheme), Spark parses them

1 participant