Skip to content

Add an opt-in regex evaluation suite - #79

Draft
leoromanovsky wants to merge 1 commit into
mainfrom
leo.romanovsky/regex-linear-time-suite
Draft

leoromanovsky wants to merge 1 commit into
mainfrom
leo.romanovsky/regex-linear-time-suite

Conversation

@leoromanovsky

Copy link
Copy Markdown
Collaborator

Motivation

SDKs evaluate MATCHES and NOT_MATCHES with different regex engines. The libdatadog evaluator uses the Rust regex crate, which runs in linear time. Java uses java.util.regex, which backtracks. A long attribute value can make a backtracking engine take milliseconds or more for one evaluation.

This suite gives Rust and Java (and any other evaluator that moves to a linear-time engine with the Rust dialect) one shared set of expected results. It includes inputs that make a backtracking engine slow.

Changes

  • Add the opt-in suite evaluation-suites/regex/: 39 flags and 63 cases in 4 files.
    • literals.json: patterns that are only text, with anchors, escapes, a leading or trailing .*, and groups of literal alternatives.
    • semantics.json: places where dialects differ: $ before a final line feed, . and line feeds, the m, s, i, and u flags, POSIX classes, counted repetition, Unicode word boundaries, and NOT_MATCHES with a missing attribute.
    • backtracking.json: nested quantifiers, overlapping alternatives, a leading .* on a 20,000-character input, and chains of .*.
    • unsupported-syntax.json: lookahead, lookbehind, backreferences, and atomic groups give PARSE_ERROR.
  • Document the suite in README.md.

Decisions

  • An opt-in suite, not core. Every SDK runs core, including JavaScript and .NET, which backtrack and use other dialects. Earlier regex additions to core were reverted (Revert broken regex evaluator fixtures #18, revert: remove targeting regex conformance fixture #35). A consumer enables this suite only when its evaluator uses the Rust dialect and a linear-time engine.
  • One flag for each pattern. Cases cannot overlap under first-match semantics. This was the problem that Revert broken regex evaluator fixtures #18 fixed.
  • The expected results come from libdatadog. I ran every case through the libdd-ffe rules-based evaluator at libdatadog main@49d7bcaf8: 63 of 63 pass, and each case takes at most about 15 µs.
  • Syntax that the Rust dialect rejects gives PARSE_ERROR. This is what libdatadog does, and it agrees with the core case test-case-invalid-regex-isolation.json.
  • Some cases are left out on purpose, because RE2-family engines and the Rust crate do not agree on them:
    • a*+: Rust reads it as (a*)+; RE2 rejects it.
    • The x and R flags.
    • Unicode inputs for \d, \w, and \s.
    • The class operations && and --.
  • Sizes of the backtracking inputs. A linear-time engine finishes each case in microseconds. The JDK 17 engine does not finish .*.*.*.*.*= on 400 characters within 5 s, and it takes 364 ms for .*@(site-a|site-b)\.example$ on 20,000 characters. The JDK guards make ^(a+)+$ fast, but JavaScript and .NET do not have these guards.

Validation

  • python3 ci/validate-fixtures.py: 4 suites and 516 cases are valid.
  • python3 -m unittest discover -s ci -p 'test_*.py': 68 tests ran. 2 test modules did not import, because jsonschema is not installed locally. Those modules are for precomputed fixtures and response signing, which this change does not touch.
  • libdatadog libdd-ffe rules-based evaluator: 63 of 63 cases pass. I used a local probe test that is not part of this PR.
  • dd-openfeature (Java) on a local branch that uses literal fast paths and RE2/J: 63 of 63 cases pass.
  • Today's dd-openfeature logic (java.util.regex find()) differs on 9 cases:
    • It does not finish the .* chain within 5 s.
    • It matches $ before a final line feed, in 2 cases.
    • Its . does not match \r.
    • It matches 5 unsupported-syntax cases instead of giving PARSE_ERROR.

Add evaluation-suites/regex with 39 MATCHES and NOT_MATCHES flags and
63 cases in the dialect of the Rust regex crate: literal patterns,
dialect differences, inputs that make a backtracking engine slow, and
syntax that the dialect rejects. Document the suite in the README.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant