Repository navigation
Add an opt-in regex evaluation suite - #79
Draft
leoromanovsky wants to merge 1 commit into
Draft
leoromanovsky wants to merge 1 commit into
leoromanovsky wants to merge 1 commit into
Conversation
Add evaluation-suites/regex with 39 MATCHES and NOT_MATCHES flags and 63 cases in the dialect of the Rust regex crate: literal patterns, dialect differences, inputs that make a backtracking engine slow, and syntax that the dialect rejects. Document the suite in the README.
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
SDKs evaluate MATCHES and NOT_MATCHES with different regex engines. The libdatadog evaluator uses the Rust
regexcrate, which runs in linear time. Java usesjava.util.regex, which backtracks. A long attribute value can make a backtracking engine take milliseconds or more for one evaluation.This suite gives Rust and Java (and any other evaluator that moves to a linear-time engine with the Rust dialect) one shared set of expected results. It includes inputs that make a backtracking engine slow.
Changes
evaluation-suites/regex/: 39 flags and 63 cases in 4 files.literals.json: patterns that are only text, with anchors, escapes, a leading or trailing.*, and groups of literal alternatives.semantics.json: places where dialects differ:$before a final line feed,.and line feeds, them,s,i, anduflags, POSIX classes, counted repetition, Unicode word boundaries, and NOT_MATCHES with a missing attribute.backtracking.json: nested quantifiers, overlapping alternatives, a leading.*on a 20,000-character input, and chains of.*.unsupported-syntax.json: lookahead, lookbehind, backreferences, and atomic groups givePARSE_ERROR.README.md.Decisions
core. Every SDK runscore, including JavaScript and .NET, which backtrack and use other dialects. Earlier regex additions tocorewere reverted (Revert broken regex evaluator fixtures #18, revert: remove targeting regex conformance fixture #35). A consumer enables this suite only when its evaluator uses the Rust dialect and a linear-time engine.libdd-fferules-based evaluator at libdatadogmain@49d7bcaf8: 63 of 63 pass, and each case takes at most about 15 µs.PARSE_ERROR. This is what libdatadog does, and it agrees with the core casetest-case-invalid-regex-isolation.json.a*+: Rust reads it as(a*)+; RE2 rejects it.xandRflags.\d,\w, and\s.&&and--..*.*.*.*.*=on 400 characters within 5 s, and it takes 364 ms for.*@(site-a|site-b)\.example$on 20,000 characters. The JDK guards make^(a+)+$fast, but JavaScript and .NET do not have these guards.Validation
python3 ci/validate-fixtures.py: 4 suites and 516 cases are valid.python3 -m unittest discover -s ci -p 'test_*.py': 68 tests ran. 2 test modules did not import, becausejsonschemais not installed locally. Those modules are for precomputed fixtures and response signing, which this change does not touch.libdd-fferules-based evaluator: 63 of 63 cases pass. I used a local probe test that is not part of this PR.java.util.regexfind()) differs on 9 cases:.*chain within 5 s.$before a final line feed, in 2 cases..does not match\r.PARSE_ERROR.