Skip to content

SOLR-18585: intervals script support - #5224

Merged
mkhludnev merged 11 commits into
apache:mainfrom
mkhludnev:intervals-script
Oct 31, 2026
Merged

mkhludnev merged 11 commits into
apache:mainfrom
mkhludnev:intervals-script

Conversation

@mkhludnev

@mkhludnev mkhludnev commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

https://issues.apache.org/jira/browse/SOLR-18585
This pull request adds support for a new script filter operator to the Intervals Query Parser in Solr. This operator allows users to filter intervals using Lucene Expressions scripts, with access to interval-specific variables such as interval.start, interval.end, and interval.gap. The implementation includes backend support, comprehensive tests, and documentation updates.

Intervals Query Parser Enhancements:

  • Added support for the script filter operator in the Intervals Query Parser, enabling interval filtering based on Lucene Expressions evaluated with interval.start, interval.end, and interval.gap variables. [1] [2] [3]

Testing:

  • Added unit and distributed tests to verify the correct behavior of the script filter operator, including handling of valid scripts, combined variable expressions, invalid scripts, and unknown variables. [1] [2]

Documentation:

  • Updated the reference guide to document the new script filter operator, its usage, and examples illustrating how to use interval variables in expressions. [1] [2] [3]

Changelog:

  • Added a changelog entry describing the addition of the script filter operator to the Intervals QParser.

Description

Please provide a short description of the changes you're making with this pull request.

Solution

Please provide a short description of the approach taken to implement your solution.
We appreciate transparent disclosure of significant AI Coding assistants usage.

Tests

Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.

Checklist

Please review the following and check all that apply:

  • I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
  • I have created a Jira issue and added the issue ID to my pull request title.
  • I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
  • I have developed this patch against the main branch.
  • I have run ./gradlew check.
  • I have added tests for my changes.
  • I have added documentation for the Reference Guide
  • I have added a changelog entry for my change

@mkhludnev mkhludnev added this to the 10.x milestone Oct 24, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation tests cat:search labels Oct 24, 2026
@nick-boss-tech

nick-boss-tech commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Review TL/DR:

  • P2: Per-interval allocation overhead. accept() rebuilds the values array and adapters for every candidate interval. Reuse them per iterator to reduce search-loop work. Performance impact was not benchmarked.
  • P3: Incorrect documentation example. The “single position” example spans two positions. It also needs ordered: true if intended to match a phrase.
    No functional blocker found. All four GitHub checks are green; no local builds or tests were run.

Review details:
🤖 AI text below and above 🤖 (reviewed by Nick Shanin)

@mkhludnev I reviewed ef2072af3a01, including the parser validation and Lucene 10.4.0’s interval filtering and Matches paths. I did not find a functional correctness blocker in those paths. The [Gradle Precommit](https://github.com/apache/solr/actions/runs/40123734643) and [Solr Tests via Crave](https://github.com/apache/solr/actions/runs/40123734611) checks are green. This was a static review; I did not independently run tests or benchmark the implementation.

Two suggestions:

  1. Consider preparing the script’s value adapters once per iterator. In [ScriptFilterIntervalsSource.accept()](https://github.com/apache/solr/blob/ef2072af3a013bf335e621b49904619592ad53a2/solr/core/src/java/org/apache/solr/search/ScriptFilterIntervalsSource.java#L148-L164), each candidate interval creates a new DoubleValues[], resolves the bindings, and constructs one DoubleValues adapter per referenced variable. This also happens for intervals the script rejects. For the three-variable example, that is one array and three adapter constructions per candidate.

    Since Lucene calls this predicate inside its interval iteration loop, I would consider moving that setup outside accept() and reusing the adapters while they read the current iterator’s position. Any mutable evaluation state should belong to the individual iterator, including the Matches path, because the source can be shared by concurrent searches.

    The repeated setup is visible in the code, but its runtime cost is unmeasured, and the JVM may eliminate some allocations. I would treat this as a performance concern to assess rather than evidence of an observed slowdown.

  2. Correct the positional example in the guide. The [“spans a single position” example](

    Example — match a phrase only when it spans a single position:
    [source,json]
    ----
    {
    "match": {
    "query": "apache solr",
    "filter": {
    "script": { "source": "interval.end - interval.start == 1" }
    ) uses interval.end - interval.start == 1. With inclusive endpoints, that spans two adjacent positions. “Spans two adjacent positions” would describe it accurately. If the example is intended to match the phrase apache solr, it should also specify ordered: true; otherwise reversed adjacent tokens can match.

My recommendation is to correct the example here and consider reusing the adapters before merging. If retaining the current implementation keeps this change substantially simpler, a focused allocation benchmark would help decide whether the optimization belongs in a follow-up. Neither point calls for widening the script language or changing its semantics.

Would you prefer to address the adapter setup in this PR, or assess its cost separately and keep this change focused on the initial functionality?

@mkhludnev

Copy link
Copy Markdown
Member Author

@nick-boss-tech thanks for keen review.
🤖, please address both points!

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The matches implementation can incorrectly report a named query as matching when its script rejects every interval.

1 open issue
What changed in this PR

Adds a script filter to Solr’s Intervals Query Parser, allowing Lucene Expressions to select intervals by their positions and gaps.

Changes:

  • Implements script compilation, variable validation, and interval filtering.
  • Adds unit and distributed tests for expressions and error handling.
  • Documents the operator and records the feature in the changelog.
File Description
solr/​solr-ref-guide/​modules/​query-guide/​pages/​intervals-query-parser.adoc Documents script variables and usage examples.
solr/​core/​src/​test/​org/​apache/​solr/​search/​TestIntervalsQParserPlugin.java Tests variable filtering and invalid expressions.
solr/​core/​src/​test/​org/​apache/​solr/​search/​DistributedQParserTest.java Tests script filtering across shards.
solr/​core/​src/​java/​org/​apache/​solr/​search/​ScriptFilterIntervalsSource.java Implements expression-based interval and match filtering.
solr/​core/​src/​java/​org/​apache/​solr/​search/​IntervalsQParserPlugin.java Parses scripts and validates supported variables.
changelog/​unreleased/​intervals-script-filter.yml Records the new script filter feature.

Review effort: Balanced

mkhludnev and others added 2 commits October 25, 2026 01:15
Add logic to check for acceptance in ScriptFilterIntervalsSource.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new test class has unresolved imports and closes its index reader before consuming reader-backed iterators.

2 open issues
1 resolved since last review

Review effort: Balanced

mkhludnev and others added 2 commits October 25, 2026 09:26
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@mkhludnev
mkhludnev requested a balanced review from Copilot October 25, 2026 20:14

This comment was marked as resolved.

@mkhludnev
mkhludnev requested a balanced review from Copilot October 25, 2026 20:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@nick-boss-tech

Copy link
Copy Markdown
Contributor

Hi @mkhludnev,
🤖 AI text below 🤖 (reviewed by Nick Shanin)

The current helper closes its DirectoryReader before callers consume the returned iterator. I changed it to borrow a LeafReaderContext; each test now owns its reader and keeps it open through all iterator assertions. Only TestScriptFilterIntervalsSource.java changes. Production code and the existing assertions are unchanged.
I did not rerun the parent helper or demonstrate a failing-before case.

I recommend including this small change so the tests use reader-backed postings only while the reader is open. Would you be comfortable incorporating it here?
Test-only diff through 72a0daacd374

@mkhludnev

Copy link
Copy Markdown
Member Author

cherry=picked it, thanks @nick-boss-tech

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the expression-based interval filtering change.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@mkhludnev
mkhludnev merged commit 83d5600 into apache:main Oct 31, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cat:search documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants