Skip to content

Fix index pruning YAML test flakiness #5813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dai-chen wants to merge 1 commit into
base: main
Choose a base branch
from
+29 −3
Open

Fix index pruning YAML test flakiness #5813

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions integ-test/src/yamlRestTest/resources/rest-api-spec/test/ppl/index_pruning.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Index pruning: https://github.com/opensearch-project/sql/issues/5698
#
# Asserted on point_in_time_total because no skipped shards or metrics are observable from the query
# response or the explain/profile API for now, and point_in_time_current is already gone by the time
# a non-paginated query responds.
# Assert point_in_time_total because skipped-shard counts and pruning metrics are not exposed in the
# query response or by explain/profile. When a PIT closes, point_in_time_current decrements and
# point_in_time_total increments. Since the plugin returns the response before closing its plan and
# PIT, tests explicitly delete all PITs before checking the total metric.
#
# Each index has one shard, so a query on the PIT path takes point_in_time_total from 0 to 1 on
# every index it reads: a 0 proves pruning excluded that index, a 1 only says it was still read.
Expand All @@ -14,6 +15,7 @@ setup:
- skip:
features:
- headers
- allowed_warnings
- do:
query.settings:
body:
Expand All @@ -22,6 +24,8 @@ setup:
# Without this a Calcite failure is silently answered by the v2 engine, which never prunes.
plugins.calcite.fallback.allowed: false
- do:
allowed_warnings:
- 'index template [pruning-it] has index patterns [pruning-it-*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [pruning-it] will take precedence during new index creation'
indices.put_index_template:
name: pruning-it
body:
Expand Down Expand Up @@ -81,6 +85,8 @@ teardown:
ppl:
body:
query: "source=pruning-it-* | where @timestamp >= '2026-02-01 00:00:00' and @timestamp <= '2026-02-28 23:59:59' | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-it-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand All @@ -96,6 +102,8 @@ teardown:
ppl:
body:
query: "source=pruning-it-* | where @timestamp >= '2026-01-01 00:00:00' and @timestamp <= '2026-02-28 23:59:59' | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-it-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand All @@ -111,6 +119,8 @@ teardown:
ppl:
body:
query: "source=pruning-it-* | where @timestamp >= '2026-02-01 00:00:00' | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-it-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand All @@ -126,6 +136,8 @@ teardown:
ppl:
body:
query: "source=pruning-it-* | where status = 2 | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-it-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand All @@ -141,6 +153,8 @@ teardown:
ppl:
body:
query: "source=pruning-it-* | where status >= 1 and status <= 100 | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-it-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand All @@ -161,6 +175,8 @@ teardown:
ppl:
body:
query: "source=pruning-it-* | where @timestamp >= '2026-02-01 00:00:00' and @timestamp <= '2026-02-28 23:59:59' | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-it-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand Down Expand Up @@ -188,6 +204,8 @@ teardown:
---
"Declines to prune a data stream":
- do:
allowed_warnings:
- 'index template [pruning-ds] has index patterns [pruning-ds-*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [pruning-ds] will take precedence during new index creation'
indices.put_index_template:
name: pruning-ds
body:
Expand Down Expand Up @@ -225,6 +243,8 @@ teardown:
ppl:
body:
query: "source=pruning-ds-* | where @timestamp >= '2026-01-01 00:00:00' and @timestamp <= '2026-01-31 23:59:59' | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-ds-*', metric: search, filter_path: '_all.total.search.point_in_time_total'}
- match: {_all.total.search.point_in_time_total: 2}
Expand Down Expand Up @@ -309,6 +329,8 @@ teardown:
query: "source=pruning-mixed-* | where @timestamp >= '2026-01-01 00:00:00' and @timestamp <= '2026-02-28 23:59:59' | sort @timestamp | fields body | head 5"
# One row from each type proves the pruned expression spanned the conflict, not one side of it.
- match: {datarows: [["january"], ["february"]]}
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-mixed-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand All @@ -324,6 +346,8 @@ teardown:
ppl:
body:
query: "source=pruning-it-2026-01,pruning-it-2026-0* | where @timestamp >= '2026-02-01 00:00:00' and @timestamp <= '2026-02-28 23:59:59' | head 5"
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-it-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand Down Expand Up @@ -360,6 +384,8 @@ teardown:
body:
query: "source=pruning-nomap-* | where @timestamp >= '2026-02-01 00:00:00' | fields body | head 5"
- match: {datarows: [["february"]]}
- do:
delete_all_pits: {}
- do:
indices.stats: {index: 'pruning-nomap-*', metric: search, filter_path: 'indices.*.total.search.point_in_time_total'}
- match:
Expand Down
Loading