Repository navigation
Test every JDBC and Elasticsearch storage DAO and the Elasticsearch client on each request format, and fix what the tests found - #14252
Merged
Conversation
…lient on each request format, and fix what the tests found Storage DAO ITs bring JDBC and Elasticsearch to the BanyanDB suite's coverage: each IT installs the schema the way OAP does at boot, writes through OAP's write path, reads through the DAO the storage provider registers, and compares whole results with near misses, time bounds, paging and order. JDBC runs on PostgreSQL 13 and MariaDB 11.4. Elasticsearch runs on Elasticsearch 8.18.8 and 9.3.0 and OpenSearch 1.3.20, 2.8.0 and 3.0.0, and on Elasticsearch 9.3.0 and OpenSearch 3.0.0 with logic sharding and custom routing, as nested ITs of one IT per server. The Elasticsearch client IT sends every request OAP's storage sends to a server of each request format the client picks by version. The Elasticsearch e2e cases run on 8.18.8 and 9.3.0, and the mini-program e2e quotes log content so a log holding a quote no longer breaks the check. Fixes: - Elasticsearch client: escape document ids in request paths, so a runtime rule named like k8s/k8s-cluster can be written, read and deleted; apply indexTemplateOrder as a legacy template's order or a composable template's priority, and read a legacy template's name. - Elasticsearch: keep every component of a topology relation; read every JFR and pprof upload of a task, one per page; find a process by id in the default storage mode; filter events by endpoint, and include an event on the duration's first millisecond; read the recovery time of every alarm of a page. - Every storage: the process topology keeps every client-side component of a call. - Profiling tasks: Elasticsearch writes a row into the daily index of its bucket's day whatever the bucket's precision, and the TTL deletes the hour-named indices earlier versions left; a continuous-profiling task and the eBPF keep-alive copy carry their minute; BanyanDB task reads include the range's last bucket; the trace profiling conflict check reads the latest two tasks and compares both ends.
wu-sheng
force-pushed
the
test/jdbc-es-dao-coverage
branch
from
November 19, 2026 02:25
459e075 to
36d4966
Compare
wankai123
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test every JDBC and Elasticsearch storage DAO and the Elasticsearch client on each request format, and fix what the tests found
The BanyanDB storage has an IT for every DAO method OAP can reach (#14216). This PR brings the JDBC and Elasticsearch storages to the same coverage. It also tests the Elasticsearch client on a server of each request format it picks by version, and runs the Elasticsearch e2e cases on Elasticsearch 9. Each fix below has a test that failed before the fix and passes after it.
Storage DAO ITs. Each IT installs the schema the way OAP does at boot, writes through OAP's own write path, and reads through the DAO the storage provider registers. It compares whole results: near misses for every filter, rows on and just outside the time bounds, paging, order and limits. Rows sit on both sides of midnight, so a read crosses two day tables or two daily indices. Every DAO method of both storages is now called by an IT, and no IT is disabled.
JDBC<Area>ITBaseand a subclass per database.Elasticsearch client IT. It has one nested class per server version. Each sends every request OAP's storage sends, in the shape OAP sends it:
The versions cover each request format the client picks:
Each server starts once for all its tests, so the job takes about 3 minutes instead of 20.
e2e.
ES_VERSIONand runs on 8.18.8 and 9.3.0, adding 25 jobs. Before, only the storage case ran on 9.3.0. Outside CI the cases default to 8.18.8.contentwithprintf "%q". A log holding a'broke the expected YAML, and when it was the oldest log the verifier failed the case, about 1 run in 4.Fixes.
Elasticsearch client:
indexTemplateOrderreached the server only on Elasticsearch 6. A legacy template now sends itsorder, a composable template itspriority, and both are read back as 64-bit values. A template stored with another order is updated at boot. A legacy template keeps a negative order; a composable priority cannot be negative, so it takes 0./,?or#could not be written; examples are a runtime rule named likeaws-gateway/gateway-service, or the continuous-profiling policy of a service whose Base64 id holds/. The server decoded a percent escape, storinga%20basa b, which a read by id never found; such a document, written by an earlier version, is not found after the upgrade. Horizon's template names never hold these characters. A+still reaches the server as+, because the HTTP client decodes%2B, and Elasticsearch before 7.4 reads it as a space.Elasticsearch DAOs:
getProcessnever found a process in the default (merged) mode.Every storage:
ProcessTopologyBuilder).Profiling task rows:
dayStep> 1. The TTL now deletes those hour-named indices with their day, so an upgrade needs no manual cleanup. Async-profiler and pprof task rows keep the second buckets earlier versions wrote, so rows from before and after an upgrade are read the same way; an eBPF task that a continuous-profiling policy triggers now carries the minute it is created in, as its model's downsampling says.Accepted Elasticsearch behavior, documented at the aggregations.
sortMetrics) is a terms aggregation ordered byavg. The Zipkin and OTLP latest-first searches order bymindescending.shard_size, which issize * 1.5 + 10by default.Validation (local, JDK 11):
Elasticsearch DAO ITs: 1,981 run, 0 failed, 0 skipped.
JDBC DAO ITs: 515 run, 0 failed, 0 skipped.
BanyanDB ITs: 189 run, 0 failed.
Elasticsearch client IT: 108 of 108 on the nine servers.
Unit tests: core 426, eBPF receiver 13, JDBC 118, Elasticsearch 68, BanyanDB 73, Elasticsearch client 12, all passing. Checkstyle passes on every changed module.
The e2e changes run in CI.
Update the
CHANGESlog.