Skip to content

Test every JDBC and Elasticsearch storage DAO and the Elasticsearch client on each request format, and fix what the tests found - #14252

Merged
wu-sheng merged 1 commit into
masterfrom
test/jdbc-es-dao-coverage
Nov 19, 2026
Merged

wu-sheng merged 1 commit into
masterfrom
test/jdbc-es-dao-coverage

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Test every JDBC and Elasticsearch storage DAO and the Elasticsearch client on each request format, and fix what the tests found

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

The BanyanDB storage has an IT for every DAO method OAP can reach (#14216). This PR brings the JDBC and Elasticsearch storages to the same coverage. It also tests the Elasticsearch client on a server of each request format it picks by version, and runs the Elasticsearch e2e cases on Elasticsearch 9. Each fix below has a test that failed before the fix and passes after it.

Storage DAO ITs. Each IT installs the schema the way OAP does at boot, writes through OAP's own write path, and reads through the DAO the storage provider registers. It compares whole results: near misses for every filter, rows on and just outside the time bounds, paging, order and limits. Rows sit on both sides of midnight, so a read crosses two day tables or two daily indices. Every DAO method of both storages is now called by an IT, and no IT is disabled.

  • JDBC: PostgreSQL 13 and MariaDB 11.4, with one JDBC<Area>ITBase and a subclass per database.
  • Elasticsearch: one IT per server, with each of its 21 areas as a nested class, so each server starts once and each area writes under its own index namespace.
    • Servers: Elasticsearch 8.18.8 and 9.3.0, OpenSearch 1.3.20, 2.8.0 and 3.0.0, and Elasticsearch 9.3.0 and OpenSearch 3.0.0 with logic sharding and custom routing.
    • Writes refresh their indices.
    • A new installer, with nothing cached, must find every model an IT installs, as on OAP's next boot.
    • The Zipkin and span-attached-event ITs moved into this layout.

Elasticsearch client IT. It has one nested class per server version. Each sends every request OAP's storage sends, in the shape OAP sends it:

  • index templates with their alias and order;
  • indices and appended mappings, read back for the installer's comparison;
  • refreshed single-document writes, with ids that need escaping in a path;
  • bulk inserts, updates and routed inserts;
  • every query and aggregation shape the DAOs build;
  • routed searches;
  • searches over day indices of which some are missing;
  • scrolls, with their context cleared.

The versions cover each request format the client picks:

Client request format Versions
Elasticsearch 6 6.8.23 (added)
Elasticsearch 7.0-7.7, legacy index templates 7.7.1 (was 7.4.2)
Elasticsearch 7.8-7.x, composable index templates 7.8.0, 7.17.29 (was 7.15.0)
Elasticsearch 8 and later 8.19.25, 9.3.0 (8.9.0 removed)
OpenSearch 1.3.20, 2.8.0, 3.0.0 (3.0.0 added; 2.4.0 removed)

Each server starts once for all its tests, so the job takes about 3 minutes instead of 20.

e2e.

  • Every case that stores in Elasticsearch now takes ES_VERSION and runs on 8.18.8 and 9.3.0, adding 25 jobs. Before, only the storage case ran on 9.3.0. Outside CI the cases default to 8.18.8.
  • The OTLP native trace case keeps 7.15.0 as well.
  • The Elasticsearch monitoring case does not store in Elasticsearch and stays on 8.18.8.
  • The mini-program log expectations now quote content with printf "%q". A log holding a ' broke the expected YAML, and when it was the oldest log the verifier failed the case, about 1 run in 4.

Fixes.

Elasticsearch client:

  • indexTemplateOrder reached the server only on Elasticsearch 6. A legacy template now sends its order, a composable template its priority, and both are read back as 64-bit values. A template stored with another order is updated at boot. A legacy template keeps a negative order; a composable priority cannot be negative, so it takes 0.
  • Document ids went into request paths unescaped. An id holding /, ? or # could not be written; examples are a runtime rule named like aws-gateway/gateway-service, or the continuous-profiling policy of a service whose Base64 id holds /. The server decoded a percent escape, storing a%20b as a b, which a read by id never found; such a document, written by an earlier version, is not found after the upgrade. Horizon's template names never hold these characters. A + still reaches the server as +, because the HTTP client decodes %2B, and Elasticsearch before 7.4 reads it as a space.
  • A legacy template was read back without its name.

Elasticsearch DAOs:

  • Topology kept one component of a relation whose component changed within the range.
  • The JFR and pprof data reads stopped at the default 10 hits. They now read every upload, one per page, so several uploads together never exceed the client's 10 MiB response limit.
  • getProcess never found a process in the default (merged) mode.
  • The event query's endpoint filter used a field that the event mapping does not have, and an event on the duration's first millisecond was left out.
  • Alarm recovery times were lost past the 10th alarm of a page.

Every storage:

  • The process topology kept only one client-side component of a call (ProcessTopologyBuilder).

Profiling task rows:

  • Elasticsearch now writes a row into the daily index of its time bucket's day, whatever the bucket's precision. Async-profiler, pprof and continuous-profiling tasks carry a second-precision bucket on a minute model, so they went into an index per hour that the TTL never deleted, and could not be written at all with dayStep > 1. The TTL now deletes those hour-named indices with their day, so an upgrade needs no manual cleanup. Async-profiler and pprof task rows keep the second buckets earlier versions wrote, so rows from before and after an upgrade are read the same way; an eBPF task that a continuous-profiling policy triggers now carries the minute it is created in, as its model's downsampling says.
  • The eBPF keep-alive task copy carried time bucket 0, an index dated 0 that the next TTL run deleted. It now carries the minute it is written.
  • BanyanDB's task reads missed a task in a range's last time bucket.
  • The trace profiling conflict check read the latest task by whole minutes, so on JDBC and Elasticsearch, and now on BanyanDB, it refused a new task when another starts later in the minute the new one ends. It now reads the latest two tasks and compares both ends; a service's tasks never overlap and last a minute or more, so at most one starts in a minute.

Accepted Elasticsearch behavior, documented at the aggregations.

  • Top-N (sortMetrics) is a terms aggregation ordered by avg. The Zipkin and OTLP latest-first searches order by min descending.
  • Elastic documents that sorting terms by a sub-aggregation "generally produces incorrect ordering" across shards. Each shard returns its own top shard_size, which is size * 1.5 + 10 by default.
  • It is exact when one shard holds the data. Across several indices or shards, an entity can be missed or ranked by part of its data.
  • Reading every entity, or making extra requests, was judged not worth the cost.
  • The ES ITs assert what the servers return for these cases. The JDBC ITs assert the exact answer for the same fixtures.

Validation (local, JDK 11):

  • Elasticsearch DAO ITs: 1,981 run, 0 failed, 0 skipped.

  • JDBC DAO ITs: 515 run, 0 failed, 0 skipped.

  • BanyanDB ITs: 189 run, 0 failed.

  • Elasticsearch client IT: 108 of 108 on the nine servers.

  • Unit tests: core 426, eBPF receiver 13, JDBC 118, Elasticsearch 68, BanyanDB 73, Elasticsearch client 12, all passing. Checkstyle passes on every changed module.

  • The e2e changes run in CI.

  • Update the CHANGES log.

@wu-sheng wu-sheng added this to the 11.2.0 milestone Nov 18, 2026
@wu-sheng wu-sheng added bug Something isn't working and you are sure it's a bug! test Test requirements about performance, feature or before release. labels Nov 18, 2026
…lient on each request format, and fix what the tests found

Storage DAO ITs bring JDBC and Elasticsearch to the BanyanDB suite's
coverage: each IT installs the schema the way OAP does at boot, writes
through OAP's write path, reads through the DAO the storage provider
registers, and compares whole results with near misses, time bounds,
paging and order. JDBC runs on PostgreSQL 13 and MariaDB 11.4.
Elasticsearch runs on Elasticsearch 8.18.8 and 9.3.0 and OpenSearch
1.3.20, 2.8.0 and 3.0.0, and on Elasticsearch 9.3.0 and OpenSearch 3.0.0
with logic sharding and custom routing, as nested ITs of one IT per server.

The Elasticsearch client IT sends every request OAP's storage sends to a
server of each request format the client picks by version. The
Elasticsearch e2e cases run on 8.18.8 and 9.3.0, and the mini-program e2e
quotes log content so a log holding a quote no longer breaks the check.

Fixes:
- Elasticsearch client: escape document ids in request paths, so a
  runtime rule named like k8s/k8s-cluster can be written, read and
  deleted; apply indexTemplateOrder as a legacy template's order or a
  composable template's priority, and read a legacy template's name.
- Elasticsearch: keep every component of a topology relation; read every
  JFR and pprof upload of a task, one per page; find a process by id in
  the default storage mode; filter events by endpoint, and include an
  event on the duration's first millisecond; read the recovery time of
  every alarm of a page.
- Every storage: the process topology keeps every client-side component
  of a call.
- Profiling tasks: Elasticsearch writes a row into the daily index of its
  bucket's day whatever the bucket's precision, and the TTL deletes the
  hour-named indices earlier versions left; a continuous-profiling task
  and the eBPF keep-alive copy carry their minute; BanyanDB task reads
  include the range's last bucket; the trace profiling conflict check
  reads the latest two tasks and compares both ends.
@wu-sheng
wu-sheng force-pushed the test/jdbc-es-dao-coverage branch from 459e075 to 36d4966 Compare November 19, 2026 02:25
@wu-sheng
wu-sheng merged commit 11eb86d into master Nov 19, 2026
539 of 543 checks passed
@wu-sheng
wu-sheng deleted the test/jdbc-es-dao-coverage branch November 19, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working and you are sure it's a bug! test Test requirements about performance, feature or before release.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants