Fixes collate#2985: bound the audit log count and index its filters

[mohityadav766]

Describe your changes:

Fixes open-metadata/openmetadata-collate#2985

GET /api/v1/audit/logs?limit=25&q=admin spent 150s in the database across 27 round trips:

Slow request detected - endpoint: v1/audit/logs, total: 150618ms,
db: 150538ms (99%), search: 0ms (0%), internal: 80ms (0%), dbOps: 27, searchOps: 0

dbOps: 27 decomposes exactly: 1 list + 1 count + 25 per-row entity-reference lookups.

#28851 already removed the full-row scan with a deferred join. This picks up the three items #28850 explicitly listed as "Follow-ups (not part of this fix)" and adds what the profile turned up alongside them.

What changed

1. The count is bounded at 10k rows. Paging here is keyset-based, so the exact total only renders a page count — but computing it visited every matching row on every request, including every keystroke of the debounced search box.
2. Filter predicates are emitted only when set. The (:param IS NULL OR col = :param) form kept all nine filters in the plan, so none of the (col, event_ts DESC) composite indexes was reachable. entityFQN also filtered on the unindexed entity_fqn column while its indexed hash sat unused.
3. No more per-row lookup. A missing entity FQN is read from the stored change event, or from the acting user for auth events, instead of one query per row. userLogin/userLogout rows — exactly what q=admin returns — always had a null entity_fqn. Auth events now record the FQN at write time.
4. Export counts once, bounded by the export limit, instead of once per batch (a 100k export ran ~101 counts), and its 1000-row batches are no longer silently capped to 200 by the page-size guard.
5. isBot lookups are cached (1000 entries / 10 min), so the write path stops querying the user table once per change event.

Migration. 1.12.1's ALTER TABLE audit_log_event ADD COLUMN search_text has no IF NOT EXISTS, so on any deployment where the column already existed the script aborted before CREATE FULLTEXT INDEX — and every q= search since has been a full table scan. The migration recreates it idempotently, and adds the (event_type, event_ts DESC) / (entity_type, event_ts DESC) indexes #28850 called for.

Measured — 400k rows, MySQL 8.0.46 and PostgreSQL 15.19, plans confirmed with EXPLAIN ANALYZE / EXPLAIN FORMAT=TREE:

	Postgres	MySQL
count, q=admin + time filter	744ms → 30ms	594ms → 155ms
count, eventType filter	—	65ms → 1.4ms
count, unfiltered	—	36ms → 0.9ms
entityFQN filter	19ms → 0.16ms	120ms → 0.19ms
db round trips / request	27 → 2	27 → 2

Known limitation, not addressed here. On MySQL the list query for a common search term is still ~580ms at 400k rows and is what scales to the 150s in the report. MySQL always drives from the FULLTEXT index and sorts the whole match set; FORCE INDEX (idx_audit_log_event_ts) makes common terms 5.5× faster (104ms) but makes rare terms ~10× slower, so there is no safe hint. The real fix is indexing audit events into the search cluster — note searchOps: 0 in the report, the cluster is idle. Happy to open a follow-up issue.

Review response. A first revision also changed agent detection to match whole name parts, justified by a claim that "management".contains("agent"). That is false — management is managem-ent — so there was nothing to fix, and the change regressed real cases (AutoClassificationAgent lowercases to a single token and stopped being detected). Reverted in d86a49a; agent classification was never part of the latency fix.

Type of change:

• Improvement

High-level design:

The list path builds its WHERE clause as a string that JDBI splices into <condition>, so making the filters conditional is a change to buildBaseCondition alone — every caller (list, count, export) inherits it. Verified that JDBI tolerates the now-unused bindings: it only rejects superfluous named parameters when the query declares zero, and :limit / :countLimit always survive. Confirmed empirically against a running Postgres through the real AuditLogDAO, not just by reading ArgumentBinder.

The bounded count is a derived table with a LIMIT, which both engines plan as "stop after N matching rows":

SELECT COUNT(*) FROM (SELECT 1 FROM audit_log_event <condition> LIMIT :countLimit) bounded

list() splits into a public wrapper that counts and a private one that does not, so the export loop stops paying for a count per batch. The wrapper is also where sanitizeLimit's 200-row cap now lives, which is what un-caps the export's 1000-row batches.

Alternatives rejected:

• Dropping total entirely. Cursor paging does not need it, but NextPrevious renders a page count from paging.total. Bounding it keeps the control working and degrades to currentPage + 1, which is what the component already falls back to.
• A time window defaulted onto q. Would bound the MySQL scan and is how every audit product behaves, but it silently changes what a search returns. Left for a product call.
• FORCE INDEX on the list query. Helps common terms, hurts rare ones by ~10×. Not a trade worth hard-coding.
• A read-time LRU for entity references. Unnecessary once the FQN is read from the change event that is already deserialized on that line.

Compatibility. paging.total now saturates at 10 000 — the one user-visible change. Include-style filtering, cursors, and ordering are untouched. The migration is idempotent on both engines and safe on deployments that already have the indexes.

Tests:

Use cases covered

• Listing audit logs with no filters, with a single filter, and with a cursor emits only the predicates that are set
• Filtering by entityFQN routes through the indexed entity_fqn_hash
• The reported total is bounded rather than exact
• A page of userLogin events resolves its entity FQN without a per-row database lookup
• Re-running the 2.1.0 migration on a table that already has the indexes is a no-op

Unit tests

• I added unit tests for the new/changed logic.
• Files updated: openmetadata-service/src/test/java/org/openmetadata/service/audit/AuditLogRepositoryTest.java (+4 tests)
• Coverage on AuditLogRepository.java, from unit tests only: 23.0% → 44.8% line, 12.1% → 30.0% branch (mvn -P static-code-analysis jacoco:prepare-agent test jacoco:report, measured with a clean jacoco.exec on each side)
• Below the 90% target: the remainder of the class is the CSV/export path, cursor decoding and summary formatting, which are exercised through the API by AuditLogResourceIT rather than by unit tests.
• AuditLogRepositoryTest 7/7, AuditLogConsumerTest 22/22 pass.

Backend integration tests

• Not applicable — no new or changed API endpoint; request and response shapes are unchanged apart from paging.total saturating.
• The riskiest change here (entity_fqn → entity_fqn_hash) is already covered: AuditLogResourceIT.waitForAuditLogEntryMultipleTypes hard-fail()s if an entityFQN + entityType + eventType query returns nothing, and many tests route through it. CI's integration suites passed on mysql-elasticsearch, postgres-elasticsearch-redis and postgres-opensearch.
• The 10k ceiling is not IT-testable at a reasonable fixture size.

Ingestion integration tests

• Not applicable — no ingestion changes.

Playwright (UI) tests

• Not applicable — no UI changes.

Manual testing performed

Benchmarked against the dev Docker databases rather than the full stack:

1. docker compose -f docker/development/docker-compose.yml up -d (MySQL 8.0.46, PostgreSQL 15.19)
2. Loaded a 400k-row audit_log_event with the production index set, ~1/3 of rows attributed to admin and 1 in 5 a userLogin
3. Ran each old and new query shape under EXPLAIN ANALYZE (Postgres) and EXPLAIN FORMAT=TREE + timed execution (MySQL), 2–3 warm repetitions each — numbers in the table above
4. Exercised AuditLogDAO.list / .count through real JDBI against Postgres to confirm the unused bindings and the derived-table count both work outside of mocks
5. Applied the 2.1.0 migration block twice on a fresh table on both engines, confirming all three indexes appear and the second pass is a no-op
6. mvn spotless:check clean; recompiled and re-ran the audit tests after rebasing onto current main

Not done: I did not click through the audit log page in a running UI. The one thing worth eyeballing there is the page count now saturating at 400 pages.

UI screen recording / screenshots:

Not applicable — no UI changes.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: not applicable — no schema changes. Migration scripts added for the index changes.
• For UI changes: not applicable.
• I have added tests (unit / integration / Playwright as applicable) and listed them above.

• I have added tests around the new logic.
• For connector/ingestion changes: not applicable.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue open-metadata/openmetadata-collate#2985 does not exist or is not accessible.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[Copilot]

🟢 Approval recommended

The changes are coherent with the stated performance goals, update both DAO + callers consistently, include targeted unit test coverage for the new query-shaping logic, and migrations are idempotent on both engines.

Pull request overview

This PR improves audit log API performance and scalability by reducing database work in the list/count/export paths and by adding missing audit-log indexes via migrations, targeting the latency + round-trip issues described in collate#2985.

Changes:

• Replace the unbounded COUNT with a bounded count (ceiling 10k for list, bounded by export limit for exports) to avoid scanning the full match set on each request.
• Emit SQL filter predicates only when corresponding filters are set (and route entityFQN filtering through entity_fqn_hash) to make composite indexes usable.
• Remove per-row entity reference lookups by resolving missing FQNs from stored change events / auth event context, and cache isBot lookups on the write path.

File summaries

File	Description
openmetadata-service/src/main/java/org/openmetadata/service/audit/AuditLogRepository.java	Implements bounded counts, conditional predicate emission, avoids per-row lookups, optimizes export batching, and adds bounded isBot caching + improved agent detection.
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/ActivityAuditDAOs.java	Updates the DAO count query to a bounded derived-table count and threads countLimit through the DAO contract.
openmetadata-service/src/test/java/org/openmetadata/service/audit/AuditLogRepositoryTest.java	Adds unit tests covering conditional predicate emission, hash-based entityFQN filtering, bounded totals, and auth-event FQN resolution without per-row lookups.
bootstrap/sql/migrations/native/2.1.0/mysql/schemaChanges.sql	Adds idempotent creation of the missing FULLTEXT index and adds composite indexes for (event_type, event_ts) and (entity_type, event_ts) on MySQL.
bootstrap/sql/migrations/native/2.1.0/postgres/schemaChanges.sql	Adds idempotent GIN full-text index on search_text and composite indexes for (event_type, event_ts) and (entity_type, event_ts) on Postgres.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit a9806b03f96f2ca63af4dc7ed695cad5c28f23bc in Playwright run 34340378148, attempt 1.

✅ 4479 passed · ❌ 0 failed · 🟡 9 flaky · ⏭️ 1 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 56m 30s

⏱️ Max setup 4m 23s · max shard execution 20m 18s · max shard-job elapsed before upload 24m 33s · reporting 19s

🌐 217.93 requests/attempt · 2.31 app boots/UI scenario · 36.86% common-shard skew

Optimization targets still in progress:

• Common shard skew was 36.86% (convergence target: at most 15%).
• Browser traffic was 217.93 requests per attempt (convergence target: fewer than 200).
• Application boot ratio was 2.31 per UI scenario (10932 boots / 4739 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard advanced-search-01	130	0	0	0	0	0
✅ Shard chromium-01	153	0	0	0	0	0
✅ Shard chromium-02	152	0	0	0	0	0
✅ Shard chromium-03	187	0	0	0	0	0
✅ Shard chromium-04	121	0	0	0	0	0
✅ Shard chromium-05	130	0	0	0	0	0
✅ Shard chromium-06	139	0	0	0	0	0
🟡 Shard chromium-07	158	0	1	0	0	0
🟡 Shard chromium-08	148	0	2	0	0	0
✅ Shard chromium-09	144	0	0	1	0	0
🟡 Shard chromium-10	120	0	1	0	0	0
✅ Shard chromium-11	166	0	0	0	0	0
✅ Shard chromium-12	170	0	0	0	0	0
✅ Shard chromium-13	200	0	0	0	0	0
🟡 Shard chromium-14	145	0	1	0	0	0
🟡 Shard chromium-15	162	0	1	0	0	0
✅ Shard chromium-16	184	0	0	0	0	0
✅ Shard chromium-17	177	0	0	0	0	0
✅ Shard chromium-18	167	0	0	0	0	0
✅ Shard chromium-19	177	0	0	0	0	0
✅ Shard chromium-20	167	0	0	0	0	0
✅ Shard chromium-21	154	0	0	0	0	0
🟡 Shard chromium-22	138	0	1	0	0	0
✅ Shard chromium-23	144	0	0	0	0	0
🟡 Shard chromium-24	172	0	1	0	0	0
🟡 Shard chromium-25	154	0	1	0	0	0
✅ Shard data-asset-rules-01	65	0	0	0	0	0
✅ Shard domain-isolation-01	16	0	0	0	0	0
✅ Shard global-state-01	34	0	0	0	0	0
✅ Shard import-export-01	43	0	0	0	0	0
✅ Shard import-export-02	107	0	0	0	0	0
✅ Shard ingestion-01	35	0	0	0	0	0
✅ Shard ingestion-02	51	0	0	0	0	0
✅ Shard reindex-01	28	0	0	0	0	0
✅ Shard search-01	12	0	0	0	0	0
✅ Shard search-rbac-01	29	0	0	0	0	0

🟡 9 flaky test(s) (passed on retry)

• Pages/TasksUIFlow.spec.ts › Create and resolve description task for Pipeline via UI (shard chromium-07, 1 retry)
• Pages/ExplorePageRightPanel.spec.ts › Should verify deleted user not visible in owner selection for table (shard chromium-08, 1 retry)
• Pages/ExplorePageRightPanel.spec.ts › Should verify deleted tag not visible in tag selection for databaseSchema (shard chromium-08, 1 retry)
• Pages/SubDomainPagination.spec.ts › Verify subdomain count and pagination functionality (shard chromium-10, 1 retry)
• VersionPages/ServiceEntityVersionPage.spec.ts › Database (shard chromium-14, 1 retry)
• Pages/EntityDataConsumer.spec.ts › Tier Add, Update and Remove (shard chromium-15, 1 retry)
• Features/Glossary/GlossaryWorkflow.spec.ts › should delete parent term and cascade delete children (shard chromium-22, 1 retry)
• Pages/DescriptionVisibility.spec.ts › Customized Table detail page Description widget shows long description (shard chromium-24, 1 retry)
• Flow/ExploreAggregationCountsMatching.spec.ts › should verify left panel counts and tab search results for normal search (shard chromium-25, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Fixes a critical audit log performance issue where GET /api/v1/audit/logs with filters was spending 150s in the database across 27 round trips. The fix bounds the audit log count at 10k rows, emits filter predicates only when set to unblock composite indexes, eliminates per-row entity-reference lookups by reading FQNs from stored change events, and caches isBot lookups. Measured improvements: count queries drop from 744ms to 30ms (Postgres) and 594ms to 155ms (MySQL), and database round trips per request reduce from 27 to 2. The agent detection regression introduced in an earlier revision has been reverted. No issues found.

✅ 1 resolved

✅ Edge Case: Agent detection now misses concatenated/camelCase names

📄 openmetadata-service/src/main/java/org/openmetadata/service/audit/AuditLogRepository.java:195-200
Switching from lowerName.contains(indicator) to splitting on [^a-z0-9]+ and matching whole parts means a username with an indicator embedded without a separator is no longer classified — e.g. "AutoClassificationAgent" lowercases to a single token "autoclassificationagent" and matches none of {agent, documentation, classification, automator}, so it is logged as USER instead of AGENT. The previous substring form caught these. Also, the justifying comment's example is factually wrong: "management".contains("agent") is false (management has no "agent" substring), so the stated over-match motivation doesn't hold. If agent/app users are always separator-delimited (e.g. "ingestion-bot") the whole-part approach is fine; otherwise consider retaining a substring fallback or splitting camelCase, and correct the comment.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}