Repository navigation
Make the YARA scanner asynchronous to avoid blocking the event loop #704
Copy link
Copy link
Open
Labels
needs: configIndicates the issue requires changes in the config file/flagsIndicates the issue requires changes in the config file/flagsscope: yaraAnything related to libyara and pattern matchingAnything related to libyara and pattern matchingtype: enhancement
Description
Motivation
The YARA scanner currently runs Scan() synchronously on the caller's goroutine, which means every scan, including the underlying libyara call executes inline on the ETW event processing path. This couples scan latency directly to event throughput: a slow or backed-up scan stalls the event loop rather than just delaying the scan result, which is a problem under high event volume or when scanning larger payloads (e.g. ADS reads).
Expectation
Scanning should no longer block event processing.
Scan()returns immediately after enqueuing the work; scan results reach consumers only through the existing alert-sender pipeline- Under load, once the pending-scan queue is full, new scan requests are dropped (not blocked), and the drop is observable via a metric
- The number of concurrent scan workers is bounded and configurable
- A
Wait()/Flush()mechanism allows draining in-flight scans deterministically (used by tests, and by shutdown)
Proposal
- Introduce a bounded queue (buffered, size controlled by a new AsyncQueueSize config option, default 1024).
Scan()does a non-blocking send. A full channel drops the event and increments ayara.dropped.scansmetric - Add a semaphore (
sem chan struct{}, capacity MaxWorkers, default 32, also configurable), gating how many scan goroutines run concurrently; a single dispatch() goroutine acquires a slot before launching each worker - Track every launched goroutine with a drainWg sync.WaitGroup so Close() can drain all in-flight work before destroying the YARA compiler, avoiding calls into libyara after
Destroy() - Extract the eligibility/setup logic currently inline in
Scan()into a function that runs synchronously on the caller's goroutine (fast path); only the actual yara.Scanner invocation moves into the worker pool. This includes making the ADSsys.ReadFileread synchronous (capped at 1MB with a 1s timeout) so scan bytes are captured before the event may be recycled - Protect the
rwxs/mmapsmaps with a mutex, since worker goroutines now write to them concurrently - Update the
Scan()interface contract: it currently returns (bool, error) with the bool indicating a match, which is incompatible with async execution. It should always return (false, nil), with match results delivered exclusively via the alert-sender pipeline - Add new config fields for
AsyncQueueSizeandMaxWorkers
Here is the proposed architecture diagram:

Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
needs: configIndicates the issue requires changes in the config file/flagsIndicates the issue requires changes in the config file/flagsscope: yaraAnything related to libyara and pattern matchingAnything related to libyara and pattern matchingtype: enhancement
Activity
[-]Make the YARA scanner asynchronous to avoid blocking the ETW event loop[/-][+]Make the YARA scanner asynchronous to avoid blocking the event loop[/+][-]Make the YARA scanner asynchronous to avoid blocking the event loop[/-][+]Make the `YARA` scanner asynchronous to avoid blocking the event loop[/+]