Skip to content

Unify DSL source attribution across OAL, MAL, LAL and Hierarchy - #13972

Merged
wu-sheng merged 1 commit into
masterfrom
feat/dsl-sourcefile-provenance
Aug 10, 2026
Merged

Unify DSL source attribution across OAL, MAL, LAL and Hierarchy#13972
wu-sheng merged 1 commit into
masterfrom
feat/dsl-sourcefile-provenance

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Aug 10, 2026

Copy link
Copy Markdown
Member

Unify DSL source attribution across OAL, MAL, LAL and Hierarchy

OAP compiles four DSLs to bytecode at runtime — OAL, MAL, LAL and Hierarchy. Each one
independently grew its own answer to the same question: when a generated class shows up in a
stack frame, what does it say it came from?
The workflows are near-identical (rule file + line →
generated class name, SourceFile, LineNumberTable, optional .java dump), but because each
compiler implemented it separately, they drifted — differently, and silently.

This PR extracts that one workflow into org.apache.skywalking.oap.server.core.dsl and makes all
four use it.

What a frame said before, and says now

The .java sidecar is written only under SW_DYNAMIC_CLASS_ENGINE_DEBUG; Javassist compiles
from an in-memory string, so in a released image there is no .java on disk — in the container,
in a binary package, or in an IDE. A SourceFile naming only the generated class therefore
addressed nothing an operator could open.

before after
MAL rule vm_L25_cpu_total.java (otel-rules/vm.yaml:25)vm_L25_cpu_total.java
LAL rule default_L3_default.java (lal/default.yaml:3)default_L3_default.java
Hierarchy rule hierarchy_definition_Lunknown_name.java (hierarchy-definition.yml:2)hierarchy_definition_L2_name.java
OAL metric ServiceRespTimeMetrics.java (core.oal:20)ServiceRespTimeMetrics.java
Zabbix rule (no coordinate at all) (zabbix-rules/agent.yaml:20)agent_L20_cpu.java

The rule leads, then the generated file name, so the coordinate survives whether or not the
sidecar exists.

The shared kernel

New, in server-core:

  • DslSourceRef — the value object for one coordinate. The only parser of "file:line",
    the only renderer of (file:line)Class.java, and the only builder of the _L<line>_ class-name
    segment. Promoted from MAL's MalSourceRef, which had been written but never instantiated.
  • DslGeneratedFileWriter — all bytecode/disk side effects: writeClassFile,
    writeSourceFile, setSourceFile, attachSignatureLine, addLocalVariableTable,
    lineOfMethod. Split out of DslSourceRef so the value object performs no I/O.
  • DslClassNamingstem / allocate. The stem is shared and deterministic; the
    allocation policy stays with each generator, because it genuinely differs (MAL and LAL give
    each runtime-rule apply its own classloader and must not dedup process-wide; Hierarchy defines
    into the shared loader and must).
  • DslJavaSourceTexttoIdentifier / toLiteral: making arbitrary rule text safe to embed
    in generated Java source, in identifier position and in string-literal position.
  • DslYamlLineIndex — moved up from meter-analyzer (was MalYamlLineIndex), now serving
    MAL, LAL and Zabbix by rule ordinal, plus a new keyLines for Hierarchy, whose rules are a YAML
    mapping rather than a sequence and so cannot be matched positionally.

What actually collapsed (counted against master):

duplicated logic before after
"file:line" parsers 3 1
SourceFile string built + attribute written 4 1
_L<line>_ class-name builders 3 1
identifier sanitisers 3 1
Java-literal escapers 4 1
process-wide dedup sets 3 1
.java source-file writers 3 1
.class file writers 4 1
LocalVariableTable writers 3 1
fixed line-offset constants 1 0

The remaining per-generator wrappers that only forwarded to the shared class were removed too, so
each generator now calls it directly — a wrapper that adds a hop without value would have made the
consolidation nominal rather than real. Three more copies went the same way:

  • escapeJava was quadruplicated, char-identical in MAL, LAL and Hierarchy, with a
    null-tolerant fourth in OAL behind a "kept local so this generator stays self-contained" comment
    that its own imports had already falsified. Deliberately not replaced by commons-text
    StringEscapeUtils.escapeJava: that also escapes non-ASCII to a u-prefixed hex escape, and
    Javassist has no unicode-escape pre-lex phase the way javac does, so a rule with a non-ASCII tag
    key would compile with the escape taken literally.
  • sanitizeName left residue after the migration — LAL's copy was dead, MAL's had one live
    caller, and that caller names a generated variable, not a class.
  • Both therefore landed in a new DslJavaSourceText (toIdentifier / toLiteral) rather than in
    DslClassNaming: they are one concern — embedding arbitrary rule text in generated Java source —
    with two positions, and neither is class naming. DslClassNaming keeps stem and allocate.
  • The two remaining *.v2.dsldebug packages became *.v2.dsl.debug, matching
    core.dsl.debug. They stay in their modules: the dependency direction is one-way, so moving them
    into server-core would create a cycle. Two of the references are string literals in codegen,
    which no compiler would have caught.
  • readAll in HierarchyDefinitionService hand-rolled a buffer loop while every other rule
    loader in this PR reads new String(bytes, UTF_8). Now the same idiom.

Loader fixes — the coordinates had to exist before they could be unified

Three loaders never stamped what the generators were about to read:

  • MAL (Rules.parseRule) — nested rules lost their ruleset directory, and a .yml file was
    reported as .yaml. Now stamps rulesetDir + "/" + relPath with the real extension. Rule
    needed an explicit getSourcePath() because Lombok @Data was generating a getter that
    shadowed the interface default.
  • LAL (LALConfigs) — resolved no line at all; now indexes the rules: sequence and stamps a
    catalog-qualified lal/<file>. The runtime-rule applier (LalFileApplier) stamps lineNo too,
    so a hot-updated rule is attributed the same as a bundled one.
  • Zabbix (ZabbixConfigs) — ZabbixConfig implements MetricRuleConfig but took every
    default, so getSourceName() was null and its classes carried neither file nor line. Now stamps
    both, indexing under zabbix's own metrics: key.

Identity and attribution are separate fields. sourcePath is the catalog-qualified path a
generated class names; sourceName is the rule file's identity, and on the boot route it becomes
the middle component of the dsl-debugging RuleKey (LAL, sourceName, ruleName). An earlier
revision of this PR overloaded one field for both, catalog-qualifying sourceName for attribution
and thereby re-keying every static binding. They are now split, both derived through one
LALConfigs.stampSource, and MetricConvert guards on the field it actually passes rather than
its sibling.

Bug fix: a hot-updated LAL rule stranded its own debug binding

Separately from the fields above, the dsl-debugging RuleKey (catalog, name, ruleName) names a
rule FILE in its middle component — and the two routes that publish it disagreed on the extension.
The boot loader published (LAL, "default.yaml", rule); the runtime-rule engine
(LalRuleEngine.publishDebugBindings, which never reads sourceName — it builds its own key from
the rule's bare name) published (LAL, "default", rule). Both land in the same
LALHolderRegistry map, so the two keys simply never met: a hot update added a second binding
instead of replacing the first, unpublishDebugBindings could never remove the first either, and
an operator addressing the older spelling enabled probes on a GateHolder belonging to a compiled
rule that no longer evaluates anything — indistinguishable from a rule with no traffic. No
exception, no log line.

RuleKey now drops a trailing .yaml/.yml from that component, in its constructor rather than
at each publish site: normalising per-site leaves the next site free to get it wrong, and doing it
in the key also lets the REST API keep accepting both spellings, so no existing operator script
breaks. Only a YAML extension is stripped — OAL files stay core.oal, nested MAL bundles stay
activemq/activemq-broker, and vm.linux.yaml keeps its vm.linux.

Rule execution was never affected. The maps that decide which rules run are keyed by layer and
rule name (LogFilterListener.Factory.dsls) and contain no file name at all; this was confined to
the debugging registry. The mismatch predates this PR — MAL was never affected, because both its
routes already publish the bare name.

Per-method line numbers

LineNumberTable entries now point at a method's own signature line in the generated source
(attachSignatureLine + lineOfMethod), for OAL metrics/dispatcher methods, LAL's execute and
private methods, and MAL companions.

Per-statement tables are deliberately not emitted outside MAL's expression codegen. The scan
that produces them marks boundaries at stores to a result slot; LAL and OAL bodies are largely
void invocations that store nothing there, so the numbers came out as statement ordinals
(1, 2, 3…) — not lines in any file. Emitting a wrong line is worse than emitting none.

Line lookup: an identifier boundary, and two hand-counted envelopes deleted

lineOfMethod matched a declaration with contains(name + "("), which also accepts a longer
method whose name ENDS with the wanted one. serialize is a suffix of deserialize and OAL
declares both on every metrics class; it resolved correctly only because the template list happens
to emit the shorter one first, and nothing said that ordering was load-bearing. Operator-authored
OAL reaches the same shape — metrics cpm and commando_cpm in one scope give doCpm and
doCommandoCpm on the shared dispatcher. The match now requires a non-identifier character before
the name.

MAL still computed two signature lines by counting its own source envelope: 10 + closures + injection for run(), and a bare 9 for a filter class. Both are gone, replaced by the same
search. The filter constant had no artifact-resolving test at all — the branch that varied the
other one (SW_DSL_DEBUGGING_INJECTION_ENABLED) never runs in CI — so an envelope edit shifted
every filter frame silently. A guard was needed with the change: statement lines are counted
forward from the signature, so an unresolved signature (-1) turns positive from the third
statement on and would have slipped past the existing per-entry bounds check. It now emits no table
rather than a wrong one.

Breaking change: HierarchyRuleProvider SPI

// before
Map<String, BiFunction<Service, Service, Boolean>> buildRules(Map<String, String> ruleExpressions);
// after
Map<String, BiFunction<Service, Service, Boolean>> buildRules(
    Map<String, String> ruleExpressions, Map<String, Integer> ruleLines);

A default bridge was considered and rejected: it would leave any implementation that didn't
override the new method silently producing _Lunknown_ classes — exactly the bug being fixed.
Third-party implementations (if any exist) should adopt the new signature and supply the line map.

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

Tests

Every new test drives the production loader or generator and reads back the real artifact
(class name, SourceFile, LineNumberTable) — none injects a synthetic coordinate, which is how
the missing coordinates went unnoticed in the first place. Each assertion was mutation-checked:
reverting the production change fails the test.

  • OALSourceAttributionTest (new, 5) — generates through OALClassGeneratorV2 and asserts each
    metrics class names its own statement line, the dispatcher names the file without one (one
    dispatcher spans every metric of a scope, so no single line is true for the class), the builder
    carries provenance but no line numbers, and every templated method carries exactly one
    LineNumberTable entry landing on its own declaration. OAL is the only DSL whose bodies come
    from FreeMarker rather than its own codegen, so lineOfMethod must find a signature it did not
    write — and a miss there is silent by design.
  • HierarchyProviderCoordinateTest (+2) — asserts the compiled class is named
    hierarchy_definition_L2_lower_short_name, and forces a throw from inside a generated rule to
    assert the JVM-reported frame reads (hierarchy-definition.yml:7)….java.
  • DslGeneratedFileWriterLineOfMethodTest (new, 5) — pins the identifier boundary with the real
    colliding pair, and that the answer no longer depends on declaration order.
  • MalFilterLineAttributionTest (new, 3) — resolves the filter class's LineNumberTable against
    the generated source that was written, the artifact the deleted constant never had a test for.
  • RuleKeyTest (+3) — the boot and hot-update spellings are one key, one registry entry, and only
    a YAML extension is stripped.
  • RuleSourcePathTest, RulesLoaderTest, LALConfigsSourceCoordinateTest,
    LALConfigsLoaderTest, ZabbixConfigSourceCoordinateTest, ZabbixConfigsLoaderTest,
    ZabbixGeneratedClassCoordinateTest, DslYamlLineIndexKeyLinesTest,
    MalLineAttributionTest, MalCompanionSourceTest, MalClosureLineAttributionTest,
    LALSourceFileResolvesTest, LALSourceAttributionScriptTest.

Operator-visible: generated class names change

A generated class's stem is the rule file's path, with the catalog kept only where the class's
package does not already imply it. MAL's catalogs share one rt package and two of them ship a
vm.yaml, so MAL gains it: vm_L25_cpu_total_percentage becomes
otel_rules_vm_L25_cpu_total_percentage. LAL has one catalog and its own package, so its names are
unchanged at default_L3_defaultlal/ appears in SourceFile, where it is a path an operator
opens, and not in the name, where it would only repeat the package. Nothing addresses these classes
by name (generated, loaded reflectively, never named in configuration), but they appear in stack
traces, SW_DYNAMIC_CLASS_ENGINE_DEBUG dumps and dsl-debugging output, so saved greps on MAL names
need updating.

Scope

DSL compilation internals only. No agent analysis, ALS, business or telemetry path is touched; the
full diff was audited against that boundary.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
  • Update the CHANGES log.

@wu-sheng
wu-sheng requested a balanced review from Copilot August 10, 2026 01:51
@wu-sheng wu-sheng added this to the 11.0.0 milestone Aug 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@wu-sheng wu-sheng added the enhancement Enhancement on performance or codes label Aug 10, 2026
@wu-sheng
wu-sheng force-pushed the feat/dsl-sourcefile-provenance branch from 65b5bc0 to e1ed855 Compare August 10, 2026 02:59
@wu-sheng wu-sheng changed the title Name the rule file in SourceFile, not a sidecar that is not there Unify DSL source attribution across OAL, MAL, LAL and Hierarchy Aug 10, 2026
@wu-sheng
wu-sheng force-pushed the feat/dsl-sourcefile-provenance branch 6 times, most recently from c2d42f1 to 86ee990 Compare August 10, 2026 12:25
Extract the shared rule-file/line -> generated-class workflow into
org.apache.skywalking.oap.server.core.dsl (DslSourceRef,
DslGeneratedFileWriter, DslClassNaming, DslJavaSourceText, DslYamlLineIndex)
and route all four DSL compilers through it. The identifier sanitiser, the
Java-literal escaper, the SourceFile writers, the LocalVariableTable writers
and the class/source file writers each had three or four copies; they now
have one.

SourceFile leads with the rule coordinate, then the generated file name, so a
stack frame resolves in a released image where no .java exists on disk. The
MAL, LAL and Zabbix loaders now stamp the coordinate the generators read;
Zabbix previously supplied none.

Fix a hot-updated LAL rule stranding its own dsl-debugging binding. The
RuleKey naming a rule file was spelled "default.yaml" by the boot loader and
"default" by the runtime-rule engine, so the two never met in the holder
registry and an operator addressing the older spelling enabled probes on a
rule that no longer evaluates anything. RuleKey now canonicalises that
component. Rule execution was never affected: the maps deciding which rules
run are keyed by layer and rule name, not by file name.

lineOfMethod now requires an identifier boundary before the method name.
"serialize" is a suffix of "deserialize" and OAL declares both on every
metrics class, so the shorter name resolved to the longer method's line
whenever the template order changed.

MAL's hand-counted line envelopes (10 + closures + injection, and 9 for the
filter) are replaced by the same signature search, with a guard so an
unresolved signature emits no table rather than a wrong one.

HierarchyRuleProvider.buildRules takes the rule line map, a deliberate SPI
break: a default bridge would leave non-overriding implementations silently
producing _Lunknown_ classes.

CLAUDE.md now states that a released SWIP is frozen even when a later
refactor makes its text stale.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMkDwMcr7139gZ95XCMQ6i
@wu-sheng
wu-sheng force-pushed the feat/dsl-sourcefile-provenance branch from 86ee990 to e6467d4 Compare August 10, 2026 12:50
@wu-sheng
wu-sheng merged commit a2f1f16 into master Aug 10, 2026
449 of 457 checks passed
@wu-sheng
wu-sheng deleted the feat/dsl-sourcefile-provenance branch August 10, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhancement on performance or codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants