Unify DSL source attribution across OAL, MAL, LAL and Hierarchy - #13972
Merged
Conversation
wu-sheng
force-pushed
the
feat/dsl-sourcefile-provenance
branch
from
August 10, 2026 02:59
65b5bc0 to
e1ed855
Compare
wu-sheng
force-pushed
the
feat/dsl-sourcefile-provenance
branch
6 times, most recently
from
August 10, 2026 12:25
c2d42f1 to
86ee990
Compare
Extract the shared rule-file/line -> generated-class workflow into org.apache.skywalking.oap.server.core.dsl (DslSourceRef, DslGeneratedFileWriter, DslClassNaming, DslJavaSourceText, DslYamlLineIndex) and route all four DSL compilers through it. The identifier sanitiser, the Java-literal escaper, the SourceFile writers, the LocalVariableTable writers and the class/source file writers each had three or four copies; they now have one. SourceFile leads with the rule coordinate, then the generated file name, so a stack frame resolves in a released image where no .java exists on disk. The MAL, LAL and Zabbix loaders now stamp the coordinate the generators read; Zabbix previously supplied none. Fix a hot-updated LAL rule stranding its own dsl-debugging binding. The RuleKey naming a rule file was spelled "default.yaml" by the boot loader and "default" by the runtime-rule engine, so the two never met in the holder registry and an operator addressing the older spelling enabled probes on a rule that no longer evaluates anything. RuleKey now canonicalises that component. Rule execution was never affected: the maps deciding which rules run are keyed by layer and rule name, not by file name. lineOfMethod now requires an identifier boundary before the method name. "serialize" is a suffix of "deserialize" and OAL declares both on every metrics class, so the shorter name resolved to the longer method's line whenever the template order changed. MAL's hand-counted line envelopes (10 + closures + injection, and 9 for the filter) are replaced by the same signature search, with a guard so an unresolved signature emits no table rather than a wrong one. HierarchyRuleProvider.buildRules takes the rule line map, a deliberate SPI break: a default bridge would leave non-overriding implementations silently producing _Lunknown_ classes. CLAUDE.md now states that a released SWIP is frozen even when a later refactor makes its text stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CMkDwMcr7139gZ95XCMQ6i
wu-sheng
force-pushed
the
feat/dsl-sourcefile-provenance
branch
from
August 10, 2026 12:50
86ee990 to
e6467d4
Compare
hanahmily
approved these changes
Aug 10, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unify DSL source attribution across OAL, MAL, LAL and Hierarchy
OAP compiles four DSLs to bytecode at runtime — OAL, MAL, LAL and Hierarchy. Each one
independently grew its own answer to the same question: when a generated class shows up in a
stack frame, what does it say it came from? The workflows are near-identical (rule file + line →
generated class name,
SourceFile,LineNumberTable, optional.javadump), but because eachcompiler implemented it separately, they drifted — differently, and silently.
This PR extracts that one workflow into
org.apache.skywalking.oap.server.core.dsland makes allfour use it.
What a frame said before, and says now
The
.javasidecar is written only underSW_DYNAMIC_CLASS_ENGINE_DEBUG; Javassist compilesfrom an in-memory string, so in a released image there is no
.javaon disk — in the container,in a binary package, or in an IDE. A
SourceFilenaming only the generated class thereforeaddressed nothing an operator could open.
vm_L25_cpu_total.java(otel-rules/vm.yaml:25)vm_L25_cpu_total.javadefault_L3_default.java(lal/default.yaml:3)default_L3_default.javahierarchy_definition_Lunknown_name.java(hierarchy-definition.yml:2)hierarchy_definition_L2_name.javaServiceRespTimeMetrics.java(core.oal:20)ServiceRespTimeMetrics.java(zabbix-rules/agent.yaml:20)agent_L20_cpu.javaThe rule leads, then the generated file name, so the coordinate survives whether or not the
sidecar exists.
The shared kernel
New, in
server-core:DslSourceRef— the value object for one coordinate. The only parser of"file:line",the only renderer of
(file:line)Class.java, and the only builder of the_L<line>_class-namesegment. Promoted from MAL's
MalSourceRef, which had been written but never instantiated.DslGeneratedFileWriter— all bytecode/disk side effects:writeClassFile,writeSourceFile,setSourceFile,attachSignatureLine,addLocalVariableTable,lineOfMethod. Split out ofDslSourceRefso the value object performs no I/O.DslClassNaming—stem/allocate. The stem is shared and deterministic; theallocation policy stays with each generator, because it genuinely differs (MAL and LAL give
each runtime-rule apply its own classloader and must not dedup process-wide; Hierarchy defines
into the shared loader and must).
DslJavaSourceText—toIdentifier/toLiteral: making arbitrary rule text safe to embedin generated Java source, in identifier position and in string-literal position.
DslYamlLineIndex— moved up frommeter-analyzer(wasMalYamlLineIndex), now servingMAL, LAL and Zabbix by rule ordinal, plus a new
keyLinesfor Hierarchy, whose rules are a YAMLmapping rather than a sequence and so cannot be matched positionally.
What actually collapsed (counted against
master):"file:line"parsersSourceFilestring built + attribute written_L<line>_class-name builders.javasource-file writers.classfile writersLocalVariableTablewritersThe remaining per-generator wrappers that only forwarded to the shared class were removed too, so
each generator now calls it directly — a wrapper that adds a hop without value would have made the
consolidation nominal rather than real. Three more copies went the same way:
escapeJavawas quadruplicated, char-identical in MAL, LAL and Hierarchy, with anull-tolerant fourth in OAL behind a "kept local so this generator stays self-contained" comment
that its own imports had already falsified. Deliberately not replaced by
commons-textStringEscapeUtils.escapeJava: that also escapes non-ASCII to au-prefixed hex escape, andJavassist has no unicode-escape pre-lex phase the way javac does, so a rule with a non-ASCII tag
key would compile with the escape taken literally.
sanitizeNameleft residue after the migration — LAL's copy was dead, MAL's had one livecaller, and that caller names a generated variable, not a class.
DslJavaSourceText(toIdentifier/toLiteral) rather than inDslClassNaming: they are one concern — embedding arbitrary rule text in generated Java source —with two positions, and neither is class naming.
DslClassNamingkeepsstemandallocate.*.v2.dsldebugpackages became*.v2.dsl.debug, matchingcore.dsl.debug. They stay in their modules: the dependency direction is one-way, so moving theminto
server-corewould create a cycle. Two of the references are string literals in codegen,which no compiler would have caught.
readAllinHierarchyDefinitionServicehand-rolled a buffer loop while every other ruleloader in this PR reads
new String(bytes, UTF_8). Now the same idiom.Loader fixes — the coordinates had to exist before they could be unified
Three loaders never stamped what the generators were about to read:
Rules.parseRule) — nested rules lost their ruleset directory, and a.ymlfile wasreported as
.yaml. Now stampsrulesetDir + "/" + relPathwith the real extension.Ruleneeded an explicit
getSourcePath()because Lombok@Datawas generating a getter thatshadowed the interface default.
LALConfigs) — resolved no line at all; now indexes therules:sequence and stamps acatalog-qualified
lal/<file>. The runtime-rule applier (LalFileApplier) stampslineNotoo,so a hot-updated rule is attributed the same as a bundled one.
ZabbixConfigs) —ZabbixConfigimplementsMetricRuleConfigbut took everydefault, so
getSourceName()was null and its classes carried neither file nor line. Now stampsboth, indexing under zabbix's own
metrics:key.Identity and attribution are separate fields.
sourcePathis the catalog-qualified path agenerated class names;
sourceNameis the rule file's identity, and on the boot route it becomesthe middle component of the dsl-debugging
RuleKey (LAL, sourceName, ruleName). An earlierrevision of this PR overloaded one field for both, catalog-qualifying
sourceNamefor attributionand thereby re-keying every static binding. They are now split, both derived through one
LALConfigs.stampSource, andMetricConvertguards on the field it actually passes rather thanits sibling.
Bug fix: a hot-updated LAL rule stranded its own debug binding
Separately from the fields above, the dsl-debugging
RuleKey (catalog, name, ruleName)names arule FILE in its middle component — and the two routes that publish it disagreed on the extension.
The boot loader published
(LAL, "default.yaml", rule); the runtime-rule engine(
LalRuleEngine.publishDebugBindings, which never readssourceName— it builds its own key fromthe rule's bare name) published
(LAL, "default", rule). Both land in the sameLALHolderRegistrymap, so the two keys simply never met: a hot update added a second bindinginstead of replacing the first,
unpublishDebugBindingscould never remove the first either, andan operator addressing the older spelling enabled probes on a
GateHolderbelonging to a compiledrule that no longer evaluates anything — indistinguishable from a rule with no traffic. No
exception, no log line.
RuleKeynow drops a trailing.yaml/.ymlfrom that component, in its constructor rather thanat each publish site: normalising per-site leaves the next site free to get it wrong, and doing it
in the key also lets the REST API keep accepting both spellings, so no existing operator script
breaks. Only a YAML extension is stripped — OAL files stay
core.oal, nested MAL bundles stayactivemq/activemq-broker, andvm.linux.yamlkeeps itsvm.linux.Rule execution was never affected. The maps that decide which rules run are keyed by layer and
rule name (
LogFilterListener.Factory.dsls) and contain no file name at all; this was confined tothe debugging registry. The mismatch predates this PR — MAL was never affected, because both its
routes already publish the bare name.
Per-method line numbers
LineNumberTableentries now point at a method's own signature line in the generated source(
attachSignatureLine+lineOfMethod), for OAL metrics/dispatcher methods, LAL'sexecuteandprivate methods, and MAL companions.
Per-statement tables are deliberately not emitted outside MAL's expression codegen. The scan
that produces them marks boundaries at stores to a result slot; LAL and OAL bodies are largely
void invocations that store nothing there, so the numbers came out as statement ordinals
(1, 2, 3…) — not lines in any file. Emitting a wrong line is worse than emitting none.
Line lookup: an identifier boundary, and two hand-counted envelopes deleted
lineOfMethodmatched a declaration withcontains(name + "("), which also accepts a longermethod whose name ENDS with the wanted one.
serializeis a suffix ofdeserializeand OALdeclares both on every metrics class; it resolved correctly only because the template list happens
to emit the shorter one first, and nothing said that ordering was load-bearing. Operator-authored
OAL reaches the same shape — metrics
cpmandcommando_cpmin one scope givedoCpmanddoCommandoCpmon the shared dispatcher. The match now requires a non-identifier character beforethe name.
MAL still computed two signature lines by counting its own source envelope:
10 + closures + injectionforrun(), and a bare9for a filter class. Both are gone, replaced by the samesearch. The filter constant had no artifact-resolving test at all — the branch that varied the
other one (
SW_DSL_DEBUGGING_INJECTION_ENABLED) never runs in CI — so an envelope edit shiftedevery filter frame silently. A guard was needed with the change: statement lines are counted
forward from the signature, so an unresolved signature (
-1) turns positive from the thirdstatement on and would have slipped past the existing per-entry bounds check. It now emits no table
rather than a wrong one.
Breaking change:
HierarchyRuleProviderSPIA
defaultbridge was considered and rejected: it would leave any implementation that didn'toverride the new method silently producing
_Lunknown_classes — exactly the bug being fixed.Third-party implementations (if any exist) should adopt the new signature and supply the line map.
Tests
Every new test drives the production loader or generator and reads back the real artifact
(class name,
SourceFile,LineNumberTable) — none injects a synthetic coordinate, which is howthe missing coordinates went unnoticed in the first place. Each assertion was mutation-checked:
reverting the production change fails the test.
OALSourceAttributionTest(new, 5) — generates throughOALClassGeneratorV2and asserts eachmetrics class names its own statement line, the dispatcher names the file without one (one
dispatcher spans every metric of a scope, so no single line is true for the class), the builder
carries provenance but no line numbers, and every templated method carries exactly one
LineNumberTableentry landing on its own declaration. OAL is the only DSL whose bodies comefrom FreeMarker rather than its own codegen, so
lineOfMethodmust find a signature it did notwrite — and a miss there is silent by design.
HierarchyProviderCoordinateTest(+2) — asserts the compiled class is namedhierarchy_definition_L2_lower_short_name, and forces a throw from inside a generated rule toassert the JVM-reported frame reads
(hierarchy-definition.yml:7)….java.DslGeneratedFileWriterLineOfMethodTest(new, 5) — pins the identifier boundary with the realcolliding pair, and that the answer no longer depends on declaration order.
MalFilterLineAttributionTest(new, 3) — resolves the filter class'sLineNumberTableagainstthe generated source that was written, the artifact the deleted constant never had a test for.
RuleKeyTest(+3) — the boot and hot-update spellings are one key, one registry entry, and onlya YAML extension is stripped.
RuleSourcePathTest,RulesLoaderTest,LALConfigsSourceCoordinateTest,LALConfigsLoaderTest,ZabbixConfigSourceCoordinateTest,ZabbixConfigsLoaderTest,ZabbixGeneratedClassCoordinateTest,DslYamlLineIndexKeyLinesTest,MalLineAttributionTest,MalCompanionSourceTest,MalClosureLineAttributionTest,LALSourceFileResolvesTest,LALSourceAttributionScriptTest.Operator-visible: generated class names change
A generated class's stem is the rule file's path, with the catalog kept only where the class's
package does not already imply it. MAL's catalogs share one
rtpackage and two of them ship avm.yaml, so MAL gains it:vm_L25_cpu_total_percentagebecomesotel_rules_vm_L25_cpu_total_percentage. LAL has one catalog and its own package, so its names areunchanged at
default_L3_default—lal/appears inSourceFile, where it is a path an operatoropens, and not in the name, where it would only repeat the package. Nothing addresses these classes
by name (generated, loaded reflectively, never named in configuration), but they appear in stack
traces,
SW_DYNAMIC_CLASS_ENGINE_DEBUGdumps and dsl-debugging output, so saved greps on MAL namesneed updating.
Scope
DSL compilation internals only. No agent analysis, ALS, business or telemetry path is touched; the
full diff was audited against that boundary.
CHANGESlog.