Skip to content

[common] Keep the selected variant schema when its evidence is gone #9532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
PDGGK wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[common] Keep the selected variant schema when its evidence is gone #9532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f222ba6
[common] Keep the selected variant schema when evidence is gone
Sep 1, 2026
6c52495
Debit the shared width budget for a retained schema
Sep 2, 2026
20d2ec7
Keep the retained field at the last budget unit
Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion paimon-common/src/main/java/org/apache/paimon/data/variant/InferVariantShreddingSchema.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -574,7 +574,11 @@ private DataType finalizeAdaptiveSchema(
if (current != null && !(current instanceof VariantType)) {
combined = current;
} else if (previousSelected != null) {
combined = previousSelected;
// This node has no evidence in the current file. A previously selected schema
// is not evidence - its fields carry no counts - so it cannot be run through
// admission and retention again; carry it forward as it is, debiting the shared
// width budget for what it holds.
return retainSelectedSchema(previousSelected, maxFields);
} else {
return DataTypes.VARIANT();
}
Expand Down Expand Up @@ -654,6 +658,43 @@ private DataType finalizeAdaptiveSchema(
return selectScalarType(combined, current, previousSelected);
}

/**
* Carries a previously selected schema forward for a node the current file has no evidence for.
* The selection is already final, so nothing is re-thresholded, but its nodes still consume the
* shared width budget. The entry unit for this node has already been spent by the caller, so
* this mirrors what finalizeAdaptiveSchema does from that point on: a child is entered only
* while budget remains, entering it spends one unit, and a child that exhausts the budget
* becomes VARIANT while its field or array container is still kept.
*/
private DataType retainSelectedSchema(DataType selected, MaxFields maxFields) {
if (selected instanceof RowType) {
List<DataField> fields = new ArrayList<>();
for (DataField field : ((RowType) selected).getFields()) {
if (maxFields.remaining <= 0) {

@JingsongLi JingsongLi Sep 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the retained field/container at the last budget unit

This does not match the evidence-driven walk described above. When one unit remains, that walk enters the child, decrements the unit to zero, returns VARIANT, and still adds the parent DataField; this helper decrements first and breaks before adding it. If it was the only field, line 678 then collapses the whole retained RowType to the root VARIANT. The Array branch similarly collapses the array instead of retaining ARRAY<VARIANT>.

I reproduced this with maxSchemaWidth = 7 and two Variant columns. Round 1 is (1, 5), round 2 is (1, {q:1}), and round 3 is ({x:1,y:1}, null). Column a consumes five units, leaving exactly the root plus one field unit for retained column b. The updated code returns an untyped root for b and loses q; a focused assertion that b still contains q fails.

Please mirror finalizeAdaptiveSchema at exhaustion: retain the row field/array container and downgrade the exhausted child to VARIANT, rather than breaking/returning before preserving the parent shape.

break;
}
maxFields.remaining--;
DataType retained =
maxFields.remaining <= 0
? DataTypes.VARIANT()
: retainSelectedSchema(field.type(), maxFields);
fields.add(new DataField(fields.size(), field.name(), retained));
}
return fields.isEmpty() ? DataTypes.VARIANT() : new RowType(fields);
}
if (selected instanceof ArrayType) {
maxFields.remaining--;
DataType element =
maxFields.remaining <= 0
? DataTypes.VARIANT()
: retainSelectedSchema(
((ArrayType) selected).getElementType(), maxFields);
return new ArrayType(element);
}
maxFields.remaining--;

@JingsongLi JingsongLi Sep 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Do not charge retained VARIANT leaves twice

The caller already spends the node entry unit before retainSelectedSchema is reached. For a VariantType, there is no typed-value child to spend another unit on; the normal finalizeAdaptiveSchema path likewise returns VARIANT after only the entry debit. This branch therefore overcharges retained VARIANT leaves.

A minimal boundary case reproduces the regression: use two top-level variant columns with maxSchemaWidth = 3, infer/commit [null, 5], then infer [null, 6]. The first retained VARIANT should cost one unit and leave two for the second BIGINT, but this extra decrement exhausts the budget and the second column becomes untyped VARIANT. I confirmed that assertion fails against this head while the existing 27 tests pass. Please special-case VariantType here and add the shared-budget regression.

return selected;
}

private DataType selectScalarType(
DataType combined, DataType current, DataType previousSelected) {
if (current == null) {
Expand Down
116 changes: 116 additions & 0 deletions ...-common/src/test/java/org/apache/paimon/data/variant/InferVariantShreddingSchemaTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
Expand Down Expand Up @@ -315,6 +316,121 @@ void testAdaptiveInferenceWidensScalarSelectedFromPriorEvidence() {
new String[] {"historical"})));
}

/**
* A node can drift from scalar to object, which degrades its combined evidence to VARIANT while
* the selected schema for it is a ROW, and then be absent from the next file. With no evidence
* to fall back on, the selected schema is the only thing left - but it is not evidence: its
* fields carry no counts, so it cannot be run through admission and retention a second time.
*/
@Test
void testAdaptiveInferenceKeepsSelectedRowWhenEvidenceDegradedAndNodeIsAbsent() {
RowType schema = RowType.of(new DataType[] {DataTypes.VARIANT()}, new String[] {"v"});
VariantShreddingInferenceSession session =
new VariantShreddingInferenceSession(
new InferVariantShreddingSchema(schema, 300, 50, 0.1), 256, 0.1, 0.05);

session.inferSchema(
Collections.singletonList(
GenericRow.of(GenericVariant.fromJson("{\"k\":1,\"p\":5}"))));
session.commitPendingInference();
session.inferSchema(
Collections.singletonList(
GenericRow.of(GenericVariant.fromJson("{\"k\":1,\"p\":{\"x\":1}}"))));
session.commitPendingInference();

RowType afterAbsence =
session.inferSchema(
Collections.singletonList(
GenericRow.of(GenericVariant.fromJson("{\"k\":1}"))));

assertThat(afterAbsence.getField("v").type())
.isEqualTo(
variantShreddingSchema(
RowType.of(
new DataType[] {
DataTypes.BIGINT(),
RowType.of(
new DataType[] {DataTypes.BIGINT()},
new String[] {"x"})
},
new String[] {"k", "p"})));
}

/**
* maxSchemaWidth is one budget shared by every variant column. A schema carried forward for a
* node with no evidence still occupies it, so a later column must not get to spend what the
* carried-forward schema is holding.
*/
@Test
void testRetainedSchemaStillConsumesTheSharedWidthBudget() {
RowType schema =
RowType.of(
new DataType[] {DataTypes.VARIANT(), DataTypes.VARIANT()},
new String[] {"a", "b"});
VariantShreddingInferenceSession session =
new VariantShreddingInferenceSession(
new InferVariantShreddingSchema(schema, 8, 50, 0.1), 256, 0.1, 0.05);

session.inferSchema(
Collections.singletonList(
GenericRow.of(
GenericVariant.fromJson("{\"p\":5}"),
GenericVariant.fromJson("{\"q\":1}"))));
session.commitPendingInference();
session.inferSchema(
Collections.singletonList(
GenericRow.of(
GenericVariant.fromJson("{\"p\":{\"x\":1}}"),
GenericVariant.fromJson("{\"q\":1}"))));
session.commitPendingInference();

RowType afterAbsence =
session.inferSchema(
Collections.singletonList(
GenericRow.of(
GenericVariant.fromJson("{}"),
GenericVariant.fromJson("{\"q\":1,\"r\":1}"))));

// "a" keeps ROW<x BIGINT> under "p", and the budget it holds leaves "r" untyped in "b".
assertThat(afterAbsence.getField("b").type().toString())
.contains("`q` ROW<`value` BYTES, `typed_value` BIGINT>")
.doesNotContain("`r` ROW<`value` BYTES, `typed_value`");
}

/**
* At the last budget unit the evidence-driven walk still keeps the field and downgrades its
* child to VARIANT. Retaining a schema has to do the same rather than drop the field, or the
* whole retained node collapses.
*/
@Test
void testRetainedSchemaKeepsItsFieldsAtTheLastBudgetUnit() {
RowType schema =
RowType.of(
new DataType[] {DataTypes.VARIANT(), DataTypes.VARIANT()},
new String[] {"a", "b"});
VariantShreddingInferenceSession session =
new VariantShreddingInferenceSession(
new InferVariantShreddingSchema(schema, 7, 50, 0.1), 256, 0.1, 0.05);

session.inferSchema(
Collections.singletonList(
GenericRow.of(GenericVariant.fromJson("1"), GenericVariant.fromJson("5"))));
session.commitPendingInference();
session.inferSchema(
Collections.singletonList(
GenericRow.of(
GenericVariant.fromJson("1"),
GenericVariant.fromJson("{\"q\":1}"))));
session.commitPendingInference();

RowType afterAbsence =
session.inferSchema(
Collections.singletonList(
GenericRow.of(GenericVariant.fromJson("{\"x\":1,\"y\":1}"), null)));

assertThat(afterAbsence.getField("b").type().toString()).contains("`q`");
}

@Test
void testInferSchemaWithDeepNesting() {
// Schema: row<v: variant>
Expand Down
Loading