Conversation
PR Reviewer Guide 🔍(Review updated until commit 4c0a18c)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 4c0a18c Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit ffe79d1
Suggestions up to commit 25d44a2
Suggestions up to commit 680d0e4
Suggestions up to commit 9d2e6e0
Suggestions up to commit 02f905d
|
Codecov Report❌ Patch coverage is
❌ Your project check has failed because the head coverage (63.18%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #5794 +/- ##
============================================
- Coverage 63.23% 63.18% -0.06%
Complexity 8823 8823
============================================
Files 938 938
Lines 40236 40272 +36
Branches 4537 4550 +13
============================================
Hits 25445 25445
- Misses 13966 14002 +36
Partials 825 825
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Persistent review updated to latest commit 02f905d |
|
Persistent review updated to latest commit 9d2e6e0 |
The analytics engine stores an OpenSearch `object` as a native Parquet struct and declares it to the planner as a ROW, so `fields city.geo.lat` has to read into that column. Today it cannot: planning fails with "Cannot infer type of field 'geo.lat' within ROW type", which makes every object sub-field unreadable on that route. The leftover path segments were joined into one ITEM key. That is right for the v2 path, which declares an object as MAP because it stores objects flattened, so a dotted key is genuinely one key there. A ROW has no field of that name. Descend a ROW one segment at a time with makeFieldAccess. Anything that is not a ROW keeps the joined ITEM key, so the v2 path is unchanged. Signed-off-by: Marc Handalian <handalm@amazon.com>
Descending a dotted path into a ROW one segment at a time left the miss case
unguarded. When a segment names no field of the ROW, the descent loop breaks
with part of the path unconsumed, and the leftover was then joined into an
ITEM key regardless of what it was applied to.
ITEM(<ROW>, 'x') is precisely the shape that descending exists to avoid.
SqlItemOperator looks the key up as a field of the ROW, does not find it, and
throws AssertionError "Cannot infer type of field 'x' within ROW type".
AssertionError is an Error, not an Exception, so it escapes the
catch (Exception) around field resolution and reaches the client as a 500.
The effect was that a misspelled field under an object, such as
`city.nonexistent`, failed with an internal error instead of the ordinary
"Field [...] not found" that every other unresolvable name produces.
Fall back to ITEM only when the node being indexed is not a struct. When
descent stops inside a ROW, report the path as unresolved: the prefix walk
then continues to shorter prefixes, since a shorter one may match a
different column, and an exhausted walk ends at the normal not-found with
its available-field suggestions.
Tests, in the existing CalcitePPLStructFieldTest fixture:
- an undeclared field of a ROW is a not-found, not an AssertionError
(fails without this change)
- a ROW path used in a predicate, not only a projection
- a backtick-quoted dotted path, which is one part with nothing to walk,
reaching the same field access as the unquoted form
Signed-off-by: Marc Handalian <handalm@amazon.com>
9d2e6e0 to
680d0e4
Compare
|
Persistent review updated to latest commit 680d0e4 |
The split-and-retry added for ROW columns also applied to MAP columns, which resolved names that are meant to be gone. When a container column is rebuilt its dotted subtree is shed, so a literal dotted column created in between becomes unreachable on purpose -- a second `spath input=body output=data` drops a `data.custom` evaluated after the first. Looking `data.custom` up literally then fails, which is the intended answer. Splitting it into [data, custom] and retrying found the rebuilt MAP and produced ITEM(data, 'custom'), so the query quietly returned null instead of reporting the field as missing, and CalcitePPLSpathCollisionIT.testRepeatedSpathShadowsInterveningLiteralDottedColumn stopped throwing. Splitting exists to reach into a ROW, where a dotted name really is a path and no ITEM key can express it. Over anything else the literal lookup was already the right question. So the retry now accepts a resolution only when the leftover path is consumed by descending a ROW, and reports the name unresolved when it would become an ITEM key. The first, unsplit attempt is unchanged, so an unquoted `data.custom` over a MAP still becomes one ITEM key as the v2 path needs. Tests: a quoted dotted name over a MAP stays unresolved (fails without this change, in the same "nothing was thrown" shape as the IT), an unquoted one over the same MAP still yields ITEM, and the quoted form over a ROW still descends. Signed-off-by: Marc Handalian <handalm@amazon.com>
|
Persistent review updated to latest commit 25d44a2 |
|
Persistent review updated to latest commit ffe79d1 |
ahkcs
left a comment
There was a problem hiding this comment.
Two notes on failed-descent handling and on nullability, inline.
| Optional<RexNode> fieldNode = tryToResolveField(alias, field, context, inputCount); | ||
| if (fieldNode.isPresent()) { | ||
| return Optional.of(resolveFieldAccess(context, parts, 1, length, fieldNode.get())); | ||
| return Optional.of(resolveFieldAccess(context, parts, 1, length, fieldNode.get(), false)); |
There was a problem hiding this comment.
Failed descent has three exits and only one is handled.
(a) resolveFieldAccess is now nullable (375), but this call is still Optional.of(...) → NPE on source=docs | where docs.city.nope = 1: resolveFieldWithAlias runs before resolveFieldWithoutAlias (79–81) and first in join conditions (61), alias docs matches field city at length=1, descent breaks on nope, and the guard returns null. Optional.ofNullable here, as resolveFromParts already does.
(b) The guard at 375 only fires when descent stops while still on a ROW. If it ends on a scalar child with segments left — fields city.name.bogus, city.name VARCHAR — isStruct() is false, the guard is skipped, and 388 builds ITEM(VARCHAR, 'bogus'). SqlItemOperator.inferReturnType's switch only covers ARRAY/MAP/ROW/ANY/DYNAMIC_STAR/VARIANT, so VARCHAR hits default: throw new AssertionError() — no message, and as an Error it escapes the catch (Exception) this method's own javadoc describes. Rejecting any leftover path once a ROW has been entered would cover both exits.
| if (child == null) { | ||
| break; | ||
| } | ||
| current = context.rexBuilder.makeFieldAccess(current, child.getIndex()); |
There was a problem hiding this comment.
makeFieldAccess loses the container's nullability. It delegates to makeFieldAccessInternal, which special-cases only RexRangeRef, so the result is a RexFieldAccess typed exactly field.getType(). The ITEM path it replaces did propagate: SqlItemOperator.inferReturnType's ROW branch ends with if (operandType.isNullable()) enforceTypeWithNullability(type, true).
The fixture in this PR has the shape that breaks: geo is a reference field so JavaType makes it nullable, lat is a primitive double so it is NOT NULL — $1.geo.lat is therefore DOUBLE NOT NULL under a nullable parent. RexSimplify.simplifyIsNull folds is null on a non-nullable safe expression to literal FALSE (and SafeRexVisitor.visitFieldAccess returns true unconditionally), so where city.geo.lat is null silently drops every document missing the object. Suggest re-applying nullability when any ancestor in the path is nullable.
Worth an executing test for this — nothing here calls verifyResult, so a row with a null struct is never evaluated.
`fields sourceNode.keyAttributes` returned the right values under the wrong names: the response schema said `$f0`, `$f1`, ... instead of the paths asked for, so anything reading a column by name found nothing. visitProject re-applies the user-visible name after resolving a dotted path, because the resolved node carries none of its own -- but it only did so for SqlKind.ITEM, which is what a flattened object resolves to. An object stored as a struct resolves to a chain of field accesses instead, so nothing re-applied the name and Calcite derived one. Alias anything that is not already a plain column reference. Resolution was always correct, which is why the rows were right and only the schema was wrong. The ITEM-only condition predates struct support, so the flattened-object path is unaffected. Signed-off-by: Marc Handalian <handalm@amazon.com>
Two ways a path into an object failed hard rather than being reported: fields city.name.bogus -> 500 source=docs as d | fields d.city.nope -> NPE The first ran past the end of the object. `city.name` is a VARCHAR, so descent stopped having already left the ROW, and the leftover was turned into ITEM(VARCHAR, 'bogus'). SqlItemOperator accepts only ARRAY / MAP / ROW / ANY / VARIANT and otherwise throws a bare AssertionError, which is an Error and so escapes the catch (Exception) around resolution. A leftover is now rejected unless the type can take an ITEM key at all. That is narrower than rejecting any leftover once a ROW was entered: a flat_object inside an object is a MAP child of the struct, so `city.meta.region` legitimately descends the ROW and then keys into the map. Tested both ways. The second is the alias branch of resolveFieldWithAlias, which wrapped the result in Optional.of even though resolveFieldAccess became nullable for a failed descent. It now keeps walking shorter prefixes, as resolveFromParts does. The alias branch runs first, so it is what a qualified miss hits. Also skip empty segments when splitting a quoted dotted name -- `a..b` and a trailing dot cannot name a field. Signed-off-by: Marc Handalian <handalm@amazon.com>
|
Persistent review updated to latest commit 4c0a18c |
Description
The analytics engine stores an OpenSearch
objectas a native Parquet struct and declares it to the planner as a ROW, sofields city.geo.lathas to read into that column. Today it cannot: planning fails with "Cannot infer type of field 'geo.lat' within ROW type", which makes every object sub-field unreadable on that route.The leftover path segments were joined into one ITEM key. That is right for the v2 path, which declares an object as MAP because it stores objects flattened, so a dotted key is genuinely one key there. A ROW has no field of that name.
Descend a ROW one segment at a time with makeFieldAccess. Anything that is not a ROW keeps the joined ITEM key, so the v2 path is unchanged.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.