Repository navigation
fix(data): preserve feature positions across Parquet shards and workers - #199
Draft
DevChiniwala wants to merge 2 commits into
Draft
DevChiniwala wants to merge 2 commits into
DevChiniwala wants to merge 2 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both streaming Parquet readers derive feature positions separately from each shard. If shards contain the same named features in different physical column orders, the model silently receives swapped inputs. A PyTorch worker assigned only a later shard can also use different positions from another worker.
Anchor feature order to the first file's schema for the dataset, validate the target and feature names on every shard, and select columns in that canonical order in both the PyTorch iterable dataset and NumPy batch generator. Preserve batching, target dtypes, sharding and return types. An incompatible shard now raises a clear ValueError instead of silently changing the model's inputs. Document the ordering and schema requirements.
Validation on Python 3.12.7, pytest 8.4.2, PyArrow 22.0.0 and CPU PyTorch 2.7.1:
a1e05f6: 11 regression cases fail and the first-worker control passes. Failures demonstrate swapped feature values, incorrect later-worker inputs, inconsistent/missing schema handling and loss of canonical order when the first shard is empty.git diff --checkpass on the final changes.Draft because the repository-prescribed staged integration run requires Anthropic credentials and Spark/Java, which were unavailable here. Live provider/training integration, actual multi-process workers and a distributed cluster have not been claimed as tested.
Repository workflow metadata: the patch version is bumped from 1.4.4 to 1.4.5 as required by the source-change version gate.
poetry check --lockand the branch/title/version checks pass locally. Dependencies and the lockfile are unchanged.