Skip to content

[SDV 2.0] Add high_cardinality flag to ordinal and categorical column metadata - #2991

Open
R-Palazzo wants to merge 9 commits into
v2.0.0from
issue-2990-high-cardinality
Open

R-Palazzo wants to merge 9 commits into
v2.0.0from
issue-2990-high-cardinality

Conversation

@R-Palazzo

Copy link
Copy Markdown
Contributor

Resolve #2990
86bc1nxb8

@R-Palazzo R-Palazzo self-assigned this Sep 17, 2026
@R-Palazzo
R-Palazzo requested a review from a team as a code owner September 17, 2026 16:25
@R-Palazzo
R-Palazzo removed the request for review from a team September 17, 2026 16:25
@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.45455% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.92%. Comparing base (9b7c327) to head (012a93e).

Files with missing lines Patch % Lines
sdv/data_processing/data_processor.py 80.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           v2.0.0    #2991   +/-   ##
=======================================
  Coverage   97.92%   97.92%           
=======================================
  Files          66       66           
  Lines        7754     7758    +4     
=======================================
+ Hits         7593     7597    +4     
  Misses        161      161           
Flag Coverage Δ
integration 84.27% <81.81%> (-0.02%) ⬇️
unit 96.22% <95.45%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sdv-team

Copy link
Copy Markdown
Contributor

This Pull Request is not linked to an issue. To ensure our community is able to accurately track resolved issues, please link any issue that will be closed by this PR!

Comment on lines 410 to +411
'range_values',
'high_cardinality',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we add high_cardinality here, we shouldn't need to pop it from the parameters in the the _get_ordinal_transformer and _get_categorical_transformer methods. Also, range_values should not be in this list, since we do want the range_values when instantiating ordinal transformers.

table_str = f" for table '{table_name}'" if table_name else ''
sys.stdout.write(f'\nDetecting primary key{table_str}:\n')
_print_primary_key_detection(chosen_pk, sdtype_updated, pii_removed)
_print_primary_key_detection(chosen_pk)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that all the detection results are printed together at the end, I don't think we need to mention when the sdtype was updated anymore.

Before, it made sense because the output was printed during detection. Now it can be confusing to see sdtype='id' (updated to 'id') when we're already showing the final state.

Let me know if it makes sense this way.

def _print_detection(

@sdv-dev sdv-dev deleted a comment from sdv-team Sep 21, 2026
@sdv-dev sdv-dev deleted a comment from sdv-team Sep 21, 2026
@sdv-dev sdv-dev deleted a comment from sdv-team Sep 21, 2026
@sdv-dev sdv-dev deleted a comment from sdv-team Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants