Skip to content

feat: OpenLineage consumer improvements - versioning, column lineage and tree - #6834

Open
ntkathole wants to merge 1 commit into
feast-dev:masterfrom
ntkathole:extended_openlineage
Open

ntkathole wants to merge 1 commit into
feast-dev:masterfrom
ntkathole:extended_openlineage

Conversation

@ntkathole

Copy link
Copy Markdown
Member

What this PR does / why we need it:

Summary

Extends the Feast OpenLineage consumer with spec-compliant facet extraction,
adds dataset versioning, and enriches SavedDataset producer events.

Consumer improvements

  • Dataset versioning — new openlineage_dataset_versions table; a version
    snapshot is created for each output dataset on every COMPLETE RunEvent
  • Column-level lineage — parses ColumnLineageDatasetFacet, stores
    field-to-field mappings in openlineage_column_lineage table
  • Ownership indexing — extracts OwnershipDatasetFacet into queryable
    owner_name/owner_type columns and a dedicated openlineage_dataset_ownership
    table (supports multiple owners per dataset)
  • Lifecycle tracking — extracts LifecycleStateChangeDatasetFacet
    (CREATE/ALTER/DROP) into lifecycle_state column on datasets
  • Execution hierarchy — parses ParentRunFacet to populate parent_run_id
    and root_run_id on runs; adds run-tree query API

New REST API endpoints

Endpoint Description
GET /datasets/{ns}/{name}/versions Paginated version history
GET /datasets/{ns}/{name}/columns Column-level lineage
GET /runs/{run_id}/tree Full execution tree

Ownership, lifecycle, version, and owner fields are returned inline on
dataset nodes in the existing full-graph response — no separate endpoints
needed.

SavedDataset producer

emit_saved_dataset_lineage() now emits an additional DatasetEvent with:

  • SymlinksDatasetFacet linking logical name → physical storage URI
  • OwnershipDatasetFacet from tags/owner field
  • LifecycleStateChangeDatasetFacet (CREATE)

UI

Added version, lifecycle, and owner badges to the OpenLineage graph
node detail panel (inline with existing producer/type badges).

@ntkathole ntkathole self-assigned this Sep 14, 2026
@ntkathole
ntkathole requested a review from a team as a code owner September 14, 2026 04:35
@codecov-commenter

codecov-commenter commented Sep 14, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 67.12329% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.32%. Comparing base (787845a) to head (3ea5ee4).

Files with missing lines Patch % Lines
sdk/python/feast/openlineage/emitter.py 5.66% 50 Missing ⚠️
sdk/python/feast/openlineage/processor.py 69.09% 10 Missing and 7 partials ⚠️
sdk/python/feast/openlineage/consumer.py 28.57% 15 Missing ⚠️
sdk/python/feast/openlineage/store.py 90.78% 6 Missing and 8 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6834      +/-   ##
==========================================
+ Coverage   47.18%   47.32%   +0.14%     
==========================================
  Files         419      419              
  Lines       51964    52253     +289     
  Branches     7548     7605      +57     
==========================================
+ Hits        24521    24731     +210     
- Misses      25689    25754      +65     
- Partials     1754     1768      +14     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.64% <67.12%> (+0.14%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/openlineage/models.py 100.00% <100.00%> (ø)
sdk/python/feast/openlineage/store.py 87.35% <90.78%> (+3.45%) ⬆️
sdk/python/feast/openlineage/consumer.py 64.14% <28.57%> (-4.23%) ⬇️
sdk/python/feast/openlineage/processor.py 84.07% <69.09%> (-1.65%) ⬇️
sdk/python/feast/openlineage/emitter.py 7.41% <5.66%> (-0.20%) ⬇️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 787845a...3ea5ee4. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ntkathole
ntkathole force-pushed the extended_openlineage branch 3 times, most recently from 9ade90e to c25aac0 Compare September 15, 2026 07:41
…lineage, ownership

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants