Skip to content

fix(workflow-operator): Text Input operator using offset with an empty limit emits no rows - #7347

Merged
aglinxinyuan merged 2 commits into
apache:mainfrom
eugenegujing:fix/text-input-offset-overflow
Aug 10, 2026
Merged

fix(workflow-operator): Text Input operator using offset with an empty limit emits no rows#7347
aglinxinyuan merged 2 commits into
apache:mainfrom
eugenegujing:fix/text-input-offset-overflow

Conversation

@eugenegujing

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

TextInputSourceOpExec computed its line window as slice(offset, offset + limit.getOrElse(Int.MaxValue)). With an Offset set and the Limit left empty, the addition overflows Int to a negative bound, which Scala 2.13's Iterator.slice clamps to 0 and then returns an empty iterator — so the operator silently emitted zero rows while the workflow reported success. Any Offset ≥ 1 with an empty Limit is affected, and an explicit large Limit (e.g. Int.MaxValue) overflows the same way. This contradicts the Limit property's own description, "Leave empty to read all lines."

This PR replaces the slice with drop(offset) + take(limit), the same idiom the CSV, Arrow, and JSONL scan sources already use. There is no addition, so nothing can overflow; every configuration that previously worked is unchanged (verified case-by-case, including negative offsets and isSingle attribute types, which keep ignoring offset/limit as documented).

Before the fix (current main) — Offset = 1, Limit left empty, five-line input a b c d e: the result is an empty set even though the workflow completes successfully. Expected: the four rows b, c, d, e.

Screenshot 2026-08-05 at 2 21 56 PM

Control on the same build — Offset = 0, Limit left empty returns all five rows:

Screenshot 2026-08-05 at 2 22 05 PM

Any related issues, documentation, discussions?

Closes #7346.

Same class of defect as #7245 (JSONL File Scan dropping rows when Offset is set), which was fixed by #7247.

How was this PR tested?

TDD: the regression tests were written first and confirmed to fail on the unfixed code — the offset-without-limit case and the offset-with-Int.MaxValue-limit case both produced empty output — then the fix was applied and all tests pass.

Seven new cases were added to TextInputSourceOpDescSpec (the spec that already exercises produceTuple()): offset without limit, offset with an Int.MaxValue limit, offset+limit window, limit only, offset at/past the end of the input, negative offset treated as zero, and SINGLE_STRING ignoring offset/limit (documented behavior, pinned).

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.text.TextInputSourceOpDescSpec"
# 17 tests, all passed (10 pre-existing + 7 new)

sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.scan.*"
# 17 suites, 123 tests, all passed

sbt "WorkflowOperator/scalafixAll --check"
# passed, no lint issues

sbt scalafmtCheckAll
# passed, no mis-formatted files

Also verified manually in the UI with the same two-operator workflow shown in the screenshots above:
Screenshot 2026-08-05 at 5 15 55 PM

Was this PR authored or co-authored using generative AI tooling?

Co-authored by: Claude Code (Claude Fable 5)

…y limit emits no rows

TextInputSourceOpExec computed its line window as slice(offset, offset + limit.getOrElse(Int.MaxValue)). With an offset set and the limit left empty, the addition overflows Int to a negative bound, which Iterator.slice clamps to 0, so the operator silently emitted zero rows while the workflow reported success. An explicit large limit overflows the same way. This contradicts the Limit property's own description, "Leave empty to read all lines."

Replace the slice with drop(offset) + take(limit), the same idiom the CSV, Arrow, and JSONL scan sources already use. Every configuration that previously worked is unchanged.

Add regression tests covering offset-without-limit, offset with an Int.MaxValue limit, the offset+limit window, limit-only, offset at and past EOF, a negative offset, and SINGLE_STRING ignoring offset/limit.

Closes apache#7346

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Aug 6, 2026
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Yicong-Huang, @aglinxinyuan
    You can notify them by mentioning @Yicong-Huang, @aglinxinyuan in a comment.

@github-actions
github-actions Bot requested a review from xuang7 August 6, 2026 00:18
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch Analysis
release/v1.2 Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7.

Auto-label run.

@codecov-commenter

codecov-commenter commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.25%. Comparing base (88ca47f) to head (91af32d).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #7347      +/-   ##
============================================
- Coverage     86.24%   83.25%   -3.00%     
- Complexity     4211     4741     +530     
============================================
  Files          1169     1228      +59     
  Lines         46736    51386    +4650     
  Branches       5203     5847     +644     
============================================
+ Hits          40308    42780    +2472     
- Misses         4691     6715    +2024     
- Partials       1737     1891     +154     
Flag Coverage Δ *Carryforward flag
access-control-service 71.10% <ø> (+1.10%) ⬆️
agent-service 86.87% <ø> (ø) Carriedforward from aa791a4
amber 76.78% <100.00%> (-5.10%) ⬇️
computing-unit-managing-service 35.12% <ø> (-15.61%) ⬇️
config-service 64.02% <ø> (-1.96%) ⬇️
file-service 64.13% <ø> (-4.92%) ⬇️
frontend 88.91% <ø> (ø) Carriedforward from aa791a4
notebook-migration-service 78.89% <ø> (ø)
pyamber 97.55% <ø> (ø) Carriedforward from aa791a4
workflow-compiling-service 51.78% <ø> (+25.46%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 6 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main 88ca47f benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 367 0.224 24,564/44,465/44,465 us 🔴 +41.0% / 🔴 +171.3%
bs=100 sw=10 sl=64 793 0.484 124,540/141,683/141,683 us ⚪ within ±5% / 🔴 +25.9%
🔴 bs=1000 sw=10 sl=64 907 0.553 1,088,989/1,298,526/1,298,526 us 🔴 +6.0% / 🔴 +20.7%
Baseline details

Latest main 88ca47f from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 367 tuples/sec 423 tuples/sec 743.73 tuples/sec -13.2% -50.7%
bs=10 sw=10 sl=64 MB/s 0.224 MB/s 0.258 MB/s 0.454 MB/s -13.2% -50.7%
bs=10 sw=10 sl=64 p50 24,564 us 24,064 us 13,130 us +2.1% +87.1%
bs=10 sw=10 sl=64 p95 44,465 us 31,537 us 16,391 us +41.0% +171.3%
bs=10 sw=10 sl=64 p99 44,465 us 31,537 us 19,408 us +41.0% +129.1%
bs=100 sw=10 sl=64 throughput 793 tuples/sec 826 tuples/sec 944.04 tuples/sec -4.0% -16.0%
bs=100 sw=10 sl=64 MB/s 0.484 MB/s 0.504 MB/s 0.576 MB/s -4.0% -16.0%
bs=100 sw=10 sl=64 p50 124,540 us 120,038 us 105,853 us +3.8% +17.7%
bs=100 sw=10 sl=64 p95 141,683 us 148,526 us 112,493 us -4.6% +25.9%
bs=100 sw=10 sl=64 p99 141,683 us 148,526 us 122,200 us -4.6% +15.9%
bs=1000 sw=10 sl=64 throughput 907 tuples/sec 927 tuples/sec 972.22 tuples/sec -2.2% -6.7%
bs=1000 sw=10 sl=64 MB/s 0.553 MB/s 0.566 MB/s 0.593 MB/s -2.3% -6.8%
bs=1000 sw=10 sl=64 p50 1,088,989 us 1,072,675 us 1,033,856 us +1.5% +5.3%
bs=1000 sw=10 sl=64 p95 1,298,526 us 1,225,461 us 1,076,083 us +6.0% +20.7%
bs=1000 sw=10 sl=64 p99 1,298,526 us 1,225,461 us 1,107,701 us +6.0% +17.2%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,544.97,200,128000,367,0.224,24564.02,44464.60,44464.60
1,100,10,64,20,2523.41,2000,1280000,793,0.484,124540.29,141682.77,141682.77
2,1000,10,64,20,22058.75,20000,12800000,907,0.553,1088989.31,1298525.63,1298525.63

@eugenegujing

Copy link
Copy Markdown
Contributor Author

/request-review @aglinxinyuan

@github-actions
github-actions Bot requested a review from aglinxinyuan August 6, 2026 23:51
@chenlica

chenlica commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@carloea2 Can you review it first?

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@aglinxinyuan
aglinxinyuan enabled auto-merge August 10, 2026 02:27
@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Aug 10, 2026
Merged via the queue into apache:main with commit 91235fb Aug 10, 2026
45 of 51 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened: draft #7495 (#7495) to release/v1.2, assigned to @eugenegujing — needs manual work because the backported tree failed its pre-merge build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common fix release/v1.2 back porting to release/v1.2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Text Input emits zero rows when Offset is set and Limit is left empty

6 participants