fix(workflow-operator): Text Input operator using offset with an empty limit emits no rows - #7347
Conversation
…y limit emits no rows TextInputSourceOpExec computed its line window as slice(offset, offset + limit.getOrElse(Int.MaxValue)). With an offset set and the limit left empty, the addition overflows Int to a negative bound, which Iterator.slice clamps to 0, so the operator silently emitted zero rows while the workflow reported success. An explicit large limit overflows the same way. This contradicts the Limit property's own description, "Leave empty to read all lines." Replace the slice with drop(offset) + take(limit), the same idiom the CSV, Arrow, and JSONL scan sources already use. Every configuration that previously worked is unchanged. Add regression tests covering offset-without-limit, offset with an Int.MaxValue limit, the offset+limit window, limit-only, offset at and past EOF, a negative offset, and SINGLE_STRING ignoring offset/limit. Closes apache#7346 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Backport auto-label reportThis
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7347 +/- ##
============================================
- Coverage 86.24% 83.25% -3.00%
- Complexity 4211 4741 +530
============================================
Files 1169 1228 +59
Lines 46736 51386 +4650
Branches 5203 5847 +644
============================================
+ Hits 40308 42780 +2472
- Misses 4691 6715 +2024
- Partials 1737 1891 +154
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 367 | 0.224 | 24,564/44,465/44,465 us | 🔴 +41.0% / 🔴 +171.3% |
| ⚪ | bs=100 sw=10 sl=64 | 793 | 0.484 | 124,540/141,683/141,683 us | ⚪ within ±5% / 🔴 +25.9% |
| 🔴 | bs=1000 sw=10 sl=64 | 907 | 0.553 | 1,088,989/1,298,526/1,298,526 us | 🔴 +6.0% / 🔴 +20.7% |
Baseline details
Latest main 88ca47f from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 367 tuples/sec | 423 tuples/sec | 743.73 tuples/sec | -13.2% | -50.7% |
| bs=10 sw=10 sl=64 | MB/s | 0.224 MB/s | 0.258 MB/s | 0.454 MB/s | -13.2% | -50.7% |
| bs=10 sw=10 sl=64 | p50 | 24,564 us | 24,064 us | 13,130 us | +2.1% | +87.1% |
| bs=10 sw=10 sl=64 | p95 | 44,465 us | 31,537 us | 16,391 us | +41.0% | +171.3% |
| bs=10 sw=10 sl=64 | p99 | 44,465 us | 31,537 us | 19,408 us | +41.0% | +129.1% |
| bs=100 sw=10 sl=64 | throughput | 793 tuples/sec | 826 tuples/sec | 944.04 tuples/sec | -4.0% | -16.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.484 MB/s | 0.504 MB/s | 0.576 MB/s | -4.0% | -16.0% |
| bs=100 sw=10 sl=64 | p50 | 124,540 us | 120,038 us | 105,853 us | +3.8% | +17.7% |
| bs=100 sw=10 sl=64 | p95 | 141,683 us | 148,526 us | 112,493 us | -4.6% | +25.9% |
| bs=100 sw=10 sl=64 | p99 | 141,683 us | 148,526 us | 122,200 us | -4.6% | +15.9% |
| bs=1000 sw=10 sl=64 | throughput | 907 tuples/sec | 927 tuples/sec | 972.22 tuples/sec | -2.2% | -6.7% |
| bs=1000 sw=10 sl=64 | MB/s | 0.553 MB/s | 0.566 MB/s | 0.593 MB/s | -2.3% | -6.8% |
| bs=1000 sw=10 sl=64 | p50 | 1,088,989 us | 1,072,675 us | 1,033,856 us | +1.5% | +5.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,298,526 us | 1,225,461 us | 1,076,083 us | +6.0% | +20.7% |
| bs=1000 sw=10 sl=64 | p99 | 1,298,526 us | 1,225,461 us | 1,107,701 us | +6.0% | +17.2% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,544.97,200,128000,367,0.224,24564.02,44464.60,44464.60
1,100,10,64,20,2523.41,2000,1280000,793,0.484,124540.29,141682.77,141682.77
2,1000,10,64,20,22058.75,20000,12800000,907,0.553,1088989.31,1298525.63,1298525.63|
/request-review @aglinxinyuan |
|
@carloea2 Can you review it first? |
|
Backport PR opened: draft #7495 (#7495) to |
What changes were proposed in this PR?
TextInputSourceOpExeccomputed its line window asslice(offset, offset + limit.getOrElse(Int.MaxValue)). With an Offset set and the Limit left empty, the addition overflowsIntto a negative bound, which Scala 2.13'sIterator.sliceclamps to 0 and then returns an empty iterator — so the operator silently emitted zero rows while the workflow reported success. Any Offset ≥ 1 with an empty Limit is affected, and an explicit large Limit (e.g.Int.MaxValue) overflows the same way. This contradicts the Limit property's own description, "Leave empty to read all lines."This PR replaces the slice with
drop(offset)+take(limit), the same idiom the CSV, Arrow, and JSONL scan sources already use. There is no addition, so nothing can overflow; every configuration that previously worked is unchanged (verified case-by-case, including negative offsets andisSingleattribute types, which keep ignoring offset/limit as documented).Before the fix (current
main) — Offset = 1, Limit left empty, five-line inputa b c d e: the result is an empty set even though the workflow completes successfully. Expected: the four rowsb, c, d, e.Control on the same build — Offset = 0, Limit left empty returns all five rows:
Any related issues, documentation, discussions?
Closes #7346.
Same class of defect as #7245 (JSONL File Scan dropping rows when Offset is set), which was fixed by #7247.
How was this PR tested?
TDD: the regression tests were written first and confirmed to fail on the unfixed code — the offset-without-limit case and the offset-with-
Int.MaxValue-limit case both produced empty output — then the fix was applied and all tests pass.Seven new cases were added to
TextInputSourceOpDescSpec(the spec that already exercisesproduceTuple()): offset without limit, offset with anInt.MaxValuelimit, offset+limit window, limit only, offset at/past the end of the input, negative offset treated as zero, andSINGLE_STRINGignoring offset/limit (documented behavior, pinned).Also verified manually in the UI with the same two-operator workflow shown in the screenshots above:

Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Claude Fable 5)