Skip to content

Fixes for parallel block reading - #98

Merged
leonardehrenfried merged 4 commits into
masterfrom
parallel-block
Sep 14, 2026
Merged

leonardehrenfried merged 4 commits into
masterfrom
parallel-block

Conversation

@leonardehrenfried

@leonardehrenfried leonardehrenfried commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Reviewed, through Claude, by @simon04.

@leonardehrenfried

Copy link
Copy Markdown
Collaborator Author

Once this is released I will test this with OpenTripPlanner, which reads large-ish OSM extracts. Then I can give more information about how failure-tolerant this implementation is.

@simon04

simon04 commented Sep 11, 2026

Copy link
Copy Markdown
Member

Claude Code review:

Re-reviewed after bdb1c8d. All four points from the last round are addressed — I compiled the tree with javac and ran the new tests plus a few probe harnesses rather than just reading:

  • EOF around the block body — now matches BlockInputStream, which catches EOFException around the whole of FileBlock.process. Verified on a truncated sample.pbf (truncation landing inside a block body): both readers emit Got header block. … Complete!, so the new test's premise holds.
  • In-flight futures — cancel + shutdownNow() works. With a throwing handleBlock the exception propagates, complete() is correctly skipped, and live pool threads drop to 0. The shared-executor path is clean too: the caller's executor isn't shut down, stays usable, and no stale interrupt flag leaks into subsequent tasks.
  • Backpressure test — good catch on pipelineDepth never filling with only 4 blocks; the depth-2 test exercises the branch now, and pinning truncation behaviour against BlockInputStream instead of hard-coding the expected string is the right call.
  • Skip read-ahead — documenting the caveat rather than serialising delivery seems like the right trade-off. Instrumented traces confirm the note is accurate: depth N gives exactly N blocks of skipBlock lookahead, and depth 1 is fully sequential.

LGTM. Three doc nits, all non-blocking:

  • The note tells a stateful adaptor to "use pipelineDepth 1", but the int-taking convenience constructor treats its argument as numThreads and sets pipelineDepth = numThreads * 2 — so new ParallelBlockInputStream(in, adaptor, 1) gives depth 2 and keeps one block of read-ahead. Depth 1 needs the 4-arg constructor; might be worth saying so.
  • {@link BlockReaderAdapter#handleBlock} is the wrong anchor for the ordering contract — handleBlock's javadoc is just "Called with the data in the block."; the "in file order" guarantee is documented on skipBlock.
  • "up to pipelineDepth blocks ahead" counts only non-skipped blocks; a long run of skipped blocks never touches inflight, so skipBlock can run further ahead than that.

Separately, and not for this PR: no test in the repo makes skipBlock return true, so skipContents is uncovered for both readers — worth a follow-up issue given a short read there is only caught by an assert.

@joto

joto commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Once this is released I will test this with OpenTripPlanner, which reads large-ish OSM extracts. Then I can give more information about how failure-tolerant this implementation is.

Doesn't it make more sense to first test it and the release it?

@leonardehrenfried

Copy link
Copy Markdown
Collaborator Author

Once this is released I will test this with OpenTripPlanner, which reads large-ish OSM extracts. Then I can give more information about how failure-tolerant this implementation is.

Doesn't it make more sense to first test it and the release it?

I should have been a bit more precise. I have tested this in OTP and it worked even before this PR. What I wanted to say is that we will only get feedback on the complex back pressure and failure scenarios when this is used in OTP dev builds, which require a release.

@joto

joto commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Ah, that makes sense. :-)

I don't now enough about Java to say anything about the code. If you think it is ready, merge it.

@leonardehrenfried
leonardehrenfried merged commit 9f01ad3 into master Sep 14, 2026
26 checks passed
@leonardehrenfried
leonardehrenfried deleted the parallel-block branch September 14, 2026 11:18
social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
osm-pbf 1.8.0

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>## What's Changed
* Allow decoding of blocks in parallel by @leonardehrenfried in openstreetmap/OSM-binary#96
* Upgrade protobuf version by @leonardehrenfried in openstreetmap/OSM-binary#97
* Fixes for parallel block reading by @leonardehrenfried in openstreetmap/OSM-binary#98


**Full Changelog**: https://github.com/openstreetmap/OSM-binary/compare/v1.7.0...v1.8.0</pre>
  <p>View the full release notes at <a href="https://github.com/openstreetmap/OSM-binary/releases/tag/v1.8.0">https://github.com/openstreetmap/OSM-binary/releases/tag/v1.8.0</a>.</p>
</details>
<hr>

See merge request: Harmonybrew/homebrew-core!20194
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants