Skip to content

Allow decoding of blocks in parallel - #96

Merged
leonardehrenfried merged 1 commit into
masterfrom
parallel-block
Sep 11, 2026
Merged

leonardehrenfried merged 1 commit into
masterfrom
parallel-block

Conversation

@leonardehrenfried

Copy link
Copy Markdown
Collaborator

This adds parallel file block reader which decodes the zip stream in parallel but still delivers the decoded data as a single stream.

This makes it possible to use multiple threads to speed up decoding.

@leonardehrenfried
leonardehrenfried merged commit 9d707b5 into master Sep 11, 2026
26 checks passed
@simon04

simon04 commented Sep 11, 2026

Copy link
Copy Markdown
Member

Claude Code finding (feel free to ignore):

  1. ParallelBlockInputStream.java:89 — skip/handle ordering violates the "drop-in" contract. Read-ahead calls skipBlock() for later blocks before handleBlock() runs for earlier ones. The class javadoc promises "one at a time, in order"; a stateful adaptor that decides what to skip based on what it has seen so far will make those decisions on stale state.

  2. ParallelBlockInputStream.java:95 — EOF handling only wraps readHead. A truncated block body throws out of process(), so adaptor.complete() never runs and up to pipelineDepth already-read blocks are silently dropped. BlockInputStream tolerated truncation and completed normally.

  3. ParallelBlockInputStream.java:106 — in-flight futures aren't cancelled on the error path. With a caller-supplied executor, nothing happens at all; an owned executor gets shutdown() rather than shutdownNow(), so the queued decode backlog keeps expanding blobs (up to 32 MB each) whose results nobody will ever read.

  4. ParallelBlockInputStreamTest.java:23 — the test doesn't exercise the interesting paths. numThreads=4 → pipelineDepth=8, but sample.pbf has only 4 blocks, so neither the backpressure delivery branch nor the skip branch is ever hit. A reordering bug in the real-world path would pass green.

social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
osm-pbf 1.8.0

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>## What's Changed
* Allow decoding of blocks in parallel by @leonardehrenfried in openstreetmap/OSM-binary#96
* Upgrade protobuf version by @leonardehrenfried in openstreetmap/OSM-binary#97
* Fixes for parallel block reading by @leonardehrenfried in openstreetmap/OSM-binary#98


**Full Changelog**: https://github.com/openstreetmap/OSM-binary/compare/v1.7.0...v1.8.0</pre>
  <p>View the full release notes at <a href="https://github.com/openstreetmap/OSM-binary/releases/tag/v1.8.0">https://github.com/openstreetmap/OSM-binary/releases/tag/v1.8.0</a>.</p>
</details>
<hr>

See merge request: Harmonybrew/homebrew-core!20194
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants