perf: remove two redundant object-store calls in datasets metadata fetch

[prabhaks]

Summary

Two small, low-risk fixes in the object-store I/O behind POST /api/prism/v1/datasets, which has been observed taking 50s+ to respond on multi-TB, multi-day deployments even on a 44 vCPU / 192GB query node.

• get_manifest_list (src/query/mod.rs) unconditionally fetched the base stream.json via get_stream_json, but on Mode::Query/Mode::Prism nodes that result is never read (those modes merge every ingestor's stream.json instead via get_all_stream_jsons). Moved the fetch into the else branch that actually needs it (Mode::All).
• get_first_and_latest_event_from_storage (src/storage/object_storage.rs) awaited its two independent extract_timestamp_for_date calls sequentially. Replaced with tokio::try_join! to run them concurrently, matching the existing tokio::join! pattern already used one call up the stack in get_prism_logstream_info.

Both changes are behavior-preserving on the success path (same I/O, fewer/faster round trips). Two notes worth flagging for review:

• The get_manifest_list change means a failed/corrupt base stream.json on a Query-mode node no longer short-circuits the whole call — it now proceeds to get_all_stream_jsons instead, which can succeed where the old code would fail. This seems like a desirable side effect but is a real change in failure-path behavior, not just a speedup.
• These are part of a larger investigation into /datasets latency (bundles stats/retention/hottier/info/counts/query per stream); the dominant cost — a manifest-list scan that's unbounded by stream history when no retention policy is configured — needs a larger catalog-level change and is intentionally not part of this PR. Happy to open a separate issue for that once there's an associated tracking issue for this one (the Parseable team mentioned they'd be opening one).

Test plan

• cargo test --lib — 426 passed, 0 failed (full existing suite, no regressions)
• cargo fmt --check — clean
• cargo clippy --lib — clean
• Validate against a real multi-TB test cluster (in progress, pending network access)

Summary by CodeRabbit

• Performance Improvements
  • Improved storage processing by retrieving earliest and latest event timestamps concurrently.
• Bug Fixes
  • Refined manifest loading so each query mode uses the appropriate stream data and snapshot information.
  • Preserved error handling when retrieving event timestamps.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a804f321-b4f0-47ea-83c6-e294e587a1ed

📥 Commits

Reviewing files that changed from the base of the PR and between c018bb7 and c7a86fb.

📒 Files selected for processing (2)

• src/query/mod.rs
• src/storage/object_storage.rs

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

Walkthrough

The query code now loads the base stream JSON only for non-Query/Prism modes. The storage code now performs minimum-date and maximum-date timestamp lookups concurrently while preserving error propagation.

Changes

Query manifest loading

Layer / File(s)	Summary
Mode-specific manifest loading src/query/mod.rs	Query and Prism modes no longer load the base stream JSON before merging ingestor manifests. Other modes load and deserialize it inside their branch.

Storage timestamp lookup

Layer / File(s)	Summary
Concurrent timestamp extraction src/storage/object_storage.rs	Minimum-date and maximum-date timestamp lookups now run with tokio::try_join!. Errors from either lookup still propagate.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to c7a86

The PR removes a redundant object-store request and parallelizes two independent reads to reduce dataset metadata latency while preserving successful-path behavior; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: parmesant

Poem

A rabbit hops through query streams,
While timestamps race like silver beams.
Base JSON waits its proper turn,
Two lookups spin, then errors burn.
“Neat and quick!” the rabbit sings.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the performance-focused object-store I/O changes, although the second change reduces sequential latency rather than removing a redundant call.
Description check	✅ Passed	The description explains the motivation, implementation, failure-path impact, scope, and test results, with the remaining cluster validation clearly marked in progress.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}