Repository navigation
Conversation
…op turn out() and the float32-tail reply wrote the reply with p.write(i, b[i]) in a Java loop. Both now fill a direct ByteBuffer view of the malloc'd block (Unsafe.copyMemory); the bytes written are the same. New test: replies of one, several and many batches, multi-byte text, megabyte rows and vectors round-trip exactly. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The previous commit widened off + 4 * sink.n to long, which UnmanagedMemory.malloc does not accept (found by the first image build). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…hort, empty and missing vectors Passes against the current library (vectors of later batches travel as number text); it must keep passing when arcadedb_n_next carries them in a float32 tail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e float32 tail arcadedb_n_exec flag 4 already sends float[] values of 8 or more elements in a binary tail, but only for the first reply (256 rows). Every later batch (arcadedb_n_next, 2048 rows) sent them as number text, which Python parses with json.loads. New entry point arcadedb_n_next_bin encodes a batch with the same sink; the tail code of replyOf is now one helper (withFloatTail) used by both. lib.py uses it when the library has it (ARCADEDB_NATIVE_NEXT_BIN=0 and ARCADEDB_NATIVE_BIN_REPLY=0 turn it off); arcadedb_n_next is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 25, 2026
Author
|
Closing here: the bindings moved to ArcadeData/arcadedb-python, and this change is merged there as ArcadeData/arcadedb-python#76 (later result batches carry vectors in the float32 tail). The branch stays. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRAFT. Built into one image and measured untimed (section Measured below: wide SELECTs about 2x faster, decoded rows bit-identical to the JVM wheel); a timed ab.py A/B is queued. Do not merge before it.
STACKED ON #520: the vector-tail code in
ArcadeNative.replyOfis the block #520 changes to a bulk copy, so this branch starts fromnative-result-copy; its first two commits are #520's. Review the last two commits (test, implementation); when #520 merges this PR needs only a rebase.Problem (found by the study in
.notes/bench/repros/native-result-transport-20261025, local notes repo):arcadedb_n_execwith flag 4 sendsfloat[]values of 8 or more elements in a binary float32 tail, but only for the first reply (256 rows).arcadedb_n_next(every later batch, 2048 rows) has no sink, so an embedding column comes back as JSON number text from row 257 on, andjson.loadswith the object_hook is about half of the wall time of a 10k-100k row SELECT with a 32-float vector (untimed: 77 of 159 ms, 873 of 1,680 ms).Change:
arcadedb_n_next_bin(rs, max): the same batch asarcadedb_n_next, encoded with the float sink. The tail code ofreplyOfis one helper,withFloatTail, used by both.arcadedb_n_nextis unchanged, so an older Python on this image keeps working.lib.py:NEXT_BIN(on when the library has the entry point;ARCADEDB_NATIVE_NEXT_BIN=0turns it off for A/B runs in one image).NativeResultSet._fillreads batches withlib.call_bin("arcadedb_n_next_bin", ...)andloads(text, floats)whenNEXT_BINand the binary reply are on, else as before.struct.pack('<d'), so -0.0 differs from 0.0) on the first and on later batches; vectors of 0, 3, 8 and 32 floats, float32 extremes (1e-45, 3.4028235e38, +/-inf), and rows with no vector at all; the same read through the row-by-row cursor.The untimed numbers are in the Measured section below (notes folder
native-next-float-tail-20261025). Still to come: the timed A/B.Measured (2026-10-25, untimed, laptop, 2 runs per arm, one image built from this PR's tree)
Untimed: these are stage clocks from a probe, not an ab.py A/B; a timed A/B is queued before any merge.
Wide SELECT with a 32-float vector, microseconds per warm query (mean of runs; pass B stage clocks)
sha256 of decoded rows (first 20 hex), floats compared as the 8 bytes of the double
🤖 Generated with Claude Code