Skip to content

Native: GraphBatch.create_vertices sends columns and reads the RIDs from an int64 buffer - #390

Merged
tae898 merged 18 commits into
mainfrom
perf/388-native-graph-batch-frame
Nov 16, 2026
Merged

tae898 merged 18 commits into
mainfrom
perf/388-native-graph-batch-frame

Conversation

@tae898

@tae898 tae898 commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

On the native wheel, GraphBatch.create_vertices sent every chunk of rows to the library as one JSON text (json.dumps of the rows, a parse in Java) and read the new RIDs back as a JSON list of text. A chunk of rows with the same keys in the same order and plain values now crosses as columns by address, in the layout of arcadedb_n_insert_colp (the post1 insert_columns route), and the library writes the (bucket, position) of each new vertex into an int64 buffer that Python hands over. Python turns that buffer into the same list of str (#bucket:position, in row order) that the wrapper has always returned.

The Java side is the new entry point arcadedb_n_gb_colp in NativeExtras.java. It builds the Object[][] that GraphBatch.createVertices(type, Object[][]) takes from the columns and calls that same engine method, so the transaction, the retries and the commit are the engine's. BulkOps now holds the column table (BulkOps.Table: the parse, the length checks and cell(c, i), the one place that reads a value out of a buffer) for both the document insert and this entry; setRow is a loop over cell.

Python: bulk.create_vertices_columns (rows to columns with columnar.rows_to_columns(rows, same_order=True), columnar.pack, one crossing, bulk.rids_of) is tried first by NativeGraphBatch._create_chunk. ARCADEDB_NATIVE_GB_COLP=0 turns the route off (as the other native switches), so an A/B can run in one image.

What keeps the JSON route

A chunk the columns cannot carry is sent as before, so nothing is lost or changed: rows with other keys or the same keys in another order, a None row among dict rows, a nested list or map, a column of mixed types (int and float, bool and int), a set, a numpy scalar, a float64 or ragged or 2-D array, an integer past 64 bits, a non-finite Decimal, a lone surrogate, a pandas Timestamp with nanoseconds (microsecond ticks would drop them; this guard is in the shared columnar packer, so insert_many gets it too), an empty list, and the count form (create_vertices(type, 5)). The check is per chunk, so one such chunk does not move the others off the new route.

The stored property order is the one the JSON route gives: Java reads each JSON row into a fresh HashMap, so a vertex was stored in the iteration order of that map, not in the order of the dict. BulkOps.vertexRows orders the names the same way (a HashMap that takes them in column order), and rows with the same keys in another order keep the JSON route, so the order cannot differ. (Whether the dict order would be the better order for vertices is a separate question; this PR does not change it.)

Related issues

Refs #388 (rank 7) and #315.

Performance changes

Not measured yet. No speed or memory claim is made here; the pre-registered A/B decides.

Figure definition for the A/B: the operation is NativeGraphBatch.create_vertices(type, rows) on the native wheel (arcadedb) with the default commit_every (1000 rows per crossing and per transaction), rows as dicts with the same keys: (a) the l2 shape, gb.create_vertices('Person', chunk) with int and str properties, and (b) the e2 shape, b.create_vertices('Product', rows) with a float32 vector property as numpy arrays and as lists on a declared ARRAY_OF_FLOATS. Sizes: 1, 100 and 100,000 rows (the unit is one create_vertices call; also report per row). The A arm is the same wheel with ARCADEDB_NATIVE_GB_COLP=0; the B arm is the default. The no-change control is a shape that keeps the JSON route in both arms (rows with nested values) and an A/A pair of the A arm. Latency and peak RSS and anon growth are information figures. The parity digest is the dump of every stored record (property order, type and repr of each value) plus the returned RID list, as in tests/test_native_graph_batch_frame.py.

What the change allocates per chunk (at most 1000 rows by default): in Python, one list of values per column (one pointer per row), one int64 or float64 array per numeric column (8 bytes per row), per string column the joined text, its UTF-8 bytes and an int32 offsets array (4 bytes per row), a null bitmap (one bit per row) where a column has a null, a float32 matrix for a vector column (4 bytes per row and dimension), the 6-long address table, the RID buffer (16 bytes per row) and the list of RID texts, as before. The JSON text of the chunk, its UTF-8 copy in Java and the reply text are gone. In Java the Object[n][2k] of the rows is built as before (the engine takes it); no HashMap per row and no parsed tree.

Docs

No page describes the entry points of the native library or this route; the switch is documented where the others are, in src/arcadedb/_native/lib.py.

Checklist

  • A regression test fails on the old code and passes on the new (the stub tests in tests/test_native_graph_batch_frame_pack.py need bulk.create_vertices_columns, bulk.rids_of and rows_to_columns(same_order=...), which the old code does not have; the first commit holds the tests alone, so a run of it shows them failing. I have not run it: the native image build is CI only.)
  • The test suite that the change can affect passes (JVM and native runs dispatched on this branch)
  • uvx pre-commit run --files <changed files> passes, and Bandit as CI runs it

🤖 Generated with Claude Code

tae898 and others added 11 commits November 14, 2026 07:34
…ject_hook and without reading its side array (refs #315)

to_list() and iter_dicts() of the JVM wheel build a json object_hook for every batch and read the
side array of every batch, whether or not the batch holds a tagged value. These tests count the hook
factory, the object_hook argument of json.loads and the batch elements that are read, for a result of
plain values (no hook, no read of the side array) and for every kind of value the bridge tags (the
hook stays). They also compare to_list() and iter_dicts() with the decoding that always builds the
hook, over single shapes, a mixed result and batch boundaries.

They fail on the current decoder (every batch gets a hook and a read of the side array) and pass once
the decoder looks for the tag escape in the batch text first.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s text without an object_hook and without reading its side array (refs #315)

Every batch was decoded with json.loads(text, object_hook=_typed_hook(side)) and its side array was
read first, also when the batch held only plain values. The hook is a Python call per JSON object and
the read of the side array is a JPype operation per batch, which for iter_dicts (one row per crossing)
is one per row.

TypedRows writes every tag as {"\u0001": [kind, payload]} with the escape spelled out, writes a control
character of a string as the same escape, and never writes the raw character, and every side array
entry is referred to by a tag in the text. A text without the escape therefore holds no tag and no side
index, the hook would return each object unchanged, and plain json.loads gives the same value. The
check is a substring scan of a str that already exists (_loads_typed_row did it for one row). A string
that holds U+0001 or the text of the escape only keeps the hook, which then changes nothing.

No Java change, and the native wheel is untouched. The speed effect is not measured: it waits for a
timed A/B of a candidate wheel against 26.11.1.post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…taframe on the JVM wheel (refs #315)

Round trips of every Arrow temporal type through a real load on both wheels (every timestamp unit with and without a zone, to the nanosecond; date32 and date64 before 1970; nulls; batches; vertex and parallel loads; pandas and Polars frames), and JVM-only tests that count what one load does: no java.time call and no convert_python_to_java per cell, and each timestamp or date column reaching the bridge as one TemporalColumn per batch. The counting tests fail on the code before, which builds one LocalDateTime or LocalDate per cell in Python and converts every element again.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…te column crosses as one long[] of epoch ticks, not one JPype call per cell (refs #315)

_temporal_to_java made one LocalDateTime.ofEpochSecond or LocalDate.ofEpochDay call per cell, and insert_columns converted every element of that list again with convert_python_to_java: a million-row timestamp column was two million JPype calls. The column is now the integer epoch values in one long[] (plus one boolean[] when a row is null) wrapped in the new bridge class TemporalColumn (dateTimes(long[] ticks, long ticksPerSecond, boolean[] nulls) and dates(long[] ticks, long ticksPerDay, boolean[] nulls)), and DocumentBatcher builds each LocalDateTime or LocalDate as it fills the document of that row, as StringColumn does for text. The reading rules are the ones of the code before: UTC wall clock, a zoned timestamp is already UTC epoch time, a naive one is the clock it shows, the unit's precision is kept (a nanosecond tick is not cut to microseconds), a tick before the epoch floors, date64 is milliseconds divided by 86,400,000 with a floor. The factory checks the smallest and the largest tick, so a value past the years of java.time still raises before a row is written. The per-cell path is retired: the wheel carries its own bridge jar, so no fallback for an older one. Not measured: the speed claim waits for a timed A/B against 26.11.1.post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ly text, no free and no JSON decode (refs #315)

arcadedb_n_tx returns the transaction state as one integer (bit 0 active, bits 1 to 31 depth, bits 32 to 62 commits, -1 for an
error whose text lib.last_error holds), as arcadedb_n_cmd returns a cursor id. These tests pin that and that an error still
reaches Python as the same typed exception with the same message:

- with db.transaction() is three foreign calls (mark, begin, commit), all arcadedb_n_tx: no strlen, no copy into a str, no
  arcadedb_free (was twelve);
- begin, commit, rollback, is_transaction_active and the mark each make exactly one foreign call;
- no JSON decode in a transaction, not even for the mark that reads active and depth;
- the layout of the integer, -1 for an unknown handle with the error in last_error, commit without begin, rollback with nothing to
  roll back, a failed statement in a block, nested with blocks, an injected failure on each entry point, a database closed with a
  transaction open.

They fail on the old code: lib.tx does not exist there, and the foreign-call and decode counts are higher.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…not a JSON text (refs #315)

A begin, a commit, a rollback and a mark each crossed into the library and got back a malloc'd JSON text of about 60 bytes that
Python read with strlen, a copy into a str and arcadedb_free, and mostly never looked at. with db.transaction() is three crossings
(the mark, begin, commit), so twelve foreign calls and three strings built, copied and freed.

arcadedb_n_tx now returns a long: bit 0 active, bits 1 to 31 depth, bits 32 to 62 the low 31 bits of the commit count, -1 for an
error. It is the way arcadedb_n_cmd returns a cursor id: the error text is kept per thread and read with arcadedb_n_last_error, built
by the same errorJson as before, so the typed exception and its message do not change. Each crossing is one foreign call, and
with db.transaction() is three. The Python side reads the fields of the integer inline, so the mark, the begin and rollback states
the unwind and run_in_transaction read, and is_transaction_active no longer decode anything.

Not measured: the speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ossings, not three (refs #317)

The transaction check of a saved record's save() cannot refuse while auto-transaction is on, yet it
still crossed into the library (arcadedb_n_tx) before the SELECT and the UPDATE. These tests pin two
crossings there, bare and inside a transaction, with and without the read-back of the row, and pin
what must not move: the check is back the moment auto-transaction is off (first, before any read),
a stale record is refused under auto-transaction too, and the stored values and the next save's
baseline are those of an update inside a transaction for NULL, float, text, date-time, date,
Decimal, nested values, a float32 array and a value set unchanged.

The two-crossing tests fail on the code before the change (three crossings); the rest pass on both.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…te when auto-transaction is on (refs #317)

_save_update asked is_transaction_active() (an arcadedb_n_tx crossing) first and only then looked at
the auto-transaction flag, so the answer was thrown away whenever the flag was on. The operands are
swapped: the flag is a Python attribute, the state is a crossing. With auto-transaction on, an update
of a saved record is two crossings (the SELECT of the stale-record check and the UPDATE), not three.
With it off nothing changes: the check still comes first, outside a transaction and inside one.

The check is not replaced by the engine's refusal for the default mode: the engine runs the
before-update callbacks and the property validation before it looks for a transaction (read in
LocalDatabase.updateRecord of the 26.12.1-SNAPSHOT source), and an UPDATE of a record that is gone
raises nothing, so a save outside a transaction would run callbacks, or pass silently, instead of
raising "Transaction not begun" before it reads. A Python-side record of the transaction state is not
exact either: a SQL COMMIT or ROLLBACK statement ends the transaction without passing the wheel's
transaction calls.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… no failed assert leaves a transaction open (refs #315)

CI of the first push (run 43478436168) failed two tests on a wrong assumption of mine: I expected the outer begin to raise the depth
by one. LocalDatabase.getNestedTransactions() is the size of the thread's stack of transaction contexts
(DatabaseContext.DatabaseContextTL.transactions). The stack always holds the thread's base context once the thread has used the
database, so the depth is 1 while idle. The outer begin reuses that base context (depth 1, active). Only a begin while a transaction
is active pushes a context (2), the commit or rollback of that level pops it (1, still active), and the outermost end leaves the base
context on the stack (1, idle). popIfNotLastTransaction never removes the last one. The native tx entry point returns exactly that
number, as the old JSON reply did, so the change is right and the two expectations were not.

The fixture now rolls back whatever a test left open before it drops the database, so a failed assert is one failure and not also a
"Cannot drop the database in transaction" teardown error. The close-and-reopen test does the same in a finally.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… from an int64 buffer; the stored vertices match the JSON route (refs #388, #315)

Stub-library tests (both backends) for the routing, the header, the chunking, the RID text and every shape that keeps the JSON route; a native-only
parity test that loads the same rows with the route on and off and compares the stored records, their property order and the RIDs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…the RIDs in an int64 buffer (refs #388, #315)

arcadedb_n_gb_colp takes the address table of arcadedb_n_insert_colp, builds the rows with the objects and the property order the JSON rows give
(BulkOps.Table.cell, BulkOps.vertexRows) and calls the same GraphBatch.createVertices(type, Object[][]); (bucket, position) of each new vertex are
written into a buffer of the caller. Rows with other keys or another key order, a nested value, a mixed column, a set, a numpy scalar, an integer
past 64 bits, a lone surrogate or a pandas Timestamp with nanoseconds keep the JSON route. ARCADEDB_NATIVE_GB_COLP=0 turns it off for A/B runs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898 tae898 added python Pull requests that update python code performance Speed or memory of the bindings, with a measurement native The arcadedb package (native library build) labels Nov 14, 2026
@tae898 tae898 self-assigned this Nov 14, 2026
…ch frame routing test (refs #388, #315)

A patch of the first commit never applied (the test runner call in the same command was blocked), so the file kept: a schema row for a type with an
INTEGER property while the test sent numpy int32 scalars (a list of numpy scalars is no column: the JSON route, correctly), a stand-in that read
body["rows"] for the count form (KeyError), and a 21-row load in chunks of 10 that made two chunks where the test expected three. None of these
was JVM-specific and none was a fault of the route. The declared-property test now sends a declared ARRAY_OF_FLOATS list column, the stand-in reads
the count form, the chunk test has three chunks with the odd key in the middle one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898 and others added 6 commits November 14, 2026 17:22
…properties, and only this route refuses Timestamps with nanoseconds (refs #388, #315)

Two faults of the first commit that the native parity run showed:

1. Property order. The JSON route took float32 array columns out of the rows (take_vector_columns) and appended them after the other properties
   (NativeExtras.withVectors), so a vertex of {i, v} was stored [i, v], while the frame route stored the HashMap order [v, i]. The header now
   carries "last" (the float32 array columns), BulkOps.vertexRows puts them after the HashMap-ordered rest, in column order. Declared
   list-of-float columns stay in the HashMap order, as in the JSON rows (the parity shapes 'declared vectors from lists' and 'from float32 arrays'
   cover both).
2. Timestamps. The nanosecond guard sat in the shared packer and so changed insert_many, whose JSON route keeps microseconds
   (test_native_columnar_temporal.py::test_pandas_timestamps_keep_microseconds_as_the_json_path_does and
   test_native_mixed_chunk.py::test_a_pandas_timestamp_is_stored_as_the_statement_path_stores_it, red since that commit). pack() and _list_column() take
   exact_nanos (default off); only create_vertices_columns sets it, because GraphBatch's JSON route writes isoformat() and keeps the nanoseconds.

Own tests fixed: 'mixed int and float' with one row is a column of one float and is carried; the two refusals differ only by the type name in the
message; the pack test now pins both packer modes and the 'last' header; new end-to-end test of the paper shapes (Person rows in chunks, Product rows
with float32 vectors: only arcadedb_n_gb_colp crossings, no create_vertices JSON crossing, no codec hook, vectors stored bit-equal).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…es follow the new route (refs #388, #315)

test_native_ingest_defaults.py (3 tests: default commit_every is 1000, explicit commit_every 250 and 0): asserted the rows per create_vertices crossing
[1000,1000,500,1000,1000,500], [250]*10 and [2500] by wrapping NativeGraphBatch._op. Plain rows now cross as columns and do not pass through _op, so the
spy also records len(rows) at bulk.create_vertices_columns; the expected numbers are unchanged (the chunking is the same).

test_native_graph_batch_binary_vectors.py (7 tests) and test_native_vertex_vector_crossings.py (6 tests): asserted that create_vertices of float32
vector rows crossed arcadedb_n_gb_bin (or arcadedb_n_gb for the text arm) with the side buffer, the "vecs" entry and no per-row codec hook. A plain
vector load now crosses arcadedb_n_gb_colp (tests/test_native_graph_batch_frame*.py assert that, and the stored bits). These files' subject is the
side-buffer route, which a chunk the columns cannot carry (a nested value, ragged vectors) still takes, so their db fixture turns the columnar
route off (lib.GB_COLP False); no assertion is loosened.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…route of post3 (refs #388, #401)

The laptop A/B of this pull request measured a single row 1.63 times slower than post3 (n_gb_v_1) and batches of 100 rows and more faster: the columnar entry has a fixed cost (a header, the column specs, the buffers) that a handful of rows does not repay.

A chunk below lib.GB_COLP_MIN_ROWS (64, ARCADEDB_NATIVE_GB_COLP_MIN_ROWS, 0 turns the floor off) now takes the post3 JSON route; at or above it the columns, as before. Sizes 1 and 100 are measured and the crossover lies between them; below 64 the path is the post3 path, 100 and up is measured faster, 64 to 99 is not measured. The check is in bulk.create_vertices_columns, so a small chunk reads no schema and builds no columns.

Tests: the stand-in library of the pack tests turns the floor off so the existing tests keep pinning the columnar route; new tests pin the floor at 1, 63, 64, 65 and 100 rows, that a small chunk builds nothing, that chunks are judged one by one, the switch and the environment variable. The native parity test loads the same rows with the default floor, with the JSON route and with the columns forced, at 1, 63, 64, 65 and 100 rows, and compares every stored record and the RIDs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ro that placed the crossover (refs #388, #401)

An untimed micro of create_vertices calls in one open graph batch, this change against post3, native: slower below 40 rows, faster from 64 rows. No code change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…main; refs #388, #401)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nment test, as the repo does (refs #388, #401)

The Bandit job of CI (bandit==1.9.4 -c pyproject.toml -r src tests, low severity and confidence) flagged the import and the call of the test that starts a fresh interpreter to read lib.GB_COLP_MIN_ROWS. The call is an argv list with no shell: this interpreter and a fixed one-line program.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

native The arcadedb package (native library build) performance Speed or memory of the bindings, with a measurement python Pull requests that update python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant