Repository navigation
Conversation
This was referenced Nov 14, 2026
…nd the JSON route (refs #388) Parity (differential) tests for a typed-column route of the JVM insert_many that does not exist yet, so they fail on the parent: each stores the same rows with the typed route switched off and as shipped, and compares every record property by property through SQL (names, Java class of each value, value), for ints around 2^31 and 2^53 and 2^63, floats (NaN, infinity, subnormal, -0.0), strings, dates, date-times, nulls, None against a missing key, key order, mixed columns, date and Decimal inside lists, bytes, numpy scalars and aware date-times. Also the crossing counts, the order of mixed chunks, the commit boundary, the parallel mode, vertices, edge types, a duplicate key and a value no route can store. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… one JSON text (refs #388) A chunk whose rows are dicts with the same str keys in the same order, and whose columns each hold None and values of one exact type (int within 64 bits, float, bool, str, date, naive datetime), is read as columns: one numpy buffer per int, float or bool column, one StringColumn per string column, one TemporalColumn per date or datetime column, an Object[] only for a numeric column with a null. DocumentBatcher.insertManyColumns (and the parallel twin) builds the documents from them, and stores a whole number as the JSON text route did, an Integer when it fits an int and a Long beyond, so the stored type of a schemaless property does not depend on the route (insert_columns is unchanged and keeps Long). Every other chunk keeps the JSON text and the per-row fallback exactly as before: the whole chunk is judged first, from the types alone, before any bridge array is built. Bridge jar and Python only, no native image. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… its whole numbers Long, as the per-row path stores them (refs #388) The first CI run of the typed route (tagged 26.11.1, workflow_dispatch 43588034127, all 9 finished legs) failed 5 tests of tests/test_insert_many_typed_columns.py, the same 5 on every leg. Cause and evidence: - The parity failures (all_kinds-free, all_kinds_with_nulls-free, parallel all_kinds_with_nulls) differ in one value: row 11 (i = 0, f = -inf). The old route sends a row with a NaN, an infinity or a subnormal double down the per-row path (bulk_row_ok is False for rows 6, 9, 10 and 11 of that data, checked by script), and the per-row path stores a Python int as a Long. The text route stores the same number as an Integer. insertManyColumns narrowed every whole number to an Integer, so that one row differed. This is not engine-version specific: the per-row path is Python-side. - Fix in the code, not the test: fillRow keeps the whole numbers of such a row a Long (carriedByText: a double[] or Object[] column that holds a non-finite or subnormal double in the row). The stored data of the typed route is now the old route's for every row. The A/B candidate wheel changes (one pass over the columns per row in fillRow). - Two tests asserted the wrong thing. insert_columns still stores a Long: the type declared id INTEGER, which converts a Long to an Integer on any entry point, so it now declares nothing. The homogeneous-load test hard-coded how JPype prints a Java Boolean ('false') and a String ('str'); it now asserts the classes and the lower-cased texts. - New parity cases pin the rule: whole numbers that fit an int beside NaN, infinity, both signs of the smallest subnormal, the largest subnormal, the smallest normal value, -0.0 and a plain float, with and without nulls, on both the sync and the parallel route. - Docs: bulk-loads.adoc and the insert_many docstring state the exception. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898
force-pushed
the
perf/388-jvm-insert-many-typed
branch
from
November 20, 2026 04:26
85332a9 to
38f015b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
On the JVM wheel,
insert_manyturned every chunk of 10,000 rows into one JSON text (json_bulk_dumps_tagged), whichDocumentBatcher.insertManyJsonparsed again into a JSON tree and copied key by key. Now a chunk whose rows are dicts with the same str keys in the same order, and whose columns each holdNoneand values of one exact type (intwithin 64 bits,float,bool,str,date, or adatetimewithout a time zone), crosses as typed columns and no text: one numpy buffer per int, float or bool column, oneStringColumnper str column (whatinsert_columnsalready sends), oneTemporalColumnper date or datetime column (what #383 added), and a boxedObject[]only for a numeric column that holds a null. The new bridge entry points areDocumentBatcher.insertManyColumnsandinsertManyColumnsParallel, which sharefillRowwithinsertColumns. The synchronous and the parallel mode both take them.Every other chunk keeps the JSON text and the per-row fallback exactly as it was. The chunk is judged as a whole, from the types alone, before any bridge array is built: a row that lacks a key, has another one or lists the keys in another order, a column that mixes types (an
intbeside afloat), an int beyond 64 bits, aDecimal,bytes, numpy or pandas value, adatetimewith a time zone, a lone surrogate, a list, tuple or dict (adateinside a list included), and a float vector for anARRAY_OF_FLOATSproperty. A fallback chunk pays for the census and nothing else. The bridge jar and Python change; no native image is involved.One decision in the Java side is not obvious. The JSON text stores a whole number as an
Integerwhen it fits an int and as aLongbeyond (JSONObject.parseNumber), while a numpylong[]column would store aLongfor every number. A schemaless property would then be anIntegeror aLongdepending on which route carried its chunk, and a load with one fallback chunk would mix the two. SoinsertManyColumnsnarrows a whole number the way the text did.insertColumnsis unchanged and keepsLong(a test pins it).Nonestays a null property, never a missing one. A floatNaN, an infinity and a subnormal cross in adouble[]and are stored as the per-row path stores them (the old route sent those rows down the per-row path; the parity tests compare the stored values).What the change allocates per chunk of at most 10,000 rows and per column: one Python list of the column's values (references, not copies); an int, float or bool column also one numpy array and one
long[],double[]orboolean[]copy; a str column one UTF-8 byte buffer, oneint[]of offsets and aboolean[]when there is a null (theinsert_columnslayout); a date or datetime column one numpyint64array, onelong[]and aboolean[]when there is a null; a numeric column with a null oneObject[]of boxed values. The chunk is no longer held as one JSON text in Python, as a JavaStringcopy of it, and as a parsed JSON tree of all its rows. The memory of a load stays bounded by the chunk, as before (#294). I have not measured memory.Related issues
Refs #388 and #315.
Performance changes
Not measured yet. The change removes the text and the parse, and I have not timed it. The speed claim waits for a pre-registered timed A/B of a candidate wheel against 26.11.1.post3.
The figure to measure, so the A/B can be registered before it runs:
Database.insert_many(type_name, rows)on the JVM wheel (arcadedb-jvm), a list of dicts, defaultcommit_every, synchronous mode. The parallel mode as a second figure.insert_manycall; report rows per second and the latency per call.dateanddatetimecolumns (the shape of theinsert_columnsfigure), and the same wide row with a null in some columns of every chunk.Decimalcolumn. Both arms run the old route there, so the ratio should be one. An A/A run of the released wheel as the noise control.SELECTand compared property by property (names in the order the record lists them, the Java class of each value, the value), for each shape.tests/test_insert_many_typed_columns.pydoes this for ints around 2^31, 2^53 and 2^63, floats (NaN, infinities, subnormal,-0.0), strings, dates, datetimes, nulls,Noneagainst a missing key, key order, mixed columns, a date orDecimalinside a list,bytes, numpy scalars, a datetime with a time zone, vertices, an edge type, a duplicate key and a value no route can store.Docs
docs/modules/ROOT/pages/how-to/bulk-loads.adoc(the section on the values the bulk path carries) says which chunks cross as columns on the JVM package and which keep the JSON text. Theinsert_manydocstring says the same.README.mddoes not describe the transport.Checklist
uvx pre-commit run --files <changed files>passes (Bandit as CI runs it is clean too)🤖 Generated with Claude Code