GH-50326: [Python] Convert arrays to Python objects without per-element Scalars in to_pylist

[viirya]

Rationale for this change

pa.Array.to_pylist() converts one element at a time through Array::GetScalar plus a Python Scalar wrapper; for list types each row additionally allocates a Python Array wrapper for the row's values slice and a fresh generator before recursing per element. A sample profile shows ~20% of runtime in CPython GC (triggered by the per-row GC-tracked allocations), ~25% in GetScalar, and only ~7% doing the useful work of creating the output objects — making to_pylist several times slower than converting via to_pandas() and back, and ~24x slower than ndarray.tolist() for plain int64. Details in #50326; this hit Apache Spark's Arrow-serialized Python UDFs (apache/spark#56940, apache/spark#56943).

What changes are included in this PR?

Following review feedback, this adds a general scalar-free conversion mechanism instead of per-type to_pylist overrides:

• Array gains cdef object _getitem_py(self, int64_t i), returning self[i] as a Python object. The base implementation is GetScalar + Scalar.as_py, so any type without a specialization behaves exactly as today (dates, times, timestamps, durations, decimals, dictionary, extension, unions, views, ...).
• The baseline Array.to_pylist becomes a single loop over _getitem_py. maps_as_pydicts != None keeps the Scalar-based path, since map→dict conversion has per-entry duplicate-key semantics.
• Specializations avoid all per-element Scalar and per-row Array-wrapper allocation:
  • integers and floats (a type_id switch on NumericArray; date/time/timestamp subclasses fall through to the exact base),
  • boolean,
  • string/binary and large variants (GetValue + PyUnicode_DecodeUTF8 / PyBytes_FromStringAndSize, matching str(buf, 'utf8') / to_pybytes() exactly),
  • list/large_list/fixed_size_list (each row's list is built from the child's _getitem_py over the offset range; the wrapped child is cached on the parent array),
  • map (association list of key/value tuples, matching MapScalar.as_py),
  • struct (one dict per row; duplicate field names fall back to the Scalar path so they raise ValueError like StructScalar.as_py).

Nested types compose without any per-row wrappers. ChunkedArray.to_pylist, Table.to_pylist and ListScalar.as_py delegate here and speed up automatically. Follow-up candidates: string/binary views, run-end-encoded, dictionary, a fast path for date32.

Benchmarks (macOS arm64, M4 Max):

benchmark	before	after	speedup
flat int64 with nulls (4M)	0.39 s	0.028 s	14x (~7 ns/element, on par with ndarray.tolist)
flat string (4M)	0.83 s	0.06 s	14x
list<string> (2M rows)	1.93 s	0.46 s	4.2x
list<list<int32>> (1M rows)	2.10 s	0.40 s	5.2x
struct<int64,string> (1M rows)	0.91 s	0.07 s	13x
map<string,int64> (1M rows)	2.77 s	0.74 s	3.8x

Are these changes tested?

test_to_pylist_bulk_paths (added here) compares against the per-scalar conversion with exact element types for representative arrays including sliced views. Additionally verified with a randomized differential test against [x.as_py() for x in arr] with exact-type equality: all integer widths (incl. values beyond 2^62), floats (NaN/inf), boolean, string/binary (+large, multibyte), all list kinds, nested lists, struct (incl. empty struct, duplicate-field-name ValueError), map (incl. strict-mode duplicate-key KeyError), dictionary/null fallbacks, sliced/chunked arrays, and both maps_as_pydicts modes — no differences. pytest test_array.py test_scalars.py test_convert_builtin.py test_table.py test_types.py: 1295 passed.

Are there any user-facing changes?

No behavior changes, only performance.

• GitHub Issue: [Python] to_pylist() on list-typed arrays is several times slower than converting via to_pandas() #50326

This pull request and its description were written by Isaac.

[github-actions]

⚠️ GitHub issue #50326 has been automatically assigned in GitHub to PR creator.

[Copilot]

Pull request overview

This PR improves pyarrow.Array.to_pylist() performance for list-like arrays and (large) string arrays by adding specialized bulk conversion implementations that avoid per-element Scalar allocation and wrapper overhead, while keeping output semantics unchanged.

Changes:

• Add bulk to_pylist() implementations for ListArray, LargeListArray, and FixedSizeListArray that convert child values once and slice per row using offsets.
• Add fast to_pylist() implementations for StringArray and LargeStringArray that decode directly from the value buffer.
• Add a new test validating bulk-path results against the scalar-based reference across nested, sliced, empty, and all-null inputs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
python/pyarrow/array.pxi	Adds type-specific to_pylist() fast paths for list-like and string arrays.
python/pyarrow/tests/test_array.py	Adds a regression/differential test to ensure bulk paths match scalar-based conversion.

[viirya]

null_count() is a one-time vectorized popcount over the validity bitmap (~n/8 bytes, well under a millisecond for 2M rows), computed and cached per ArrayData. In exchange, the no-null branch skips the per-element IsNull() check entirely. Branching on null_bitmap_data() == NULL instead would save that single scan but degrade the common case of a sliced/combined array that has a bitmap yet contains no nulls in range — that would take the per-element IsNull() path forever. So the current form should be the better trade-off in practice.

[gaogaotiantian]

I'm not an expert in Cython but curious about how result.append() works. I think as long as result is not defined as a Cython list, result.append still works as a pure Python object? In that case, each append would trigger a dynamic attribute search and the list would be reallocated a few times during appending.

It would be an interesting experiment to do a full-allocation for the list before assigning the data, as we already know the length of the list. An extra step forward is to declare the return value as a list in Cython so it can optimize setitem even more.

Something like

cdef list result = [None] * n
cdef Py_ssize_t i

for i in range(n):
    result[i] = ...

return result

For a long list this might push the performance even further.

[viirya]

Good idea — I tried exactly that (cdef list result = [None] * n + indexed assignment, also letting null rows keep the prefilled None), but it benchmarks the same or slightly worse: list<string> 0.34 s → 0.35–0.37 s, list<list<int32>> 0.65 s → 0.69–0.70 s (best of 3, consistent across reruns).

Two reasons, I think: Cython already lowers result.append(x) on a local it can see is a list to a PyList_Append call, and CPython lists over-allocate geometrically, so the resize cost is amortized and tiny. Meanwhile the preallocated version pays an extra refcount round-trip per slot (every result[i] = x has to decref the prefilled None). The loops are dominated by creating the per-row slice objects / decoding UTF-8 either way, so I kept the simpler append form.

[gaogaotiantian]

Okay if the benchmark is similar this is good. Would defining result as a list help? Like a cdef list for it and do append. Just curious whether cython knows it's a list already - maybe it does and that's why it's fast.

[viirya]

Good question — Cython already knows. Its type inference marks result = [] as list, so the untyped version and an explicit cdef list result = [] compile to the identical generated C: both lower result.append(x) to Cython's inlined __Pyx_PyList_Append(), which appends in place via PyList_SET_ITEM while there's spare capacity and only falls back to PyList_Append (geometric resize) otherwise. I verified by cythonizing both variants side by side and diffing the generated C — the loop bodies are byte-identical. That's indeed why the append form is already fast, and why adding cdef list doesn't move the numbers.

[pitrou]

The cause is the per-element conversion loop ([x.as_py() for x in self]): every row allocates a C++ Scalar (Array::GetScalar), a Python Scalar wrapper and, for list types, a Python Array wrapper for the row's values slice plus a fresh generator before recursing per element.

I agree this is extremely wasteful, but the approach here seems very ad hoc and also solves the performance issue for a very limited set of types, while the performance problem is more general:

In [5]: a = np.arange(10_0000)

In [6]: %timeit a.tolist()
722 μs ± 43.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [7]: b = pa.array(a)

In [8]: %timeit b.to_pylist()
17.5 ms ± 147 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [9]: b.to_pylist() == a.tolist()
Out[9]: True

How about we do something like the following:

1. Add an Array method that indexes and returns a Python value (e.g. an int) directly instead of a PyArrow scalar (e.g. an Int64Scalar)
2. Use that new method in the baseline implementation of Array::to_pylist.

[viirya]

Thanks for the review, @pitrou! Agreed — the per-type to_pylist overrides don't generalize. I'll rework the PR along your suggestion, which I read as a scalar-free getitem:

1. Add cdef object _getitem_py(self, int64_t i, ...) on Array, whose base implementation preserves today's semantics exactly (GetScalar(i) + as_py()), so any type without a specialization behaves identically to today.
2. Make the baseline Array.to_pylist a single loop over _getitem_py.
3. Specialize _getitem_py for the common types: integers, floats, boolean, string/binary (+ large variants), list-likes (build each row's list directly from the child's _getitem_py over the offset range), struct and map — nested types then compose without allocating any Scalar or Array wrapper per row, and maps_as_pydicts threads through as an argument.

Types with non-trivial as_py semantics (dates/timestamps/decimals/dictionary/...) would keep the exact fallback initially and can be specialized incrementally in follow-ups; your int64 example above lands in the specialized bucket and should get within striking distance of ndarray.tolist.

One question before I rework: do you prefer this mechanism in Cython as sketched, or as a C++ ToPyList visitor under python/pyarrow/src/arrow/python/ (in the spirit of MonthDayNanoIntervalArrayToPyList)? The Cython version keeps the unspecialized fallback trivially exact and the diff reviewable; the C++ visitor is likely faster still (and would also serve ChunkedArray/Table uniformly) but is a much bigger surface. Happy to go either way.

[pitrou]

One question before I rework: do you prefer this mechanism in Cython as sketched, or as a C++ ToPyList visitor under python/pyarrow/src/arrow/python/ (in the spirit of MonthDayNanoIntervalArrayToPyList)? The Cython version keeps the unspecialized fallback trivially exact and the diff reviewable; the C++ visitor is likely faster still (and would also serve ChunkedArray/Table uniformly) but is a much bigger surface. Happy to go either way.

Doing this in Cython would probably be more maintainable, right? Wrangling CPython API calls from C++ is generally tedious and error-prone.

[viirya]

Reworked as suggested — the PR now adds a scalar-free cdef _getitem_py(self, int64_t i) on Array and the baseline to_pylist is a single loop over it. The base implementation is GetScalar + Scalar.as_py, so unspecialized types (dates/times/timestamps/decimals/dictionary/extension/...) behave exactly as before, and specializations exist for integers, floats, boolean, string/binary (+ large variants), list/large_list/fixed_size_list, map and struct. Nested types compose: a list row is built directly from the child's _getitem_py over its offset range, with the child wrapper cached on the parent array. maps_as_pydicts != None keeps the Scalar path since map→dict conversion has per-entry duplicate-key semantics.

Your int64 example: 4M int64 with nulls goes from ~0.39 s to 0.028 s (~7 ns/element — on par with ndarray.tolist). Other numbers (M4 Max): flat string 4M 0.83→0.06 s; list<string> 2M 1.93→0.46 s; list<list<int32>> 1M 2.10→0.40 s; struct<int64,string> 1M 0.91→0.07 s; map<string,int64> 1M 2.77→0.74 s.

Net diff is smaller than the previous approach (−104 lines) since the per-type to_pylist overrides are gone. Differential tests against the Scalar path (exact type equality, incl. slices, all-null, duplicate struct field names raising ValueError, strict-mode maps) pass, as does the pytest suite (1295 passed). PR description updated accordingly.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[viirya]

Good point — moved the declaration after the pre-existing attributes in 728584f, so type/_name offsets stay stable and the new field follows the append-only convention that Cython's size check assumes.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[viirya]

PyArrow only supports 64-bit platforms, where Py_ssize_t and int64_t have the same width; and on a hypothetical 32-bit build a single value longer than PY_SSIZE_T_MAX could not exist in the first place, since the child data buffer itself is bounded by the process address space. So the implicit conversion cannot truncate in practice.

[viirya]

Added binary/large_binary (including embedded NUL bytes), list, wide-range integers, floats, boolean and struct coverage in 728584f, plus an assertion that duplicate struct field names still raise ValueError.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[viirya]

Note on the two red CI jobs — both are unrelated to this change:

• AMD64 macOS 15-intel Python 3: the only failure is test_s3fs_limited_permissions_create_bucket, an S3/minio fixture issue (the mc admin subprocess dies with SIGPIPE before the filesystem is exercised). The same job passes 7616 tests, including all array/scalar/conversion tests. The same flake also hit the previous commit's run.
• AMD64 Conda Python 3.11 Sphinx & Numpydoc: two doctest failures in code this PR doesn't touch — Table.join_asof (the example's output chunking varies with task scheduling: [[1,3],[2,3,3]] vs [[1,3,2,3,3]]) and default_memory_pool (environment-dependent). The same job passed on the first commit of this PR with the same doctests.

[dbtsai]

Follow-up idea: reuse decoded strings for dictionary-encoded arrays

The new StringArray._getitem_py calls PyUnicode_DecodeUTF8 per element, so a repeated string value produces a fresh str object every time (N allocations + N UTF-8 decodes for N elements). Two places where value reuse could help:

1. DictionaryArray (highest value, no correctness risk): the dictionary already stores each distinct value once. A specialized path could decode the D dictionary entries once and then return interned references (decoded[index[i]] + Py_INCREF) — turning N decodes into D decodes + N pointer reuses. This PR currently doesn't specialize DictionaryArray, so it falls back to the Scalar path and this win is unclaimed. Since str is immutable, sharing references is semantically identical to copying.

2. Plain StringArray with repeated values: an explicit dedup cache (or PyUnicode_InternInPlace) would cut allocations on low-cardinality data, but adds a hash/lookup per element and regresses high-cardinality (mostly-unique) data. Probably only worth it behind a cardinality heuristic / opt-in, not by default.

Not blocking for this PR — just flagging DictionaryArray as a natural, safe follow-up where value reuse is a large win.

[dbtsai]

Follow-up ideas: further performance opportunities

The structural win here (no per-element Scalar/Array allocation + child-array caching) is solid. A few places that still leave performance on the table, roughly by impact:

1. Hoist the numeric type-ID dispatch out of the per-element loop (biggest). NumericArray._getitem_py recomputes type_id() and walks an if/elif ladder (up to 10 branches) on every element, even though the type is invariant across the array. For an int8 column that's ~6 failed comparisons per value. Resolving the type once and running a monomorphic loop over the buffer turns the "no Scalar" win into a "vectorized" win.

2. Share the fast path with as_py() / __iter__. _getitem_py is private and only called from to_pylist (when maps_as_pydicts is None). arr[i].as_py(), iteration, and ChunkedArray element access still allocate Scalars. Wiring those to reuse _getitem_py would extend the win to the other common access patterns.

3. Null fast path. Every accessor calls IsNull(i) per element even when null_count == 0. Checking null_count == 0 once and skipping the per-element null test helps the very common all-non-null column.

4. Offset reads in list/map paths. value_offset(i) / value_offset(i+1) are two virtual calls per row; reading the offsets buffer pointer once and indexing avoids that on long arrays. Also worth confirming for j in range(start, end) compiles to a C loop rather than allocating a range per row.

5. Missing specializations that still fall back to Scalars: temporal (date/timestamp/time/duration), decimal, DictionaryArray, and string_view/binary_view have no _getitem_py. Timestamps/dates and dictionaries are common enough to be worth a fast path (dictionary especially — see the separate value-reuse comment).

None of these are blocking — the PR is a clear improvement as-is. #1 and #2 are the two with real leverage.