[Python][C++] Basic compute/comparison kernels missing for string_view? #49368

Activity

1. haraldnh

   added

   Type: bug

   on Feb 22, 2026
2. github-actions

   added

   Component: C++

   Component: Python

   on Feb 22, 2026
3. raulcd

   changed the title [-]Basic compute/comparison kernels missing for string_view?[/-] [+][Python][C++] Basic compute/comparison kernels missing for string_view?[/+] on Feb 23, 2026
4. raulcd commented on Feb 23, 2026

   

   raulcd

   on Feb 23, 2026

   Last edited by raulcd

   Member

   More actions

   Hi @haraldnh just to understand a little bit better. Could you share the pyarrow wheel that you are using? As far as I understand this should work but it is true that on Arrow 20.0.0 we split the compute library into its own shared library and this errors feels like the function registry hasn't registered the kernels properly. This should be automatic with PyArrow wheels provided but it is worth investigating whether there is an issue.

   Is this only happening with PyArrow 20.0.0, what system? How are you installing PyArrow, could you share some details about the snippet you are using?
5. raulcd commented on Feb 23, 2026

   

   raulcd

   on Feb 23, 2026

   Member

   More actions

   Sorry now that I take a closer look at the error it does not seem to be related at all. I am still interested on seeing a minimal reproducer to validate whether this is a regression. If we find something that used to work and now it doesn't work I can bisect to the exact cause.
6. haraldnh commented on Feb 23, 2026

   

   haraldnh

   on Feb 23, 2026

   Author

   More actions

   It's also the other way around: It works up until 20.0.0, which is where the fields are 'string'. From 21.0.0 and onwards, things are 'string_view' and I have not been able to make things work.

   I'm reading a deltalake, partitioned on a 'date' (formatted like "2026-02-23") and 'time' (formatted "12:30") fields. I then produce a partition-filter like:

   def _make_single_day_partition_filter(
       floor: datetime, ceil: datetime, partition_scheme: str
   ) -> list[tuple[str, str, str]]:
       pfilter: list[tuple[str, str, str]] = [
           ('date', '=', f'{floor.year}-{floor.month:02d}-{floor.day:02d}')
       ]
       if partition_scheme in ('5min', 'hour'):
           pfilter.append(('time', '>=', f'{floor.hour:02d}-{floor.minute:02d}'))
           pfilter.append(('time', '<', f'{ceil.hour:02d}-{ceil.minute:02d}'))
       return pfilter

   Then feed this through dataset = deltalake.to_pyarrow_dataset(pfilter), and then data = dataset.to_table(columns). From the error, it seems to be the partition-filter comparison that crashes, but it only fails in to_table().
7. raulcd commented on Feb 23, 2026

   

   raulcd

   on Feb 23, 2026

   Member

   More actions

   oh! I understand now! Columns that were read from Parquet as "strings" are now read as "string_view" and there are no kernels for those. Got it. This is probably due to:

   • [C++][Parquet] Reading/Writing string/binary types as/with the corresponding arrow view type in Parquet #43041

   Do you manage how those Parquet files are read or is deltalake managing it?
   There's a new option binary_type which should drive this behavior. Using opts.binary_type == pa.binary() instead of opts.binary_type == pa.binary_view(). This should go back to the previous behavior unless, from what I understand, the files are stored with an Arrow Schema that says those are string view.

   @pitrou is the above correct about the new binary_type option also driving the string behavior (not only binary columns)?

   The real solution would be to provide Kernels though.
8. pitrou commented on Feb 23, 2026

   

   pitrou

   on Feb 23, 2026

   Member

   More actions

   There's a new option binary_type which should drive this behavior. Using opts.binary_type == pa.binary() instead of opts.binary_type == pa.binary_view(). This should go back to the previous behavior unless, from what I understand, the files are stored with an Arrow Schema that says those are string view.

   @pitrou is the above correct about the new binary_type option also driving the string behavior (not only binary columns)?

   This is accurate indeed. But in this case, the Arrow schema was stored (otherwise the columns would not be read as string_view in the first place).

   The real solution would be to provide Kernels though.

   Right.
9. haraldnh commented on Feb 23, 2026

   

   haraldnh

   on Feb 23, 2026

   Author

   More actions

   I can confirm that setting opts.binary_type has no effect. I don't quite know how to identify how the schema is provided, but it's the same data files that work with 20.0.0, but fail on later versions.

   I guess I'll have to stay on that version, and just wait for the kernels. Tried to have a look at the code, but I suspect I would make more of a mess than a help trying to write it. It's been a while since I did much C++ templates. 😺
10. pitrou

    closed this as a duplicateof [C++] Implement basic functionality for new Binary/Utf8View types #39634 on Feb 24, 2026
11. pitrou commented on Feb 24, 2026

    

    pitrou

    on Feb 24, 2026

    Member

    More actions

    Some issues had actually already been opened for this, so I'm closing this one: see #46856, #39634