Discussion: DLPack requires 256B alignment of data pointers

[leofang]

Recently we found that DLPack has this requirement noted in the header:
https://github.com/dmlc/dlpack/blob/a02bce20e2dfa0044b9b2ef438e8eaa7b0f95e96/include/dlpack/dlpack.h#L139-L141. Would this be an issue for all adopting libraries? As far as I know, CuPy doesn't do any alignment check and take the pointer (DLTensor.data) as is, and IIRC quite a few other libraries are also doing this.

cc: @seberg @rgommers @tqchen @kmaehashi @oleksandr-pavlyk @jakirkham @edloper @szha

[rgommers]

I have a vague memory of this alignment requirement being pointed out before and the answer being that it's unnecessary and can be ignored. Some digging turned up numpy/numpy#19013 (comment), where you (@leofang) said: "please ignore the alignment stuff. If we are talking about host-device copies, it's an implementation detail (of CUDA, HIP, or any sane memory pool implementation sitting on top). CUDA/HIP's copy APIs do not care about alignments. DLPack does not. CUDA Array Interface does not. No one does."

[leofang]

Thanks, Ralf, yes now I recall, I guess it's a rediscovery then 😅 Maybe I just need to add a note to this page for better visibility?
https://data-apis.org/array-api/latest/design_topics/data_interchange.html

[tqchen]

This is certainly something that worth discussion.

To give background of the original rationale in DLPack.

Such requirement is not necessary for host/device copies of any kind as we rightfully point out that copy APIs does not care about alignments. The alignment spec comes in handy when we build kernels that leverages those data structure, as vectorization usually have higher amount of alignment requirements.

Of course there is a need to support sliced array, which do not have the proper alignment. Similar problems will happen in the context of OpenCL, where the data ptr is opaque(you cannot add an offset directly to that field).

The byte_offset field was meant to enable such offset in the context of OpenCL, and other cases where offsetting can cause not aligned array.

[rgommers]

The alignment spec comes in handy when we build kernels that leverages those data structure, as vectorization usually have higher amount of alignment requirements.

Naive question - is there any advantage of enforcing 256 byte alignment plus always checking byte_offset, over not requiring alignment at all, or only alignment on a more "normal" boundary? 256 byte is pretty unusual I think - the linked issue for example explains that Arrow requires 8 byte and recommends 64 byte alignment. I can't remember seeing something >64 byte before elsewhere.

And if there is such an advantage, I guess you're proposing to start checking it in each implementation? This may require a slow path, with emitting warnings for at least a year before starting to raise an exception.

[tqchen]

Larger alignment generally have an advantage of making vectorization simple(in the presence of avx2 or 512) in the case of CPU. Similarly in the case of GPU can inform a more aligned loads in float4. The particular number was picked mainly because it was CUDA's Malloc default alignment.

I agree start with a warning would be useful, given most do not start with such an assumption.

[seberg]

Of course there is a need to support sliced array, which do not have the proper alignment.

Yes, so why even discuss enforcing any alignment beyond the itemsize? Slicing is a central feature, so almost all libraries have to anticipate data that is at most itemsize-aligned. I don't understand how you can possibly do any more than recommend allocating data with a larger alignment (e.g. because certain compute libs may not accept the data otherwise – which is fair).

Calculating alignment, and rejecting data that is not hard! You could also ask the exporter to provide the information (e.g. as a power-of-two alignment – many exporters likely already store it in some form; I would like to for NumPy) or you expect the importer to calculate it.
To be nice: provide a helper for it (which also helps to correctly handle length <= 1 dimensions as their stride does not matter for alignment/contiguity).

[tqchen]

Thanks @seberg. In the specific case of DLPack, there is also a need of supporting opaque data ptr (in the case of vulkan and opencl), in those cases the byte_offset was used to support slicing(while the data itself remained aligned as the original allocation) in a native way.

Just trying to bring up context here and background for both cases, not necessarily arguing one versus another

[seberg]

The use of byte_offset for opaque data pointers is interesting. Do you suggest to also use it for non-opaque data pointers to enforce/allow a bigger allocation alignment guarantee then data alignment? Is that useful, scanning that initial link @leofang posted sounded like it might be?

How is the current header comment/standard meant? Are we supposed to use byte_offset for non-opaque CPU data pointers or not?

[tqchen]

Based on the plaintext interpretation of the header comment, if the data is not aligned, a usage of byte_offset is certainly encouraged to unify that with opaque ptr situation. Of course on CPU there is not really great advantages as checking the ptr itself is equally simple.

In reality, i think a lot of the current exchange impls ignores that constraint, so it becomes an open question whether or not we encourage such a convention, or allow people to pass in their own alignment preference.

[kkraus14]

The use of byte_offset for opaque data pointers is interesting. Do you suggest to also use it for non-opaque data pointers to enforce/allow a bigger allocation alignment guarantee then data alignment? Is that useful, scanning that initial link @leofang posted sounded like it might be?

cc @shwina @jrhemstad

Yes this would be really helpful. Knowing the base allocation pointer, offset, and alignment often allows for much more efficient code.

[seberg]

@kkraus14 OK, that makes sense to me. I am still not sure if you would want to expect the importer to calculate both base and non-base pointer alignment? Or should DLPack grow a field to indicate alignment (e.g. for opaque pointers). If it is padding and alignment, would that mean that the exporter must indicate it anyway (i.e. for padding)?

I suppose alignment could be device specific, maybe some devices malloc provides those guarantees always?

Would you suggest changing the dlpack header to explicitly say:

• All exported data should be at least aligned to (device alignment for the datatype OR itemsize) (LATE EDIT: itemsize alignment is good, but could be slightly "tricky" e.g. for complex longdoubles on CPU)
• If the data is a view, byte_offset should be used, so that the data pointer represents the full allocation alignment
• Adding an indicator for the alignment (padding?) of the data pointer could be useful?

[kkraus14]

I am still not sure if you would want to expect the importer to calculate both base and non-base pointer alignment?

I would assume that allocations are aligned, which means the base pointer is aligned, but the non-base pointer is not necessarily aligned and is determined by the offset.

Or should DLPack grow a field to indicate alignment (e.g. for opaque pointers). If it is padding and alignment, would that mean that the exporter must indicate it anyway (i.e. for padding)?

I would love to specify both padding and alignment, but beggars can't be choosers 😄.

I suppose alignment could be device specific, maybe some devices malloc provides those guarantees always?

Memory pools are super common that don't necessarily have to follow the alignment / padding guarantees of the base malloc implementation.

Would you suggest changing the dlpack header to explicitly say:

• All exported data should be at least aligned to (device alignment for the datatype OR itemsize)

I don't know if we want to go to the level of device alignment given the number of memory pools out there. Requiring the alignment to be a multiple of the itemsize makes sense to me though.

• If the data is a view, byte_offset should be used, so that the data pointer represents the full allocation alignment

Yes

• Adding an indicator for the alignment (padding?) of the data pointer could be useful?

Yes, adding both would be ideal and would allow more optimized code

[seberg]

Ah, so padding means actually means "you are allowed to write over that data?" and not just "you can safely read more?". The only thing that makes me a bit unsure about itemsize alignment that I realized is that NumPy would currently have some trouble guaranteeing it for complex longdouble.