A small shift in RF-DETR mask in instance segmentation, especially on CoreML

[msluszniak]

Description

We spotted that in CoreML delegate of RF-DETR there is a small shift to the right of both masks and bounding boxes. The reason is probably in some numerical defects in ET export/runtime. On XNNPack variant there is non / way smaller shift, but can have the same origin. Photos for reference:

CoreML:

XNNPack:

Steps to reproduce

Run CV demo app and run instance segmentation with RF-DETR model.

Snack or a link to a repository

No response

React Native Executorch version

rne-rewrite branch

React Native version

0.83

Platforms

iOS

JavaScript runtime

None

Workflow

None

Architecture

None

Build type

None

Device

None

Device model

No response

AI model

No response

Performance logs

No response

Acknowledgements

Yes

[msluszniak]

Update: only f16 variant has shift, f32 works sound, probably some ANE quirks then.

[msluszniak]

Indeed compute unit ALL for CoreML does not work and gives this shift. CPU & GPU doesn't have this problem so will ship it. What is interesting, I spotted completely different, incorrect behaviour on CPU only, which completely destroys b-box and mask.

[msluszniak]

This problem is genuinely resolved here with uploaded to HF a GPU & CPU variant, now I'll investigate and escalate this issue in correct place for proper fix.