I’ve done a little experiment today to see if using APFS would improve the situation. I formatted another SSD as APFS and copied over the folder with the 700 raw images and their XMP sidecars. It felt even less responsive than the exFAT disk, indexing took longer and the move to trash behaviour was exactly the same. So on the plus side, it saves me from reformatting my main SSD as APFS, on the downside I still don’t have a solution to reliably delete rejected photos.