I know this won’t help as such, more of a comment rather than help, but I always had terrible trouble using Referenced files with photos. Indeed, it’s a well discussed thing on various forums.
Since moving mine to an entirely iCloud based system, and using a fully (optimised) managed library, it’s been very stable and rarely has bother with third party extensions like Raw Power.
It might depend on your setup, I had even worse trouble using an external HDD as storage too, but again, since putting it onto my internal SSD, and optimising to iCloud, it’s been fine.
FWIW, I think the Photos library can be a bit fragile, all sorts of anomalies can upset things, could just be a file type it doesn’t like, or just one file that may be slightly corrupt, a dodgy USB cable, or hub, or drive enclosure, anything like that.