Unfortunately, Leonie is right, the ‘referenced’ system for Photos is unreliable (although IME it goes back further than macOS 11). I’ve tried it myself in the past, and usually found it to lose connection to referenced images at some point – even in normal use it does this with random images. Also, there is no way to bulk reconnect either, you have to click on each image individually to do it.
If you check the Apple Discussion forums for Photos, it’s been a common thing to not recommend using the referenced library system. I have a fully managed library, along with iCloud storage, and it’s been very reliable.
If you still have access to the old HDD, and as you have everything on the one drive, it might be possible to use the ‘consolidate’ option and get all the images into it as a managed library, then transfer it over to the new HDD.
Generally, it’s a very bad idea to try manipulating any of the files outside of Photos, that’s likely to screw things up even more than ever (I’ve also found that out the hard way). Which is another reason the referenced system is not recommended, you can clearly use images for other purposes if they’re outside the library, but change anything outside of Photos (move a file, rename it, or edit it), and that is a recipe for disaster.
My own setup includes a backup of all my images on a separate HDD, using a standard folder structure, completely separate to Photos, and never used as a live working source. That can be used to access image files for use in other apps if needed. However, if I’m only working with a few images, I tend to just export them as ‘originals’ from within Photos, which just make a fresh untouched copy to work with. Most of the time I try to use software that work with Photos as an extension/external app, and integrates with the library that way.
Just a final thought, you did remember to check it was set as the ‘System Library’ in the Photos preferences? (might only matter if you happen to have more than one library showing in Photos).