It’s better to flow a fuse or fuses than to blow up expensive semiconductor parts. I recommend at least two fuses. One fuse should be dedicated to 5V (or 3.3V) parts. One other fuse can be used to protect the overall circuit. If you have the board space, I also recommend dedicated fuses for each power device. It is easy to design in a fuse as it only needs to be in series with the circuit to be protected.
Instead of using bulky fuse holders, you can use surface mount fuses. They are more of a pain to replace, but you buy more board space this way. To desolder a fuse, use Chipquik or a similar low-temperature solder to help you.