This, or just do a new windows install when you change the components.
Or if youve got the cash, upgrade the system cause the OP is running some old hardware.
PSU is not likely. It wouldn't be turning on if the PSU went bad, or doing other flaky things. Also, if it happened to short the GPU power connector, it would have like totally blown the connector, or failed the POST test/ burnt into the GPU.
Situations like these are unfortunate and why I recommend getting a UPS (Uninterruptable power supply). UPS' will save your machine, as they are like giant batteries in a sense, that, when power goes out, you will have X amount of time that the PC is still running to shut it down without it crashing. Good UPS' like the one I have, has software you can download that can shut the PC and related accessories off automaticallywhen the battery in the UPS has a certain amount of battery left.