'85 I believe the window power wire is plugged into the top of the fuse panel. If the fuse panel itself was bad, you'd have more problems than just windows.
The accessory terminals along the top of the panel are powered from a common (and solid) "rail", so you can try moving the plug (once you figure out which it is) into another terminal, see if the problem follows. It could be a bad connector, or the whole strip could be bad.
The rails are protected via the two silver mini-relays on the fuse panel that look like metal blade fuses. You can swap those with each other as well, and see if the problem migrates, but you could have a bad connection there as well. Never hurts to pull that stuff out and make sure the contacts aren't corroded and are tight.