Well, if it works from anywhere, its not the motor. I think you mistyped, but from what I read, it only works from the driver's side.
Sounds like either a bad switch, or a wiring problem. Not sure what type and model truck you have, but on most of them the driver side switch will override the passenger side switch.
So, it could be a bad driver side switch even though that side works.
If its an older vehicle, many of them need a relay mod done. A combination of small factory wire, older switches and old cruddy tracks makes the windows really slow. Lots of folks here add a couple of relays powered directly off the battery with heavier wire to get full voltage to the motor.
Let us know what you got.