You need to keep whacking it in between the lug studs on the flat part. Hit one a couple times, spin the drum and hit another one. Keep working it till the drum center starts to slide on the axle hub.
Once the rust frees it's grip the only thing left to hold it will be the brake shoes. If you can't get in there to adjust them down, you are just going to have to fight with the drum till it comes off. Chances are you will destroy the shoes and maybe the springs in the process. I've found that pulling on the drum with one hand and hitting it with a hammer with the other will eventually get it off. Pull and hit on one side then switch to the other. It's a long and tedious process, have patience.
Then plan on buying shoes and springs and having the drums turned (machined) to get rid of the rust ridge or just outright replaced.