I have never had any issues with destroying bearings this way. This is the same thing that I do at work, when some dumbass sends me a drum or rotor, yet is stupid enough to not knock the inner bearing and seal out before they do so.
Mostly I do it because if I don't, guaranteed the same dumbass will not clean the metal shavings from the machining operation out of the hub, and it'll be MY fault their bearing is full of crap, because they were too stupid to knock the bearings out and clean them up.
No arguing necessary. I guess you guys are thinking I am hitting it harder than I really am. A ball peen hammer hit on top with an open hand is not going to destroy a bearing. Usually that is all it takes to knock the seal out.