You want the biggest hammer you can swing in the space. It should not say "oz" on the side, only "lbs". Even 3lb is too small. I have an 8lb sledge with the handle cut short and that's pretty good persuasion - if there's enough room to swing it.
If you're grinding a bracket or other piece that will discarded, use the grinder to make the rivet slightly concave before you start with the drilling/pounding. The shorter the shank of the rivet that the remains, the better your odds. Likewise, if it's a sacrificial braket, you can drill the rivet completely from it, stopping just short of touching the frame.