You have to remember when the rivets are crimped at the factory, it's not just the heads that swell and creat the mushroom head on both sides. The shaft of the rivet swells and gets tight in the rivets holes pinning the two pieces together. They can be super tight, and even if you grind off the heads you still can't punch them out with a BFH. TRUST ME, I know, as I have removed probably hundreds of them, and I don't care what anyone says, a high powered air hammer/chissel will not always work.
As I have explained many times as this topic has come up many times, you usually have to stake or spring punch the center of the rivet shank after removing the head, and drill the center out to just past the frame surface on the oppisite side of the frame. The factory holes are usually 7/16 once the rivet is out. I don't grind the heads off, I use a cut off, and cut an "X" in the rivet head just to the frame surface. DON'T grind into the frame as that looks like a hack job IMO.
If you're ok with that, thats fine, you have to look at it not me. Anyways, Once I "X" the rivet head, I will then use a air chissel and chissel off the four small pieces, and they WILL pop off much easier then an entire head will for sure. Then flatted the surface wih a 90deg sander, then stake the shaft on center, then use a 1/8 cobalt drill and drill with high rpm into the shank as straight as you can, for about the thickness of the frame. Then drill with a 1/4 drill to the same depth, it will drill easily as you already have the center drilled out from the 1/8 drill.
Then a 3/8 drill. You can try punching after the 1/4 drill, but if it still won't punch out with a BFH and punch, then you have to drill with the 3/8 and that WILL relieve the swell pressure on the rivet shank and it will punch out easily. When you punch it out use a punch that is small enough to punch at the bottom of the drilled hole, to push it out.
I have found this method to work best for me. I don't bother using cutting oil to drill with, but if you want to that will of course help drill easier. Good luck, hope this helps.