I had 35x12.50's and went from the rear 10-bolt to a 14FF with just the minor shave and trim (grinded off the lip), and to be honest didn't notice any difference in regards to capability because of ground clearance. I did notice an overall performance gain because I could drive it harder without worrying about blowing up the rear axle.
Next step was D60 front and 13/38-16 TSL's which are pretty short for being called a "38". I would say they were closer to the typical 37, or even a tall 36. Ran those tires for years before switching to 39.5's (which are a good 3" taller). Again, can't really say I noticed any difference in ground clearance, but overall the vehicle is more capable.
Overall I would say the ability to drive the vehicle harder without breakage, and the simple idea of greater dependability, greatly outweighs the minor differences in ground clearance.