So are you trying to rationalize the decision you made or torture yourself for not going with a big block? Because it's thoughts like that to lead you down to a big block.
I've done it. For only a little bit more than what I had into my LS swap I could have done the L29 or 8.1 then. My brain soured on the lack of torque, slow uphill rides and generally loosing the "fun to drive" feeling when beating the snot out of the 5.3 added to it. So I go out and buy a big block. And the snowball is rolling down hill and catching speed.
I really think you have an ideal setup for a small block. Stick with it. Don't keep thinking about what might have been. Second guessing stuff is what keeps builds torn down perpetually.