One thing that seems to be emerging here is that using the VSS input is likely only an option with a factory computer and so far nobody really knows how it interacts.
I have no idea how the factory computer handles that, no experience tuning that.
Though using a VSS is surely an option with an aftermarket ECU.
It sounds like your idle is somewhat normal assuming the cam is what I would expect, which doesn't surprise me, you obviously know what you are doing.
I would think HP Tuners or EFI live would be able to tune the position of the IAC valve with the factory computer, but I don't know that. I know the aftermarket can, and I know the Holley can also change it based on vehicle speed (VSS signal) as well with the advanced table, if that is what you want to do.
It should not be an issue of the IAC valve not being "quick enough" because the computer should open it when you open the throttle. Then when you let off, it's already open, and it just "ramps" back down at a rate you specify. I suspect that when your engine was tuned with the new cam and upgrades with the factory ECU, they changed the idle, but didn't change the throttle open position of the IAC in the software? So it idles higher, but doesn't open the IAC further at load to compensate. It may also need to ramp down slower, to "catch" the engine on the way down. If it ramps down too quick, the engine may stall.
If the ramp rate is too slow, you lose the throttle response at low load, because it will idle down slowly. If its too fast, the engine stalls when coming from higher RPM back down faster.
With the hold position and ramp rates at low load, and increased opening and slower ramp rates at speed, you could get the best of both like you speak of with the Holley. With the factory ECU, someone else would have to answer that, I don't know.