when you're alone, being aero is important. When you're in a group, 'staying aero" is.
But you're right, a bike is as fast as the rider is. I used to ride a sub 7 kilo CAAD10 but changed it to a steel Master Extra Light with the same components (one kilogram more) and still went as fast (or slow ;-)) up mountains.