So much depends on the steering geometry and frame design. My Marin-based tourer is happy even at 40mph with front lowriders, rear panniers and a barbag. However, lighten the loads and put a trailer on and it spends all its time trying to kill me with lowriders. Take them off and it's fine again with the trailer.My town bike started out with rear rack and luggage on top. Did the job but felt heavy and swingy at the back (if that description makes sense). I've since put on a front rack with a basket and the option to mount front panniers and it feels much better. Ok, I wouldn't no-hand it, but on climbs, my front body weight with the front luggage weight seems to gel better and the back end feels lighter and riding especially on climbs feels more fluid
It's definitely a case of 'suck it and see'.