Me?
Personally I'd leave the brown bits near the seat tube as it almost enhances the look. Think patina.
The rack bosses I would use a scouring pad on, get rid of the loose scaley bits.
You can rub rust zones over with aluminium foil, that tends to get rid of rust.