Yeah, balls down is right.
Now got the headset properly adjusted, or as good as I can get it, and the wobbling is gone.
Low speed instability is a lot less and I can let go of the bars, although it's not perfect, as the bike is pulling a bit to the left.
I'm thinking I should probably replace...