I would have thought that the difference in the area of material (which is about as much as a human hand) somewhere low down like a wheel could not be perceived through the steering pulling to one side?!
The area difference, compared to the total area of bike + rider is absolutely tiny, particularly when considering the differences that come from the bagyness of a rider's clothing?
If there is any difference that can be felt I would have come from wind strength, wide direction or clothing choices, far, far bigger variables, even if the wind feels the same to the rider?