That's good you don't need surgery.
As to the physio, get onto them to get it started as soon as possible.
I know you've got a broken bone which needs to knit but when I got my elbow dislocated playing rugby, it was a month before I got an appointment. According to physio that was too late, as it's basically healed so can't then really be altered as to range of movement. I took my cast off on the Wed after it happened on the Sat as it was too tight and my hand was swelling up, it was supposed to be on for 2 weeks. By the next week I could climb, albeit carefully, at Alien Rock in Edinburgh. That's when the physio should have been started.