If I do a short hop (like a mile down to the town or whatever) I don't tend to wear a helment although I know that I should, if I'm honest. Same if I'm just messing around in the garden on the bike trying out new stuff.
If I'm off for any decent length ride though I always wear one. Had way too many offs not too. Was saved a fairly massive head injury many years ago when I hit a damp drain lid at 40mph (this was on the MaxTrax on a road section coming back from my local hill), went straight over the bars and slid down the road on my back for rather too long. I reckon that the padding I had to wear on my shoulder blades and elbows for those weeks and weeks would have been a lot more uncomfortable had they been touching that nice bare bit of skull that would have been left after scraping my head down the tarmac (I remember the state of the Bell afterwards).
And no, wearing a tiny helmet isn't going to save me from a bus but then again not a lot is - whereas at least if I do wear a helment I can save myself from all those stupid sort of accidents that can catch you out (including all the daft stuff that can catch you out off road - tank slappers, losing one end or the other on something rough, misjuding an obstacle, jump, etc).
Personally I would alwyas recommend wearing a helment (on the occasions I don't I always ride a lot more cautiously) - I think that we've all binned it enough times (be it off road, as mentioned above, or ended up in a hedge thanks to a careless driver) not too.
Btw, the 'it wouldn't save you from a bus' argument is a bit like not wearing a seat belt because it wouldn't save you from being crushed by a lorry.