I probably spend at least as much time building and tinkering as I do riding, but you know what, no guilt at all.
Fiddling with old bikes is probably the most relaxing thing I do, and sometimes it's even useful. It also stops me buying any brand new bikes, which is definitely a good thing.
Riding is great too. I wish I had the time, headspace and fitness to go off for epic day long rides, or week long tours, but currently I don't, and family comes first anyway. So it's enough to have a pootle along the towpath with jr. stupido, or a quick blast around the forest, or half an hour up and down one of my local hills. A little bit of joy is better than nothing.