My Xizang is pretty darn light considering it has bombers and I've not actually tried to make it lightweight. On rocky terrain it does seem to get bounced around in a way a heavier bike doesn't seem to, but uphill, on smoother grass, hardpack tracks it is fantastic and is a pleasure to ride. Buuuuuuuut you would get used to it, and having a few heavier bikes to ride it's always noticeable how light the light bike is. After a stint of riding my 19lb road bike, I started to feel the need to make it lighter, wheels tyres tubes etc these thoughts were soon put to bed after riding the mtb a couple of times.
So the truth be told, a light bike is noticeably faster, it's not actually that much faster, it is a lot less hardwork (inside, lack of sweat etc) building a light bike than losing the weight or training to actually be faster.