Actually, to my knowledge and apart from the early Klunker bikes of the late 70s, Paul Sadoff of Rock Lobster build the first one in the late 80s. Ventana stepped in in 1996 with a first batch of singlespeed frames and already in 1997/8 Surly put their 'Rat pack' out there, what would become the 1x1.
2004 is indeed the height of the singlespeed craze...