Whatever it was, it is not used now. There are some very old recorded languages, such as Tamil Sanskrit, and the oldest forms of Persian, not to mention Chinese, but these are still young if you consider how long humans may have been able to speak.
We know that Neanderthals had hyoid bones and now we are beginning to understand how they sounded. Look at this article:
Neanderthals Speak Again After 30,000 Years
ScienceDaily (Apr. 21, 2008) — Dr. Robert McCarthy, an assistant professor of anthropology in the Dorothy F. Schmidt College of Arts and Letters at Florida Atlantic University, has reconstructed vocal tracts that simulate the sound of the Neanderthal voice.
Using 50,000-year-old fossils from France and a computer synthesizer, McCarthy’s team has generated a recording of how a Neanderthal would pronounce the letter “e.” The brief recording doesn’t sound like any letter in modern languages, but McCarthy says that’s because Neanderthals lacked the “quantal vowels” modern humans use. Quantal vowels provide cues that help speakers with different size vocal tracts understand one another.
“They would have spoken a bit differently,” McCarthy said at the annual meeting of the American Association of Physical Anthropologists in Columbus, Ohio in April. “They wouldn't have been able to produce these quantal vowels that form the basis of spoken language.”
Though quantal vowels make subtle differences in speech, their absence would have limited Neanderthal speech. For example, Neanderthals would not be able to distinguish between the words ‘beat’ and ‘bit.’
For scientists, McCarthy’s work represents an exploration of life 30,000 years ago when Neanderthal humans, our closest extinct ancestor, lived in parts of Europe, Central Asia and the Middle East. The species died out mysteriously some 28,000 years ago.
McCarthy has plans to eventually simulate an entire Neanderthal sentence.
McCarthy's simulation of a Neanderthal voice is available here. (see below)