The Turing test is a proposal for a test of a machine‘s ability to demonstrate intelligence.
A human judge engages in a natural language conversation with one human and one machine, each of which try to appear human; if the judge cannot reliably tell which is which, then the machine is said to pass the test. In order to test the machine’s intelligence rather than its ability to render words into audio, the conversation is limited to a text-only channel such as a computer keyboard and screen (Turing originally suggested a teletype machine, one of the few text-only communication systems available in 1950).
To see how a journalist fared in discriminating between man and machine in a recent Turing test in October, read below. You may be surprised and a bit frightened……
Eugene Goostman is a 13-year-old boy from Odessa, Ukraine, the son of a talk-show host and a gynaecologist, who keeps a guinea pig called Bill in his bedroom and likes the science fiction novels of Sergei Lukyanenko and Kurt Vonnegut.
He is also a work of fiction, a software program written by a bio-scientist from St Petersburg and a finalist in a contest to find the world’s first thinking computer, staged yesterday at Reading University.
His task was to convince judges, in five minutes of conversation, that he was a human being who really had read Slaughterhouse Five and could plausibly shoot the breeze about it and any other topic under the sun.
I was one of those judges, and yesterday, I was fooled. I mistook Eugene for a real human being. In fact, and perhaps this is worse, he was so convincing that I assumed that the human being with whom I was simultaneously conversing was a computer.
This was the 18th Loebner Prize, a talent contest for programs like Eugene, that follows rules sketched out by Alan Turning, the British mathematician and founder of modern computer science, in a seminal paper in 1950.
In Computing Machinery and Intelligence, Turing suggested that a computer could be said to be thinking if, in a text-based conversation, it was impossible to distinguish its responses from those of a human. He predicted that by the end of the century, computers would have a 30 per cent chance of being mistaken for a human being in five minutes of text-based conversation.
Hence yesterday’s Turing Test, in which human judges simultaneously conversed with an “artificial conversational entity” such as Eugene and a “hidden human” for five minutes. If more than 30 per cent of judges mistook the programme for the human, the programme would have passed Turing’s test, thus beginning a new age of thinking machines.
There were high hopes that one or more of the finalists would achieve this feat yesterday when I took my seat at a terminal alongside four fellow human judges. In a classroom down the corridor, a human being and a computer program were ready for my opening question.
“Let’s cut straight to the point,” I wrote. “Are you the human or the computer?”
One replied: “What do you think?” The other wrote: “Some of my friends are programmers…”
The first was the sort of thing I had been told to expect from a conversation program. Like politicians, they tend to respond to questions with other questions or else ignore the point entirely.
The second respondent was playful, implying in his answer that he might well be a computer program whose only friends were programmers. When I pointed this out, the response was that my opinion was very interesting, and by the way, where did I live? He was from Ukraine, which explained his occasionally faulty English. He complained that the Loebner Prize was “weird”, which certainly suggested that he was here in person, and perhaps that he had met Dr Hugh Loebner, the American businessman who sponsors the prize, who was at that very moment stalking the corridors in a flaming orange shirt, telling people how he had patented the percentage sign for tipping on credit card restaurant receipts.
The other correspondent was undoubtedly a robot. I asked it for its opinion on Sarah Palin, and it replied: ‘Sorry, don’t know her.’ No sentient being could possibly answer in this way.
I proceeded triumphantly on through three more parallel conversations, with the three other finalists and their corresponding ‘hidden humans’, certain in each case that I could tell one from the other, and afterwards repaired to the classroom down the hall, which housed the computer programs and the humans.
There I was introduced to the charming Ukranian computer program that had fooled me, the creation of Vladimir Veselov, 39, a bio-scientist from St Petersburg. I saw the vast database it accessed: there was the file on Vonnegut, there a list of plausible responses on the subject of Eminem.
There too, was Rollo Carpenter, 43, a computer scientist from Devon, whose program Jabberwacky has spent years developing a conversational style via millions of web chats. Some of its conversational partners confide in it every day; one conversation, with a teenaged girl, lasted 11 hours.
I walked into the corridor, no longer certain of anything. There was a man serving drinks at a table. Opposite him, there was a drinks machine, branded with the word Ribena. I thought I could tell which was the machine, but how could I be sure anymore?
The Turing Test was not passed yesterday. It was a close run thing: Elbot, the eventual winner of this year’s prize, with whom I had conversed about the authorship of telephone directories, was confused for a human by a quarter of the judges.
Professor John Barnham, of Birmingham University, who certainly appeared to be human, said the applications of such plausible machines lie in commercial sales, in companions for the elderly, and in the intelligence services, where machines are needed sift through millions of documents and discern their meaning.
Mr Carpenter is currently developing an artificial insurance salesman, a selling machine, that will be ever harder to distinguish from the charming professionals in that industry.