Researchers on an artificial intelligence (AI) system called "Torobo-kun," which is being developed with an ultimate goal of passing the entrance exam for the University of Tokyo, Japan's top university, gave up on the dream for now after studies led by the National Institute of Informatics (NII) hit a snag.
While AI has been progressing rapidly, beating a former world Go champion and showing off other skills, the system is known to be poor in its reading comprehension ability that can lead to the right answer. A team of researchers from the NII and other entities have remodeled Torobo-kun by having it try out mock university entrance exams since 2013.
This fiscal year, Torobo-kun for the first time succeeded in earning mock exam scores that show it has an 80 percent chance of passing the entrance exams for such prestigious private schools as Meiji University, Aoyama Gakuin University, Rikkyo University, Chuo University and Hosei University in the Kanto region -- which are collectively called "MARCH" from their acronyms -- as well as Kansai University, Kwansei Gakuin University, Doshisha University and Ritsumeikan University in the Kansai region -- which are collectively referred to as "Kankandoritsu." The trial exam was held by a major correspondence education company.
Furthermore, Torobo-kun also fared well in a mock exam run by a major cram school for the second-stage test of the University of Tokyo's entrance exam, posting a deviation value of 76.2 in mathematics.
However, NII researchers gave up on the lofty goal of having Torobo-kun pass the University of Tokyo admission exam this past fall. Noriko Arai, a professor at the NII, explains: "An AI system doesn't comprehend the essential meanings and has limits in its ability to read into sentences. We have found that there are no prospects for the system to score well enough to pass the University of Tokyo's entrance exam."
From the outset, Torobo-kun has performed poorly in reading comprehension tests in every subject. For example, when it tried to answer a world history question: "Who is the father of Cao Pi, who became the first emperor of Wei during the Three Kingdoms period in China?", Torobo-kun failed to give the right answer. While Torobo-kun had known Cao Pi as the "child" of Wei King Cao Cao, it failed to come up with his "father" as it didn't understand the father-child relationship.
A team of researchers tasked with developing Torobo-kun's English-language ability introduced the so-called "deep learning" method for the first time into Torobo-kun in an attempt for the AI to get more correct answers in reading ability tests involving five to 10 sentences each. Deep learning is known to be an innovative technology in that it can deepen the learner's ability by repeatedly reading a vast volume of image and text data into an AI system. However, Torobo-kun failed to earn higher marks, possibly because of data shortages, prompting the researchers to abandon the deep learning method. For an AI system to raise the percentage of questions answered correctly through deep learning, an enormous volume of data must be read into the system.
Ryuichiro Higashinaka, chief research scientist at NTT Communication Science Laboratories, who took part in the project, said, "In order to pass the entrance exam (for the University of Tokyo), a minimum requirement is to learn 1 million sets of problem statements and right answers. It would cost a lot just to prepare the data, making it an unrealistic option."
Even after Torobo-kun gave up on passing the University of Tokyo admission test, many researchers are still optimistic about developing AI's ability. Hitoshi Matsubara, professor at Future University Hakodate, is seeking to win a literary award by having an AI system that his project team has been developing write a novel. His project also aims to have the system complete the unfinished novel "Kyomu Kairo" by renowned science fiction writer Sakyo Komatsu by examining the late author's writing styles and wordings.
"Although the current method has its own limits, we may possibly reach levels where an AI system makes out the meaning by having it adopt a different learning method than humans,' " Matsubara said.
DeepMind, a subsidiary of Google Inc., took the world by surprise after its "AlphaGo" computer program defeated legendary Go player Lee Sedol. The company has been engaged in enhancing an AI system's reading comprehension ability by reading hundreds of thousands of digitized news articles into the system under the deep learning method. As a result, the company reportedly got a relatively high percentage of questions answered correctly when it made the AI system fill in the blank with proper nouns in the summarized versions of the news stories.
Another research community is aspiring to develop an AI system that can make correct judgment even with little data it has, just like human brains can make proper judgment by retrieving past similar memories when faced with an unprecedented situation.
"It is not so much that an AI system has its limits in understanding the meaning as that the area is the forefront of research," says Ken Sakamura, a computer scientist and a professor at the University of Tokyo.
NII's professor Arai commented, "As we have Torobo-kun's data at hand, attempts (to improve an AI system) are always welcome."
Who knows, an AI system may someday prove practice makes perfect.