AI Fails University of Tokyo Admission Test for Second Time
The National Institute of Informatics has given up on making its Todai Robot AI smart enough to get into the University of Tokyo.
Put one in the win column for us humans. Researchers in Japan have given up on developing an AI that is smart enough to pass the University of Tokyo’s admission test.
The National Institute of Informatics (NII) has been working on the Todai Robot AI for the last five years. The goal was to get Todai Robot into Japan’s top university by 2020.
The AI took the admission test in 2015 and scored a 511 out of 950. That score is well below the requirement, but it’s higher than the national average of 416. The researchers had high hopes the AI would perform better this time around, but it reportedly earned nearly the same score.
Noriko Arai, a professor at the NII, tells the Japan Times that the “AI is not good at answering a type of question that requires the ability to grasp meaning in a broad spectrum.” Essentially, it appears Todai Robot struggled with its critical thinking skills.
Here’s are some excerpts from a 2013 interview on the Todai Robot Project website that details some of the challenges an AI might encounter with the University of Tokyo’s admission test:
Why was passing the university entrance exam selected as the project’s goal?
Miyao The key point is that what’s difficult for people is different than what’s difficult for computers. Computers excel at calculation, and can beat professional chess and shogi players at their games. IBM’s “Watson” question-answering system*1 became a quiz show world champion. For a person, beating a professional shogi player is far harder than passing the University of Tokyo entrance exam, but for a computer, shogi is easier. What makes the University of Tokyo entrance exam harder is that the rules are less clearly defined than they are for shogi or a quiz show. From the perspective of using knowledge and data to answer questions, the university entrance exam requires a more human-like approach to information processing. However, it does not rely as much on common sense as an elementary school exam or everyday life, so it’s a reasonable target for the next step in artificial intelligence research.
Does the difficulty vary by test subject?
“What varies more than the difficulty itself are the issues that have to be tackled by artificial intelligence research. The social studies questions, which test knowledge, rely on memory, so one might assume they would be easy for computers, but it’s actually difficult for a computer to determine if the text of a problem corresponds to knowledge the computer possesses. What makes that identification possible is “Textual Entailment Recognition”*2, an area in which we are making progress, but still face many challenges. Ethics questions, on the other hand, frequently cover common sense, and require the reader to understand the Japanese language, so they are especially difficult for computers, which lack this common sense. Personally, I had a hard time with questions requiring memorization, so I picked ethics.”
The researchers now plan to shift their focus to studies related to the academic skills needed for written responses.
[Source:] Japan Times