Abstract

In 2016, Alex Graves, Greg Wayne and others at DeepMind published a paper in Nature proposing a new model for neural networks: the differentiable neural computer (DNC) [14,25]. This model is a type of neural memory network designed so that "[. . .] memory can be selectively written to as well as read, allowing iterative modification of memory content." Utilizing external memory, DeepMind’s network achieved the lowest error rate yet published on the bAbI dataset, a dataset composed of short stories followed by questions on the stories. This dataset demonstrated the DNC’s ability to reason deductively and process natural language. But the dataset itself is quite limited: it possesses only — words, all of which have at least –instances and which are in the lexicon of a small child. Moreover, each of the twenty question types in the dataset has 10,000 training examples and no question requires more than a single type of reasoning. These dataset limitations suggest that the ground breaking results published by DeepMind may mask the deep and pervasive limitations of not only the differential neural computer but also of neural networks as a whole.To push the DNC to the limits of its capabilities, we introduce it to the DREAM dataset, a dataset originally presented by Kal Sun et al [27]. This dataset is much smaller than the bAbI dataset, it contains only 10,197 questions split into fivedifferent types. Additionally, the dataset is makes use of – words, where many words have as little as – instances. While Sun et al originally publish the dataset in multiple-choice format, we train the DNC in question and answer format— whichvis comparable to the bAbI dataset and further increases the difficulty of answering correctly. We expect that— provided this dataset— the DNC will experience task failure.What we did not expect is just how pervasive that failure would be.

Advisor

Sommer, Nathan

Department

Computer Science

Disciplines

Artificial Intelligence and Robotics

Keywords

Differential Neural Computer, Neural Network, Artificial Intelligence, AI, Natural Language Processing, Computer Science

Publication Date

2020

Degree Granted

Bachelor of Arts

Document Type

Senior Independent Study Thesis

Download

Share

COinS
 

© Copyright 2020 Erika Goetz