The deepest problem with deep learning

To begin with, and to clear up some misconceptions. I don’t hate deep learning, not at all; we used it in my last company (I was the CEO and a Founder), and I expect that I will use it again; I would be crazy to ignore it. I think — and I am saying this for the public record, feel free to quote me — deep learning is a terrific tool for some kinds of problems, particularly those involving perceptual classification, like recognizing syllables and objects, but also not a panacea . In my NYU debate with LeCun, I praised LeCun’s early work on convolution, which is an incredibly powerful tool. And I have been giving deep learning some (but not infinite) credit ever since I first wrote about it as such, in The New Yorker in 2012, in my January 2018 Deep Learning: A Critical Appraisal article , in which I explicitly said “I don’t think we should abandon deep learning” and on many occasions in between. LeCun has repeatedly and publicly misrepresented me as someone who has only just woken up to the utility of deep learning , and that’s simply not so.

Lecun’s assertion that I shouldn’t be allowed to comment is similarly absurd; science needs its critics (LeCun himself has been rightly critical of deep reinforcement learning and neuromorphic computing), and although I am not personally an algorithm engineer, my criticism thus far has had lasting predictive value. To take one example, experiments that I did on predecessors to deep learning, first published in 1998, continue to hold validity to this day, as shown in recent work with more modern models by folks like Brendan Lake and Marco Baroni and Bengio himself . When a field tries to stifle its critics, rather then addressing the underlying criticism, replacing scientific inquiry with politics, something has gone seriously amiss.

But LeCun is right about one thing; there is something that I hate. What I hate is this: the notion that deep learning is without demonstrable limits and might, all by itself, get us to general intelligence, if we just give it a little more time and a little more data, as captured in Andrew Ng’s 2016 suggestion that AI, by which he meant mainly deep learning, would either “now or in the near future“ be able to do “any mental task” a person could do “with less than one second of thought”.

Generally, though certainly not always, criticism of deep learning is sloughed off, either ignored, or dismissed, often in ad hominem way. Whenever anybody points out that there might be a specific limit to deep learning , there is always someone like Jeremy Howard to tell us that the idea that deep learning is overhyped is itself overhyped . Leaders in AI like LeCun acknowledge that there must be some limits, in some vague way, but rarely (and this is why Bengio’s new report was so noteworthy) do they pinpoint that what those limits are, beyond to acknowledge the data-hungry nature of the systems.

Others like to leverage the opacity of the black box of deep learning to suggest that that are no known limits. Last week, for example, Tom Dietterich said (in answer to a question about the scope of deep learning):

Dietterich is of course technically correct; nobody yet has delivered formal proofs about limits on deep learning, so there is no definite answer. And he is also right that deep learning continues to evolve. But the tweet (which expresses an argument I have heard many times, including from Dietterich more than once) neglects the fact we also do have a lot of strong suggestive evidence of at least some limit in scope, such as empirically observed limits reasoning abilities, poor performance in natural language comprehension, vulnerability to adversarial examples, and so forth. (At the end, I will even give an example in the domain of object recognition, putatively deep learning’s strong suit.)

To take another example, consider LeCun, Bengio and Hinton’s widely-read 2015 article in Nature on deep learning , which elaborates the strength of deep learning in considerable detail. There again much of what was said is true, but there was almost nothing acknowledged about limits of deep learning, and it would be easy to walk away from the paper imagining that deep learning is a much broader tool than it really is. The paper’s conclusion furthers that impression by suggesting that deep learning’s historical antithesis — symbol-manipulation/classical AI — should be replaced (“new paradigms are needed to replace the rule-based manipulation of symbolic expressions on large vectors.”). The traditional ending of many scientific papers — limits — is essentially missing, inviting the inference that the horizons for deep learning are limitless; symbol-manipulation soon to be left in the dustbin of history.

The strategy of emphasizing strength without acknowledging limits is even more pronounced in DeepMind’s 2017 Nature article on Go, which appears to imply similarly limitless horizons for deep reinforcement learning, by suggesting that Go is one of the hardest problems in AI. (“Our results comprehensively demonstrate that a pure [deep] reinforcement learning approach is fully feasible, even in the most challenging of domains”) — without acknowledging that other hard problems differ qualitatively in character (e.g., because information in most tasks is less complete than it is Go) and might not be accessible to similar approaches. (I discuss this further elsewhere .)

It worries me, greatly, when a field dwells largely or exclusively on the strengths of the latest discoveries, without publicly acknowledging possible weaknesses that have actually been well-documented.

Here’s my view: deep learning really is great, but it’s the wrong tool for the job of cognition writ large; it’s a tool for perceptual classification, when general intelligence involves so much more. What I was saying in 2012 (and have never deviated from) is that deep learning ought to be part of the workflow for AI, not the whole thing (“just one element in a very complicated ensemble of things”, as I put it then, “not a universal solvent, [just] one tool among many” as I put it in January). Deep learning is, like anything else we might consider, a tool with particular strengths, and particular weaknesses. Nobody should be surprised by this.

When I rail about deep-learning, it’s not because I think it should be “replaced” (cf. Hinton, LeCun and Bengio’s strong language above, where the name of the game is to conquer previous approaches), but because I think that (a) it has been oversold (eg that Andrew Ng quote, or the whole framing of DeepMind’s 2017 Nature paper), often with vastly greater attention to strengths than potential limitations, and (b) exuberance for deep learning is often (though not universal) accompanied by a hostility to symbol-manipulation that I believe is a foundational mistake. In the ultimate solution to AI.

I think it is far more likely that the two — deep learning and symbol-manipulation-will co-exist, with deep learning handling many aspects of perceptual classification, but symbol-manipulation playing a vital role in reasoning about abstract knowledge. Advances in narrow AI with deep learning are often taken to mean that we don’t need symbol-manipulation anymore, and I think that it is a huge mistake.

So what is symbol-manipulation, and why do I steadfastly cling to it? The idea goes back to the earliest days of computer science (and even earlier, to the development of formal logic): symbols can stand for ideas, and if you manipulate those symbols, you can make correct inferences about the inferences they stand for. If you know that P implies Q, you can infer from not Q that not P. If I tell you that plonk implies queegle but queegle is not true, then you can infer that plonk is not true.

In my 2001 book The Algebraic Mind , I argued, in the tradition of Newell and Simon, and my mentor Steven Pinker, that the human mind incorporates (among other tools) a set of mechanisms for representing structured sets of symbols, in something like the fashion of a hierachical tree. Even more critically I argued that a vital component of cognition is the ability to learn abstract relationships that are expressed over variables — analogous to what we do in algebra, when we learn an equation like x = y + 2, and then solve for x given some value of y . The process of attaching y to a specific value (say 5) is called binding ; the process that combines that value with the other elements is what I would call an operation . The central claim of the book was that symbolic processes like that — representing abstractions, instantiating variables with instances, and applying operations to those variables, was indispensible to the human mind. I showed in detail that advocates of neural networks often ignored this, at their peril.

The form of the argument was to show that neural network models fell into two classes, those (“implementational connectionism”) that had mechanisms that formally mapped onto the symbolic machinery of operations over variables, and those (“eliminative connectionism”) that lacked such mechanisms. The ones that succeeded in capturing various facts (primarily about human language) were ones that mapped on; those that didn’t failed. I also pointed out that rules allowed for what I called free generalization of universals, whereas multilayer perceptrons required large samples in order to approximate universal relationships, an issue that crops up in Bengio’s recent work on language .

Nobody yet knows how the brain implements things like variables or binding of variables to the values of their instances, but strong evidence (reviewed in the book) suggests that brains can (pretty much everyone agree that at least some humans can do this when they do mathematics and formal logic; most linguistics would agree that we do it in understanding the language; the real question is not whether human brains can do symbol-manipulation at all, it os how broad is the scope of the processes that use it.)

The secondary goal of the book was to show that that was possible to build the primitives of symbol manipulation in principle using neurons as elements. I examined some old ideas, like dynamic binding via temporal oscillation, and personally championed a slots-and-fillers approach that involved having banks of node-like units with codes, something like the ASCII code. Memory networks and differentiable programming have been doing something a little like that, with more modern (embedding) codes, but following a similar principle, the latter embracing an ever-widening array of basic micro-processor operations such as copy and compare of the sort I was lobbying for. I am cautiously optimistic that this approach might work better for things like reasoning and (once we have a solid enough machine-interpretable database of probabilistic but abstract common sense) language.

Whatever one thinks about the brain, virtually all of the world’s software is built on symbols. Every line of computer code, for example, is really a description of some set of operations over variables; if X is greater than Y, do P, otherwise do Q; concatenate A and B together to form something new, and so forth. Neural networks can (depending on their structure, and whether anything maps precisely onto operations over variables) offer a genuinely different paradigm, and are obviously useful for tasks like speech-recognition (which nobody would do with a set of rules anymore, with good reason), but nobody would build a browser by supervised learning on sets of inputs (logs of user key strokes) and output (images on screens, or packets downloading). My understanding from LeCun is that a lot of Facebook’s AI is done by neural networks, but it’s certainly not the case that the entire framework of Facebook runs without recourse to symbol-manipulation.

And although symbols may not have a home in speech recognition anymore, and clearly can’t do the full-stack of cognition and perception on their own, there’s lot of places where you might expect them to be helpful, albeit in problems that nobody, either in the symbol-manipulation-based world of classical AI or in the deep learning world, has the answers for yet — problems like abstract reasoning and language, which are, after all the domains for which the tools of formal logic and symbolic reasoning are invented. To anyone who has seriously engaged in trying to understand, say, commonsense reasoning, this seems obvious.

Yes, partly for historical reasons that date back to the earliest days of AI, the founders of deep learning have often been deeply hostile to including such machinery in their models; Hinton, for example, gave a talk at Stanford in 2015 called Aetherial symbols , in which tried to argue that the idea of reasoning with formal symbols was “as incorrect as the belief that a lightwave can only travel through space by causing disturbances in the luminiferous aether.”

Hinton didn’t really give an argument for that, so far as I can tell (I was sitting in the room). Instead, he seemed (to me) be making a suggesting for how to map hierarchical sets of symbols onto vectors. That wouldn’t render symbols “aether”, it would make them very real causal elements with a very specific implementation, a refutation of what Hinton seemed to advocate. (Hinton refused to clarify when I asked.) From a scientific perspective (as opposed to a political perspective), the question is not what we call our ultimate AI system, it’s how does it work. Does it include primitives that serve as implementations of the apparatus of symbol-manipulation (as modern computers do), or work on entirely different principles? My best guess is that the answer will be both: some but not all parts of any system for general intelligence will map perfectly onto the primitives of symbol-manipulation; others will not.