undebuggability

The simplest answer is the large overhead required by the techniques described here, and the consequential undebuggability of programs based on these techniques. This makes problems admitting more immediately testable solutions more attractive in some ways.

1980 May 12, Jon Doyle, “Introduction”, in A Model for Deliberation, Action, and Introspection (AD-A105 666), Cambridge, Mass.: Massachusetts Institute of Technology, Artificial Intelligence Laboratory, section 6 (Sketch of a Computational Argument for the Approach), subsection 1 (Why have the facts of the fundamental argument been overlooked?), subsubsection 1 (Initial Programming Complexity), page 33

Old programs do not learn, they simply fade away. So do human beings, their undebuggable programs replaced by younger, possibly less tangled, ones in other human heads. But at least until the state of undebuggability is reached, human programs are modified adaptively and repeatedly by learning processes that don’t require a knowledge of the internal representation.

1983, Herbert A[lexander] Simon, “Why Should Machines Learn?”, in Ryszard S[tanisław] Michalski, Jaime G[uillermo] Carbonell, Tom M[ichael] Mitchell, editors, Machine Learning: An Artificial Intelligence Approach (Symbolic Computation: Artificial Intelligence), Berlin; Heidelberg: Springer-Verlag Berlin Heidelberg, →ISBN, part 1 (General Issues in Machine Learning), section 6 (A Role for Learning), page 34