Machine learning pioneer Geoffrey Hinton came up with a seemingly intractable dilemma: “Suppose you have cancer and you have to choose between a black box AI surgeon that cannot explain how it works but has a 90% cure rate and a human surgeon with an 80% cure rate. Do you want the AI surgeon to be illegal?” But data practitioner Vicki Boykis responded by adding another wrinkle to the question. “Suppose you have to choose between a black box AI surgeon that runs on TensorFlow 1.0 on an EC2 instance that hasn’t been upgraded to Python 3 but has a 80% cure rate and a black box AI surgeon with an 80% cure rate that runs on Excel vlookups. Do you want to live on this planet?” For those of you not fluent in techspeak, what she is suggesting that the choice will likely come down to one kind of poorly implemented technical solution versus another kind of poorly implemented technical solution. Both hidden behind a “black box” of opacity.
Anaconda CEO Peter Wang finally exploded the entire premise of the question by noting that “[i]f the question is purely around “black boxes”, then we already have those: medicine. Every time you swallow a pill, you are making the trade about likelihood-of-cure vs likelihood-of-complications.” In other words, neither the human body and medicine that manipulates bodily functions are fully knowable. Probability governs their interaction within individual patients. Taking a step back, I submit that all of this suggests that the term “black box” does more harm than good. Because, if you think about it, all boxes are black. The black box is a convenient way for us to separate ourselves from the world we interact with, and is a tribute to the way in which ignorance is our dominant experience of that world despite our tremendous scientific and technical accomplishments.
Why? The etymology of “black box” is very complicated, simultaneously referring to actual artifacts and concepts associated with them as well as artifacts with remarkably different functions. Aircraft black box recorders are not the same thing as black boxes in electrical engineering, and neither are necessarily the same as the overarching concept of the black box. A black box is often taken to be something that executes a function but only supplies partial at best information about the structure that generates the function output. But this definition also conflicts with the actual history of black boxes as creatures of secrecy, made to provide functionality without revealing their components to military or commercial competitors. These two definitions collapse into each other quite frequently.
Making matters even more convoluted is the way in which the black box concept has been broadened over time to anything that in some way generates behavior but discourages or inhibits investigation of its origin and mechanics. Something that is “taken for granted” such as a solidified scientific concept assumed to be purely natural and independent of human artifice. For sure there is something to this idea but perhaps much less than is commonly assumed. How so? Well, let’s go back to a first-principles definition of blackboxing. Ranulph Glanville, a philosopher and engineer that devoted his life to theorizing black boxes, eventually concluded that the black box was just a convenient abstraction used to demarcate a boundary between ourselves and the unknown world.
When we are faced with something new, we do not know what the new thing is (this is what new means). Thus, we are faced with something we do not know about. That is exactly what a black box is. It is also the situation faced all the time (as far as we can tell - Piaget, 1955) by a newborn child - and hence by all of us, sometime ago. If it was the child’s situation, then all our knowledge is based on functional descriptions made of black boxes. A more philosophical reason is that which is implied in the assumption of the constructivist mantle (which is what the black box implies). Behind this assumption is the notion that, for instance, our experience of our perception of the world is in images, and not in electrical impulses - that is, that our experience is an active interaction with the (presumed) reality “out there,” (e.g., von Foerster, 1973, von Glasersfeld, 1974, Gregory, 1973). This argument is clearly founded in psychology, but similar ones abound.
Suppose you have something that can receive inputs and generate outputs. You have only partial ability to observe and understand how it works. In order to have full understanding, you would need to know every possible response for every possible interaction between the object and the surrounding environment. Clearly, this is not really feasible except for very trivial cases. Especially because even identical things can behave very differently depending on their state histories. True of both biological and engineered artifacts. What you can do is feed inputs into the object and receive outputs, and infer relationships between patterns of input and output. But note that something else is going on while you are doing this.
The object’s output is in turn YOUR input, and how you decide to act on the object after this input is YOUR output. And, from the object’s perspective, YOU are a black box it does not fully see within. This is part of why the “whitening” of the box is particular to the observer; what is revealed may not hold for another observer or even the same observer indefinitely. Thus our account of input and output, Glanville explains, is a relation between them but ultimately one that is convenient for us psychologically rather than what is necessarily true. A functional description based on what has worked in the past. And one that is particular to our interaction with an artifact and our relation to it. Ironically, we seem to accept ignorance in order to gain predictability and control over something of interest.
Glanville’s gambit here is similar to the one that Daniel Dennett makes in his notion of the “intentional stance” – it is permissible to attribute intentional properties to something that may lack them in order to predict its behavior if we cannot do so from lower levels of description. And this can frequently be hazardous. Much of the sciences of control, broadly speaking, attribute properties such as “memory” or “learning” in order to explain particular patterns of behavior. This is again necessary to make sense of behavior in the absence of full knowledge of all possible state trajectories. But there is always the possibility that one is wrong, that something that seems to be a consequence of past states could just be purely determined by the present. A system need not have the ability for past states to influence the present behavior if the only state that matters is the most immediate one it is in.
Much of the uncertainty of dealing with other people lies precisely in this dilemma. Is their behavior a product of the situation they are in? Or is it something more permanent? Did they learn, plan, or adapt over time how to do something, or did they just take advantage of an immediate opportunity? Over time, as you observe more and more data, you can become more confident about attributions you make. One time something can credibly be written off as a mistake. But it starts to look more and more attributable to the person as more instances pile up. But even then, you’re not quite in the clear. Because, as Glanville previously noted, you have a functional description that worked well in the past. It may not work in the future, and it may not be as convincing to someone with more information about the person in question than you have.
And, of course, you are a black box to the person you are interacting with, so they in turn are going through a variation on your problem. Perhaps this also clarifies some of the problems with computers and data discrimination as the targets of such discrimination are black boxes to the computer. The problem is not just that the computer is a black box to them. It cannot peer into their souls, it does not fully understand the circumstances that generate their observed behavior. It only has the data that has been supplied, from which it can infer patterns that robustly predict future behavior. This is the true darkness of the black box: not only are black boxes everywhere, but the relationships between them are circular in nature. It is a tremendous statement about the persistence of ignorance even at the height of scientific and technological achievement.
This ignorance is highlighted by the degree to which people cannot talk about computers without invoking animistic properties or attributing internal desires and goals. Because it exposes something profoundly weird about the world and our perception of it. The substance of primary intentionality is not necessary to generate behavior, but behavior can produce the subjective appearance that it was originated by primary intentionality. This is not a “new” problem to be sure. And it represents the sub-optimal responses of scientists and philosophers to a choice between two undesirable things: either attribute inner will and drive to everything or retreat into the miasma of whether a “real” world exists outside of the observer. So the black box is a cognitive hack that allows the postponement of this choice, perhaps until something better arrives that obviates the necessity of deciding between the two bad options. Fair enough. We all have to be practical.
What I have an issue with is that the cognitive hack is taken to be a real thing rather than an expedient we created to cope with this dilemma. Enormous amounts of time and energy is devoted to “opening the black box” but as Glanville argued any such openings are always going to be local and ephemeral. Moreover, this mantra also, well, blackboxes the actual problems at work by putting them into the abstraction of mysterious black boxes. Machines will always contain some margin of unknowability and uncertainty. But what are the responsibilities of humans? We manage and curate the information that goes into machines, we determine how machines interact with each other and the outside world. Much of how machines work is a product of the circumstances we choose to put them in. They are managed by large, impersonal, and often Kafka-esque bureaucracies and are designed along bureaucratic assumptions. So why would they not be cruel and impersonal, like bureaucracies are?
I do not think we need to invest ourselves in unpacking layers of mysterianism to be capable of understanding the consequences of our choices, even if rectifying them may be intractable due to the margin of uncertainty surrounding all of our enterprises. And for that to happen we need to de-emphasize black boxes. Make them technical artifacts with limited relevance as they once were prior to the theoretical explosion of black box terminology. Or, if we want to retain this terminology, note that it is a contrivance with a pragmatic justification when we use it to describe things of interest to us. I am not optimistic about our ability to tame the black box terminology. But that’s only just an observation based on what has happened in the past, particular to the evidence I’ve personally interacted with. The world is a black box, so who knows?