The following essay is reproduced with permission from 
The conversationan online publication covering the latest research.
To some people, the term “black box” conjures up the recording devices on airplanes that are invaluable for post-mortem analysis if the unthinkable happens. For others, it evokes small, poorly equipped theatres. But the black box is also an important term in the world of artificial intelligence.
AI black boxes refer to AI systems whose inner workings are invisible to the user. You can give them input and get output, but you can’t examine the system code or the logic that produced the output.
Machine learning is the dominant subset of artificial intelligence. It underpins generative AI systems like ChatGPT And DALL-E 2. Machine learning has three components: an algorithm or set of algorithms, training data, and a model. An algorithm is a set of procedures. In machine learning, an algorithm learns to identify patterns after being trained on a large number of examples – the training data. Once a machine learning algorithm has been trained, the result is a machine learning model. The model is what people use.
For example, a machine learning algorithm could be designed to identify patterns in images, and the training data could be images of dogs. The resulting machine learning model would be a dog spotter. You would give it an image as input and you would get as output if and where in the image a set of pixels represents a dog.
Any of the three components of a machine learning system can be hidden or in a black box. As is often the case, the algorithm is known to the public, which makes it less efficient to put it in a black box. So, to protect their intellectual property, AI developers often put the model in a black box. Another approach taken by software developers is to obfuscate the data used to train the model – in other words, to put the training data in a black box.
The opposite of a black box is sometimes called a glass box. An AI glass box is a system whose algorithms, training data, and model are all available to everyone. But researchers sometimes even characterize some of these aspects as a black box.
This is because researchers don’t quite understand how machine learning algorithms, in particular deep learning algorithms, work. The domain of Explainable AI works to develop algorithms that, while not necessarily a glass box, can be better understood by humans.
Why AI Black Boxes Matter
In many cases, there are good reasons to be wary of black box machine learning algorithms and models. Suppose a machine learning model has made a diagnosis about your health. Would you like the model to be a black box or a glass box? What about the doctor prescribing your treatment? Perhaps she would like to know how the model arrived at her decision.
What if a machine learning model that determines if you qualify for a business loan from a bank turns you down? Wouldn’t you like to know why? If you did, you could more effectively appeal the decision or change your circumstances to increase your chances of getting a loan the next time around.
Black boxes also have important implications for software system security. For years, many people in the IT field thought that keeping software in a black box would prevent hackers from examining it and therefore make it secure. This assumption has largely been proven wrong because hackers can to debone software – that is, building a facsimile by closely observing how software works – and discovering vulnerabilities to exploit.
If the software is in a glass box, software testers and well-meaning hackers can examine it and notify the creators of weaknesses, thereby minimizing cyberattacks.
This article was originally published on The conversation. Read it original article.