The state space of deep learning systems is vast. As we’ve seen with adversarial examples, that creates opportunity to deliberately craft inputs that fool a trained network. Forget adversarial examples for a moment though, what about the opportunity for good old-fashioned bugs to hide within that space? Experience with distributed systems tells us that there are likely to be plenty! And that raises an interesting question: how do you test a DNN?

At first glance this seems like more of the same adversarial stuff, fun as that may be, but they seem to do a better job finding real world scenarios that are misclassified. Nothing malicious, per se, just bad luck.