When AI Crashes: Classifying Failure Modes in Safety-Critical Software

It is dangerous to treat AI systems like any other type of software development. While the code may run properly, the model may still have 99% confidence that a kangaroo is a pedestrian. AI systems can be broken down into two types of failures; perception failures and planning failures. These failures are difficult to determine because they do not create error messages that a developer would see while developing the application. Instead of returning an error message stating that the model does not understand the input, the model could return a prediction such as "speed limit 45 mph".