How much are you willing to lose for A.I.?

Two months ago, I had a first meeting with one of the wind turbine industry leader. His goal was to use machine learning to make wind turbine inspection faster and cheaper.

As for each of these meetings, I started with the question :

How many errors are acceptable for you?

And he answered what everyone in the industry agrees on:

We can’t accept any error, our job is way too sensitive

Impossible perfection

Each project that involve security only accepts 0-failure processes. Cornis started in 2011 working with European and French Space Agencies. Space industry is one of the most careful to avoid mistakes.

But on January 18th, 1986, the space shuttle Challenger exploded shortly after launch.

Experts built this space shuttle. Other experts verified this space shuttle. Other experts tested this space shuttle.

And still, it exploded killing seven people. The best experts in the world made a tragic mistake.

Everything fails, the only question is how often. If any company sells you a zero-failure system, run away!

We all want our failure rate to be as far as possible from the close to 100% failure rate for plugging an USB key.

But if Space industry experts are making mistakes, you can be sure that any expert is making mistake too.

An A.I. algorithm replacing a human expert doesn’t need to make no error. It needs to make fewer errors than human experts.

We can compare expert and algorithm performance thanks to false negative.

A.I. is always beating humans… or not.

Unfortunately, Neural Networks don’t have an inherent false negative rate. These algorithms provide a score for our image, on being a defect.

The usual approach is to set a threshold at 0 to this score. By changing this threshold, the algorithm can meet more strict false negative needs.

You can decide the false negative rate (missed detection) of a trained neural network.

The consequence will on the number of false positive rate (number of bad detections). A lower false negative rate will bring more false positives.

For blade inspection, since the misses are critical we went for a very low false negative rate. As a consequence, experts still spend some time to sort out the false positive. But the inclusion of A.I. to the process didn’t affect the accuracy of their work.