Hunt for the Black Swan

Author:Murphy | View: 27414 | Time: 2025-03-23 19:22:30

When developing a new model or algorithm, it is tempting to test it over and over again with similar examples that all work perfectly. While this may be fun, it doesn't really help you in understanding and improving your model. You learn from errors, so cause your model to fail!

Imagine your data science teammate comes to you and tells you about the new model they just trained. It is so awesome, it can classify all kinds of baby animal images. Of course, your teammate will start showing you how well it works by letting the model classify a bunch of images all correctly. At this moment the best thing to ask your teammate for is the following:

Show me some cases where it fails

At first, this might sound counterintuitive. Of course, it is fun to see the model work and you don't want to demotivate your teammate by causing the model to fail, but what gives you more insight into how the model behaves: Seeing the model work or seeing it fail?

If it classified N images of super cute kittens correctly, it will, most likely, also classify the N+1th image correctly, if it looks just like the others. Are you surprised? No. Did you learn anything about the model's behavior? No.

However, if you find those images that cause the model to fail, you might eventually get an idea of the images the model still has problems with. That is worth a lot! Now you can start to understand the model and improve it even more.

Formal hypotheses and black swans

The theoretical background on the search for failures has a long tradition. For hundreds of years, very smart people were thinking and debating about the question of how we can derive common rules from observations. The sun has risen in the morning today, and so did it yesterday and the day before. Does that mean that it will rise again tomorrow? Well, not necessarily. Another example: I went to the park and all swans I saw there were white, so I might formulate my hypothesis

All swans are white.

As a good scientist, I will prove my hypothesis, of course, so I go to the next park and look at the swans. They are white too. So, is my hypothesis proven? No, because if I wanted to prove it, I would have to check all swans. Well, I don't have time for that, so what should I do? After having seen N white swans, looking at the N+1th white swan will give me no additional information. I should rather:

Try to find one black swan.

Why is that? Wouldn't that falsify my hypothesis? Yes, that is exactly what it would do. And that is what I should aim at.

Formally speaking, if I look at N white swans and derive that all swans are white, I do a logical induction. The logical induction, however, has one drawback: It is wrong.

The → can be read as "implies" and the ∧ means "and".

The above statement is an induction and we could read it as

My hypothesis H implies an observation B, and I observed this observation B. That implies, that my hypothesis H is true.

The statement, in total, is wrong though. In our example, the statement would be:

My hypothesis ‘all swans are white' (H) implies, that the next swan I observe is white (B). The next swan I observe is white indeed (B). That implies, that my hypothesis is true.

Again, that is not true. While the observation of a white swan is consistent with the hypothesis, it does not imply its truth. It is just not speaking against the hypothesis. If you are not convinced, take the following example:

My hypothesis ‘all swans are painted white by a secret organization of the US government' implies, that the next swan I observe is white (B). The next swan I observe is white indeed (B). That implies, that my hypothesis is true.

Simply not true.

However, there is still hope. While the above statement was wrong, the following is true:

The → can be read as "implies", the ∧ means "and", ¬ means "not".

We can read it as

My hypothesis H implies an observation B, and I observed not B. This implies that my hypothesis H is not true.

or phrase your example as

My hypothesis ‘all swans are white' (H) implies, that the next swan I observe is white (B). The next swan I observe is not white (¬ B). That means that my hypothesis is not true (¬ H).

That is a true statement (and for the formal logic nerds amongst you, it is called a modus tollens). If I find one example that speaks against the hypothesis, the hypothesis is proven to be wrong, and that indeed is a new piece of information I didn't have before.

All together that means, that hypotheses can never be proven or verified. They can just be falsified. On the other hand, if a hypothesis survives many of my attempts to falsify it, that speaks in favor of the hypothesis.

Back to the model

So, how is all that related to your teammate's model? You don't want to prove or falsify a plain hypothesis here, but the main idea was, that the gain of information comes from the cases that go wrong. Simply speaking, cases where the model works, don't tell you anything you didn't know already. To understand the model's behavior, look at those cases where the model fails. For example:

An entity recognizer that recognizes names can detect the names ‘Alice' and ‘Paula'.

Can it also detect ‘Kathrin'? Yes. Did you learn anything from that? No.
Can it also detect ‘Linh'? No. → Maybe it has problems with Asian names.

An image classifier can detect the location where landscape images have been taken. It correctly detects that the photos from your last vacation were taken at a beach in Indonesia.

Can it also detect the location of the photographs you took the year before in India? Yes. Did you learn anything from that? No.
Does it also work for the photographs your granddad took on his first trip to Italy in the 50s? No. → Maybe the data misses old black-and-white photographs.

A cool new robot hand is so flexible and can be controlled in so much detail, that it can play a C-major scale on a piano.

Can it also play an F-major scale? Yes. Did you learn anything from that? No.
Can it also play a Mozart sonata? No. → Maybe its flexibility is still limited and a Mozart sonata is too difficult for it.

All those examples show how you gain knowledge from the cases that fail, not from those that work. In the same way, you can learn more about your teammate's model that classifies cute animals:

Can it also classify rare animals like baby octopuses or tree kangaroos?
Does it also work with different image backgrounds?
Can it classify baby fish, which look exactly like their parents, just smaller?
What happens, if there are multiple different animals in one picture?

Those are just a few examples that can help you understand the model's behavior. When breaking the model, be creative!

Conclusion

I just showed you why breaking your model is more helpful than seeing it work. The cases where it fails are the ones that carry useful information, just like trying to falsify a hypothesis is the key to strengthening it. Now it's time to put formal logic into action! Next time your teammate comes to you to show you their new model, take a moment to appreciate it and celebrate the cases where it works fine. But after that, start to break it and find the edge cases. Those are the ones that will help you improve the model even more. Hunt for the black swans!