When not to use machine learning or AI

Adventures in wishful thinking, nonstationarity, and pattern-finding

Imagine that you’ve just managed to get your hands on a dataset from a clinical trial. Exciting! To help you get in character, I made up some data for you to look at:

Pretend that these datapoints map out the relationship between the treatment day ( input “feature ” ) and the correct dosage of some miracle cure in milligrams ( output “prediction ” ) that a patient should receive for over the course of 60 days.

#The data:
(1,28)  (2,17)  (3,92)  (4,41)  (5,9)   (6,87)  (7,54) (8,3)   (9,78)  (10,67) (11,1)  (12,67) (13,78) (14,3) (15,55) (16,86) (17,8)  (18,42) (19,92) (20,17) (21,29) (22,94) (23,28) (24,18) (25,93) (26,40) (27,9)  (28,87) (29,53) (30,3)  (31,79) (32,66) (33,1)  (34,68) (35,77) (36,3)  (37,56) (38,86) (39,8)  (40,43) (41,92) (42,16) (43,30) (44,94) (45,27) (46,19) (47,93) (48,39) (49,10) (50,88) (51,53) (52,4)  (53,80) (54,65) (55,1)  (56,69) (57,77) (58,3)  (59,57) (60,86) ...

Now imagine that you’re treating a patient and it’s day 2. What dose do you suggest we use?

I really hope you answered “17mg” since this was definitely not supposed to be a trick question. How about day 4? 41mg ? Yes indeedy!

Now, how would you build software to output the right doses on days 1–5? Would you try to use machine learning (ML)? In other words, would you try to find patterns in these data and try to turn them into a recipe ( “model” ) for going from inputs to outputs?

No, of course you wouldn’t! You’d get your software to do exactly what you’re doing: look the answer up in a table. That way, you’ll get the right answer 100% of the time for all 60 days. No need for patterns here and no need for machine learning either.

So, what sort of situation requires machine learning?

How about now? It’s day 61. What’s the right answer here?

Well, we’ve never seen data for day 61, so there’s no way we can look up the answer here. What can we do? Are we out of luck? Can machine learning help us?

That depends.

If there’s no pattern that connects the inputs with the outputs, forget it. In that case, nothing can help us… short of actual magic , which doesn’t exist (in case you thought machine learning was it). Give up now!

We must find a useful pattern

But if there is a pattern and if (that’s a big if!) we could find it, then we could try to apply it to Day 61 to try to predict/guess the right answer. Perhaps machine learning might help us.

The pattern must generalize

The trouble is that it’s not enough for there to be a pattern in our data. That would be much too convenient. The pattern also has to be relevant beyond Day 60. What if the conditions are fundamentally different in Day 61, so the pattern doesn’t generalize ? For all you know, maybe on Day 61 all patients are fully cured or dead or on an incompatible medication. Then the pattern is no good to you.

Nonstationary universes

Let this sink in. If your data aren’t a useful window into tomorrow’s world — perhaps because a pandemic changed all the rules — it doesn’t matter how good your information was yesterday. If you live in an unstable corner of the universe, you’ll have a hard time justifying what we call stationarity assumptions . These roughly translate to “I believe that the rules haven’t changed” and they’re a requirement to pretty much all forecasting. If your past data suddenly don’t apply to your nonstationary future, you’re not allowed to use yesterday to predict tomorrow with a straight face.

Image: SOURCE

Maybe you’re in luck

But if there is a pattern and if this pattern is relevant to the new situation we find ourselves in, then we’re in business. We could go and find the pattern in the old data, make a recipe based on it, and then use that recipe to succeed on Day 61 and beyond!

Finding patterns and using them is what machine learning is all about.

When to use machine learning

In applied machine learning (and AI ), you’re not in the business of regurgitating memorized examples you’ve seen before — you don’t need ML for that, just look ’em up! —you’re here to learn.

Just repeat old answers? ML can do better! It succeeds on new examples.

Your mission? To build a solution that generalizes successfully (or pull the plug on your project). (What does “successfully” mean? I have a whole guide for you on that topic.)

Not cat. SOURCE

In other words, your solution is no good if it can’t handle new examples it has never seen before. Not dramatically new examples that break all the rules of a stationary universe, but slight twists on the learned theme.

We’re not here to memorize like a parrot. We’re here to generalize to new situations. That’s the power and the beauty of machine learning.

If you haven’t seen this exact combination of input values before (Day 61), what’s the right output answer? Well, maybe we can turn old patterns into a recipe that makes a decent guess.

For example, if you trained a cat/not-cat classifier from thousands of animal photos, you can ask it to tell you if a brand new photo has a cat in it, but you shouldn’t ask it to tell you whether a painting is in the Cubist style.

Is machine learning for you?

If you’re sick of hearing me call it a thing-labeler and an alternative approach to writing code , let me try putting it another way.

Machine learning is an approach to automating repeated decisions that involves algorithmically finding patterns in data and using these to make recipes that deal correctly with brand new data.

To know if machine learning is for you, I have three guides you might enjoy:

• Is your ML/AI project a nonstarter? A 22-item reality check(list)
• Advice for finding ML/AI use cases
• Getting started with ML/AI? Start here!

So, was there a useful pattern?

Still curious about Day 61? Turns out there *is* a pattern in the toy data I made for this example. I know this because I put it there. I can even promise you that it generalizes to Day The-Biggest-Number-You-Can-Think-Of-Plus-One because in these wildly nonstationary times, I find it luxuriously comforting to work with data that plays nice for change.

#The data:
(1,28)  (2,17)  (3,92)  (4,41)  (5,9)   (6,87)  (7,54) (8,3)   (9,78)  (10,67) (11,1)  (12,67) (13,78) (14,3) (15,55) (16,86) (17,8)  (18,42) (19,92) (20,17) (21,29) (22,94) (23,28) (24,18) (25,93) (26,40) (27,9)  (28,87) (29,53) (30,3)  (31,79) (32,66) (33,1)  (34,68) (35,77) (36,3)  (37,56) (38,86) (39,8)  (40,43) (41,92) (42,16) (43,30) (44,94) (45,27) (46,19) (47,93) (48,39) (49,10) (50,88) (51,53) (52,4)  (53,80) (54,65) (55,1)  (56,69) (57,77) (58,3)  (59,57) (60,86) ...

For those who like a challenge, why don’t you try see if your favorite machine learning algorithm can find the pattern and turn it into a useful recipe? Let me know what you think the answer for Day 61 is. In exactly a week, I’ll add a footnote with the code I used to generate these data, so you’ll be able to see if you got it right.

I also suspect that there might be more folks who get it with an analytics approach than by using machine learning (see this to understand the difference, plus the clue I’ve just given you) but GLHF. May the best approach win!

If you’re keen to try ML, don’t forget to do things in the right order — here’s a step-by-step guide to help you out.