If our intuitions are unreliable …

by Greg Krehbiel on 8 January 2013

… then what becomes of the “B.S. meter”?

A few months ago I read an article in Scientific American about a push to make decisions based on lots and lots of data. That is, huge computers making calculations on tons of information. Way more than a mere mortal could process.

Remember the commercials from a few years ago about how marketers would use transactional data for product placement in stores — e.g., people who bought some particular kind of cheese were more valuable customers …. So, imagine people using the same sort of data analysis for city planning, or crime prevention, or predicting financial problems.

The results would often be counter-intuitive. The experienced beat cop may have learned to watch Joe’s Liquor Store on a Friday evening, but the data geek tells him he’s better off watching the pay phone on 4th and E. And he may be right.

The programs running off this data would often make conclusions that would challenge our intuitions. It may be that on a certain road raising the speed limit will save lives. Or the computer may discover that people who are about to bomb an airplane always change their out of office message the day before.

John and I talked about this briefly this past weekend, and my reaction to this is, first, that it could be really cool, second, that the programs probably won’t be as smart as the advocates will say they are, but third — and this is what I want to speak about here — if we were to accept a situation where we were taking orders from these programs, in some ways we’d be very similar to a religious fundamentalist.

On that last point, a problem with fundamentalists is that they learn to suppress their B.S. meter in deference to a higher authority. With most people, if you told them to put the fox in charge of the hen house, they’d say you were crazy and that would be the end of it. But the people we’re talking about will think, Gee, I don’t know, the Wise Oracle knows things I don’t know, who am I to insist on my private judgment, and so on.

The thing is, that’s a perfectly reasonable conclusion if the authority is reliable and is more likely to know the truth than the individual is. So, for example, it’s natural to think that when the Nazis come to your door and ask if there are any Jews hiding in the closet, it’s okay to lie. I don’t agree with this position, but it’s not unreasonable to take the stance that God will honor truth-telling, and that it’s better in the long run to always tell the truth — even if it seems like the wrong thing to do at the time.

IOW, the believer submits his intuition to the dictates of somebody who knows better.

The believer in Big Data is doing the same thing, because sometimes the data says to do something that doesn’t seem to make sense.

In both cases the process can be self-correcting. The religious fundamentalist will eventually learn that adding “in Jesus’ name” doesn’t guarantee that he’ll get what he asks in prayer, and the Big Data guy will go back and tinker with the program when its predictions don’t work. But lots of really stupid things can happen in the meanwhile, and that’s what I think will happen with increasing reliance on Big Data.

Human intuition has an important role to play. It’s what tells us that Jesus was exaggerating when he spoke about plucking out eyes and cutting off hands, and it’s what makes us go back and check the program when it says that doctors are more likely to prescribe the right medicine if they have less data about the patient.

The fundamentalist and the Big Data geek are extreme cases, but nobody is immune to this problem. Think of all the dumb things we’ve done because “experts” said it was the right thing to do. (Dr. Spock’s advice on child-rearing comes to mind.)

Nevertheless, my concern is that “Big Data” will have some really startling successes, we’ll start to trust it at the expense of human intuition, and then we’ll do something really, really stupid — because the computer said so.