Appraising the Role of Luck


Appraising the Role of Luck

 

About three months ago, I was returning from a visit to CBS, and having driven with a colleague who was headed another way, decided to take the train back to Pasadena. While I was looking at a sign trying to figure out my way in Union Station, I was approached by a kind gentleman who asked if I needed help. He sent me on my way. A few minutes later, I found him again on the platform of the Gold Line. This time, we engaged in conversation, and sat together in the train. He was an American, had worked in Washington, DC, had bet it all and lost it, and gone to become an English teacher in Japan. Now he was in L.A., working in a temporary job as a dentist's assistant and looking for a more permanent job. He seemed to me clearly smart. He had, he said, fallen in hard times: no girlfriend, no permanent job, no good place to live. A far cry from his previous life as a professional in DC, he said. Then he got philosophical: "In the Far East, too much credit is given to chance. But here, too little is --our Western culture forces us to accept the blame for everything that happens to us. And that is not healthy either." At that point, he realized he had gone past his station, and ran off the train, never to be seen by me again. The man had a point. In order to learn the right lessons, it is important to know what role luck had or did not have in bringing about the events in our lives.

 

Later that day, one of the engineers in my group approached me with a question: "Alex, I am seeing something very strange. The higher the conversion rate (fraction of clicks that become a sale or other desired event) we see for a keyword, the more our calculations say we need to discount the conversion rate for the probability that it happened by chance!" Although it might seem counterintuitive, this result was correct, as can be easily seen with a couple of examples: Let's say we throw a biased coin (e.g. one with a probability distribution of heads and tails different than 50-50%) a few times, winning $1 for every head and losing $1 for every tail. If we get no heads, we know we have not been lucky at all. The coin might have been biased toward tails, but whatever its probability distribution is, we know luck did not make it seem more biased toward heads than it really is. So any role luck had, if any, must have been to show more tails than would happen on average. Thus, our estimate of the probability distribution of heads and tails will be more biased toward heads than what we actually observed, which was 0% heads. Likewise, if we get all heads, luck certainly played no role against us. If there was any luck, it was for us. Thus, for a given number of coin tosses, our estimate of the underlying probability distribution should assign a larger role to good luck (i.e. discount the observed distribution more) the more favorable the result has been. Which is what the engineer had found.

 

The two events that day were seemingly very different reflections on the same phenomenon. It is important to correctly appraise the role of luck.