Notes from a fellow experimentaholic

One of the world's most famous experimentaholics was the British polymath Sir Francis Galton. Galton had a fetish for measuring, and he measured everything, from people's heights to their attractiveness to their intelligence to their fingertips. It was galton who first came up with the notion of regression to the mean, a common occurance such that people who do well on a test tend to do worse the second time one takes a test, and those who do really bad improve on the second. This has to do with the fact that people who do very well are smart, plus had a little luck. Test them again, and they may not be so lucky. Galton came up with the correlation coefficient and other handy statistical tools. He also was the father of eugenics, something perhaps not to be so proud about.

However, in leafing through issues of the journal Nature from a hundred odd years ago, I found this curious little solution to a common problem: the fact that one can have one's cake and eat it too, but not having the appetite to finish the whole thing. The problem is that those cut parts that lay exposed to air get stale. A common problem. Well, apparently, a "F.G." devised the solution described in the following Letters to Nature section. On further investigation, I discovered that our curious cake consumer F.G. is none other than Francis Galton himself!



This goes to show - it was easier back in the day to get something published in Nature. If only that were the case today!

Principles of indifference

It’s happened to you, I’m sure, as it has happened to me. You go about your life, working on your computer, editing files, processing data. You don’t get up in the morning thinking that today will be the day. The day that you fire up your computer, with all the hopes of getting that paper written, and instead are faced with the Blue Screen of Death. A message blinks “Can not find hard drive” or something equally vague and frightening.

It happened to me one winter evening in Ann Arbor, Michigan, where I was a post-doc. I hadn’t backed up any of my files for about six months. I restarted my Mac after it wouldn’t connect to the internet and . . . it never started again. Blank screen. I took it to dealers, tech support…nothing. Someone offered $5,000 to try and salvage the data off the hard drive, but I figured it wasn’t worth it. I could start from scratch, having learned an important lesson. But I lost hundreds of hours of time rewriting papers, reanalyzing data, and piecing together all that I had lost. Fortunately, some of the stuff was recoverable – colleagues who had drafts of manuscripts emailed me the files. But a lot was lost forever.

I ran into our friendly computer tech support director Harold the other day. It is because of Harold’s relentless campaign on backing up data that inspired me to burn all my work onto a CD every week. But it makes me wonder about the psychology involved in backing up one’s data. Even after my experience, I still don’t always back up my data. Some Fridays I just feel like going home – “I’ll do it tomorrow.” And I suppose one of these tomorrow’s I will fire up the machine and get that screen of death.

And who will I call? Harold. And often, what can Harold do? Nothing. This is not because Harold isn’t a great IT person – it is because Harold isn’t Zeus and can not pull a dues ex machina, coming down to earth and intervening with the ones and zeros that constitute the information on the hard drive.

I recently wrote a post about LaPlace’s 1822 Rule of Succession. Gigerenzer talks about a related notion to the Rule of Succession known as the Principle of Indifference. It basically has to do with what probability you assign to the possibility of an event given no prior knowledge. Recall my discussion of HIV infection – you can apply Bayes Theorem to situations in which you know the base rate – or frequency of – an prior probability, such as the proportion of people of a certain population that is infected with HIV. However, when you don’t know this prior probability, you are faced with a situation of indifference in which you assign equal probabilities to the likelihood of an event.

Electronics tend to follow an inverse power law in terms of their life-span. That is, take a hundred iPods. Once they leave the factory, 100 of them work. A month later, 99 work (i.e., you may have dropped it in the toilet by accident). A year later 80 of them work, and so on, until only 1 is still working some years down the line. And this is okay, because electronics are replaced with newer gadgets. But it is the probability of failure I am concerned with here: All electronic devices will someday fail. But why are we so confident that on any given time that you shut down your machine without backing up your data, that tomorrow when you start it, that the machine will in fact work rather than result in the blue screen of death.

I think the answer has to do with the law of succession, the same problem faced by Adam and Eve in the Garden of Eden. When you first buy your computer and turn it on, it is as if your computer gives you a white marble which you place in a box. Already in the box is a Blue Marble of Death, because you don’t know whether or not it will ever turn on again as it could be a lemon. Every time you start your computer and it works, your computer gives you another white marble from its box which you place in yours. Eventually your box will contain a lot of white marbles, representing your confidence in your computer.

But your computer doesn’t see it this way. Your computer is doing something different. It starts off with a lot of white marbles in its box, and one blue marble of death. And every time you turn it on, it selects a white marble from the box, and doesn’t replace it – it gives it to you, improving your confidence in it. So it may begin with a thousand white marbles and one blue marble of death. And the next time you turn it on, it selects a marble from its box. With high probability it is a white marble. But with every successive selection from this box, the probability increases and increases that the blue marble will be chosen until that day when suddenly, and unexpectedly, you turn on the computer, with a very high confidence in it working, while simultaneously, the computer selects the blue marble.

The result? You’re screwed.

The following figure provides a depiction of the expected probability that you have that your computer will start given a certain number of prior starts (by the rule of succession), the expected probability that your computer will select the blue ball of death randomly without replacement on any given start up. And then we encounter the real world. This hypothetical computer selected the white ball of hope all the way up to reboot 77. Then, on reboot 78, the computer selected the Blue Ball of Death, and CRASH! There went your work (unless you backed it up).

Do we reason this way? People are bad as assessing the probability and success and failure, which is why do many people gamble despite the odds. Hope springs eternal in the Garden of Eden, in Atlantic City, and in your office. But even with all the hope in the world, there comes a day when your luck runs out. It could happen on the 78^th time, on the 788^th time, or the 7888^th time. But it WILL happen. And you can either lose three hours of your life, or three years.

Your choice.

Hope springs eternal in the Garden of Eden

I was speaking with a friend yesterday about relationships. The conversation we were having was about the fact that just because your last relationship was bad and ended bitterly, how does that influence how you think about future relationships? Not all relationships end in misery, right? ("Just mine" you might be thinking...)

I suspect that the solution to this problem can be found,as always, in an unexpected place: probability theory. There is an appropriately-named statistical problem discussed by the French mathematician Pierre Simon LaPlace (1749-1827) called the “First Night in Paradise." Basically it goes like this:

It is the night after Adam and Eve’s first day in paradise. Together, they watch the sun rise and illuminate the marvelous trees, flowers, and birds. At some point, the air gets cooler, and the sun sank below the horizon. Will it stay dark forever? Adam and Eve wonder, What is the probability that the sun would rise again tomorrow?

Whether Adam and Eve were truly discussing this statistical conundrum, as opposed to addressing less cerebral concerns of a non-probabilistic nature, is besides the point. What is the probability that the sun will rise again?

The classic answer is that if Adam and Eve had never seen the sun rising, they would assign equal probabilities to both outcomes. Gigerenzer, in his book Calculated Risks, likens it to placing a white marble (the sun will rise again) and a black marble (the sun will not) into a bag, and picking one randomly. However, they did witness the sun rise once that morning when they woke up, so they place another white marble in the bag, which makes the probability of the sun rising the next day .66.

(Interestingly, as an aside, one might ask the question, "What was Adam's subjective assessment of the probability that Eve would still be there the next morning?" After all, she did simply just appear there, and who knows? God might just be teasing him and yank her out of the garden.)

As we, know, Adam and Eve woke from that first night in paradise to another sunny day in paradise. But what is the probability now, given two sunrises, that the sun would rise once again the next day? The answer is simple. They add another white marble to the bag, making the probability go up to .75. And the sun rises again and another marble goes in the bag, and so on. This is known as the rule of succession. Introduced by LaPlace, it has a simple formula:

(n+1)/(n+2)

If you are like me, and are thirty odd years old, the subjective probability that I assign to the sun rising tomorrow is quite high. Specifically it is

(365*30 + 1) / (365*30 + 2) = 10951 / 10952 = .99990869247626004. Pretty good odds. I am, in fact, relatively convinced that the sun also rises tomorrow.

But what about relationships? Does the law of succession apply here? What is the probability, given a certain number of previous relationships, that you believe that your next one is headed to splitsville? It is just like adding white balls representing sucess and black balls representing failure to a jar and selecting one.

I asked ten friends and colleagues today how many serious relationships they had been in over the course of their lives. The median value of that number was 7 – your typical friend of mine has been in 7 serious relationships (which I defined as lasting over 6 months).

What is the average person’s subjective estimate of the probability of the next relationship ending given the rule of succession? Quite simple: (7 + 1) / (7 + 2) = 8 / 9 = .888.

So is it the case that with a greater number of failed relationships, does one use the law of succession to determine whether one’s next relationship is going to fail? Maybe. This would be an interesting study to conduct – the correlation between the number of failed relationships one has had with one’s pessimism about the probability that one’s next relationship is also going to fail. Of course there are confounds up the wazoo, but it would be a fun and simple study to do on a larger scale than my ten friends.

But there is hope here. While you might suspect that your next relationship will fail with a probability of .88, that means that you also have a probability of .12 that the next one will work out. And I like that 12 percent – I might like to call it a “Hope Factor.” Sure, the hope factor diminishes after every successive failure, but by definition, this means that one can never lose hope entirely – the law of succession may asymptote very near zero, but it never quite reaches it, due to the fact that the denominator is always one greater than the numerator. Below is a figure of the relationship between the hope and despair factors by the number of failed relationships. You can see how the region of despair increases with every sucessive failed relationship, but that region of hope never quite goes away.

As they say: Hope springs eternal! And perhaps this is why it was the last evil remaining in Pandora's Box.

The possible and the probable

I spent the last few days at the National Science Foundation reviewing grants. Truth be told - you know you're nerdy when you walk into a restaurant after having forgotten to remove one's NSF temporary ID badge reading "Hello, my name is experimentaholic." and are reminded of this fact by an attractive yet not-so-nerdy restaurant hostess. Reviewing grant applications, in and of itself, in not very much fun, because I simply don't like being the one who decides who gets funded and who doesn't. I think everyone should get funded, myself included! But this is not the state of affairs - only some get funded, others get the apology letter. And I've gotten my share of those thin ones.

Which brings me to the topic of the probable and the possible. What does probability mean? For instance, these people who wrote grants submit them, and there is a certain probability that they will get funded, which is based on how much money NSF has to dish out. They also have a certain expectation of the probability that their grant will get funded, which may differ from that which is actually possible given the availability of funds.

But what does it mean to expect an event with a certain probability? Like dying on a plane crash. The probability of this event is quite low - something on the order of one in eight million. So we get on the plane, and most of the time, get off the plane. But when you get on that one plane that explodes, and you ARE that one in eight million, the probability is not quite 1 but certainly close to it. But what does uncertainty mean, at least at the level of a psychological construct?

I know that there is a lot of evidence on this - and most of it points to the fact that people are particularly bad at assessing probabilities and changing their behaviors based upon these assessments. People drove after 9/11 out of fear of flying, and this increase of traffic and the resulting accidents that ensued killed more people than had died on the planes of 9/11. True! I buy a lottery ticket when the powerball is in the hundreds of millions, realizing full well that the probability of actually winning is the same as correctly guessing a random a number between zero and 120,526,770.

I am at the moment in a coffee shop, it is a Friday night, and two students next to me are discussing Spinoza, and specifically free will. This makes me wonder about our free will in relation to our assessment - and flawed assessments - of probabilities. I have always dreaded flying, even knowing that the probability of dying is the same probability of guessing a number correctly between zero and eight million. But what if I am just lucky (or unlucky) that day? Everyone who has ever died in a plane crash has gotten on that plane imagining that the probability of a crash was one in eight million. Which is fine, if you're a member of that elite group of eight million. But if you're not, not.

One way to think about it perhaps is actuarially. When you get on a plane, you have to think of the degree to which taking this trip will actually reduce your life. A related situation is when you buy a lottery ticket. A lottery ticket is really worth what you paid for it, plus whatever the probability of winning the ticket multiplied by the reward. Imagine the senario of the powerball. The ticket costs $1. The jackpot is 100 million But the real value of the ticket is $1 + $100,000,000 X 1/120,526,770. Which is $1.82. Of course, afterwards, any given ticket is either worth $0 or $100,000,000...but in the seconds before the balls drop, it is worth $1.82. What about one's life? The probability of dying in a plane crash is 1 in 8 million. As a thirty year old, I expect (or I should say hope) to live another 30 years at least. More would be better. 30 years comes out to 946,080,000 seconds. But by getting on that plane I am reducing my life by 946,080,000 seconds X 1/8000000 or 118.26 seconds. Of course, after the plane lands or crashes, that amount changes to 0 seconds or 946,080,000 seconds. Is it worth it? Does this way of looking at the problem solve the problem, or just create new ones?

Which brings me back to free will. Does the fact that we live in uncertain world make it such that our free will is compromised? I don't know. How does our inability to reason about probabilities affect our decisions? Probably in some domains, probably not in others. Some people are notoriously bad at making comparative judgments of probabilities and possibilities - those who drove rather than fly after 9/11, who are now in the grave or scattered to the winds because their dread risk overcame their rational capacities.

In this sense, statistical probability and mental probabilities must be related, but they are not homologies. Keyes wrote something about this, on personal probability, but I confess I haven't read it, yet. Perhaps I'll read it someday. It's both possible and probable.

I'll have some love with that microscopic $9.95 piece of salmon

A student of mine once did an experiment using this web site called "Craig's List Missed Connections." Basically, it is a site in which people post notes to the universe that say things like, "I saw you walking down the street and you glanced at me. You were wearing a red hat and a green scarf...coffee?" And so if you think you are this person you can email this individual back and hopefully begin a life of love and lust together. That, or realize you have absolutely nothing to discuss over coffee.

Recently I was reading this site (perhaps perversely to see if someone posted something about that guy in the coffee shop cursing SPSS under his breath when the p value for his most recent experiment is greater than .05.) One thing I’ve noticed in passing through these listings is how there are many posts about missed connections at Whole Foods - the local hip foodstore in my neighborhood.

But this has gotten me into wondering…what is it about Whole Foods that inspires people into longing? Is it something about the air of an organic, expensive grocery store that makes people lust after strangers? Is it that only well-to-do people who can afford well-to-do people’s clothing and well-to-do people’s makeup and well-to-do people’s hair styling can afford shopping at the place? Or is it a unique place in this world - a world in which we communicate so often over the internet that we no longer know how to approach a stranger and say hello, but have to go back to our internet world and post a little "I think I saw you looking at me, but I am not sure and I'm oh so insecure!

(Here I would like to add that such people should read the excellent paper by Clark and Hatfield (1989) in which they had experimenters go around Florida State University's campus and randomly ask strangers if they would go out on a date with them that night. Something like 60 percent said yes! Could be something unique to Floridians, though.)

I wanted first to determine if there was an effect at all. I did a quick search and found that in the past 45 days on craigslist there were none less than 37 postings of missed connections at whole foods. But right next door to whole foods is a regular old supermarket known as Superfresh. How many quick glances and subtle smiles have been exchanged there? Only 8. And this includes all the Superfreshes in the city.

I figured I would try different cities. Let’s try Ann Arbor, Michigan, which I know has a Whole Foods because I used to shop there. They also have Kroeger’s, the Ann Arbor equivalent to Superfresh. 1 hit for whole foods, none for Kroeger. I tried Chicago…5 for whole foods, 1 for Dominick’s (there are dozens of Dominicks out there)

What is the probability that this could be due to chance? I'll stick to the Philadelphia one because it has the most data. Imagine that it were purely due to chance that sightings would occur at Whole Foods and at Super Fresh. Then the expected probability would be .50 - a fifty-fifty chance. What we observe is 8/37 or a 0.21 percent sightings at Superfresh. What are the odds that this is due to chance? Luckily, we have the good old binomial test to tell us the odds. The actual probability that this could be due to chance is 0.000007687045. That's a very small odds! There is something fishy going on, and I don't just mean the salmon I ate for dinner. That I bought at Whole Foods.

I suggest you experimentaholics and statistiphiles try this in your own cities. And if I see a post for that guy talking about Chi square at the produce section, I’ll know I’ve finally been missed.

Will Universal HIV testing lead to a suicide pandemic?

I recently read that the CDC has recommended that all people be tested for HIV as part of a routine check up with one’s doctor. You can read about it here. At first I though this was a great idea, that may have important implications for reducing a preventable and incurable disease. But then I read this book, Calculated Risks, by the psychologist Gert Gigerenzer, who is associated with the Max Planck Institute for Human Development in Berlin, Germany. I've always wanted to work with the guy, but he has some kind of beef with this other guy I like named Tom Bayes.

Gigerenzer discusses the fact that people are particularly bad at reasoning about probabilities, particularly in respect to understanding base rates. Now, this has long been known since the pioneering studies by Daniel Kanneman and Amos Tversky in the 1970s, but Gigerenzer has followed up on these ideas in new domains.

I recently posed the following scenario to about ten friends:

Imagine you go to your doctor, who informs you that you should take an HIV test as a routine part of your checkup. You are a heterosexual who has been in a few relationships over the past five years, you have never used IV drugs, and you have no other risk factors for HIV. The doctor informs you that the probability of someone like you actually having HIV are quite low, something in the order of 1 in 10,000. He also tells you that the test is 99.99 percent accurate in providing a negative test when the person is actually negative. A week later, you receive a call from the doctor informing you that the test came back positive, indicating that you are infected with HIV.

What would you do? Please circle all that apply.

1. Commit suicide
2. Ask to be tested again
3. Inform your current sexual partner
4. Do nothing – forget about it.

Given the facts as stated above, what do you believe is the probability that given a positive test, that you actually have acquired HIV?

1. 0%
2. 10%
3. 30%
4. 50%
5. 70%
6. 90%
7. 99%
8. 99.99%

We’ll get to the results of this informal study in a minute. But first, what actually is the probability? The vast majority of people would likely say 99.99%, because the test is 99.99% accurate. However, this is to forget the fact that the actual prevalence rate of the disease is extremely low – 1 in 10,000 or .001 percent of the population in question, is actually HIV positive. Why does this matter?

Well, it matters because of something known as Bayes Theorem. Bayes Theorem is a statistical solution to what is know as the inverse probability problem. The idea is this: The probability of event A conditional on event B is different than the probability of event B conditional on event A. So, the probability of a person being HIV positive given a positive HIV test is different from the probability of having a positive HIV test given the probability of being HIV positive. Bayes Theorum is a mathematical expression of the relationship among these events.

Think of it this way. The HIV test is highly accurate. 99.99 percent of the time, it will give the right answer: HIV + given HIV +, or HIV – given HIV -. But .01 percent of the time, it will give the wrong answer: HIV + given HIV – or HIV – given HIV +.

Gigerenzer argues that one should think of it in terms of frequencies rather than probabilities. He poses the problem this way: 10,000 people go in to have the test. 1 person will test positive and actually be positive. But 1 person (.01 % of 10.000 = 1) will test positive and actually be negative. And 9,998 will test negative and actually be negative. Hence, if you are from a population in which the incidence rate is extremely low, the probability that the test is inaccurate is quite high. In this particular case, it is .50. You have a 50/50 chance of actually being positive if you have a positive result and are from the population in question.

The problem is that even a test with a high degree of accuracy, if used many many times, will once in a while create a false positives and false negatives just by chance. Blood samples may get the wrong identification number, or samples could get polluted, for instance. As much as they might not like to admit it, doctors are not infallible, and neither are the technicians who actually analyze the samples. I’ve made mistakes in analyzing my own data, as my reviewers often, and often unkindly, point out.

But people don’t often think in Bayesian terms, and it is likely that the only time you encounter Bayes Theorem is in that boring college statistics class (like mine), in which it will quickly go in one ear and out the other. (I can hear my students thinking – “Inverse probability – whatever”). But here is the scary thing, from page 121 of Calculated Risks: “At a conference on AIDS held in 1987, former senator Lawton Chiles of Florida reported that of 22 blood donors in Florida who were notified that they had tested HIV-positive with ELIZA [a common test for HIV], 7 had committed suicide…The medical text documenting this tragedy many years later informed the reader that even the results of both AIDS tests (Eliza and Western blot) are positive, the chances are only 50-50 that the individual is infected.”

This blew my mind! What percentage of those 7 people could have been spared their fate by understanding Bayes Theorem and inverse probability? The thing that scares me the most is the question of whether medical professionals who inform their patients of their HIV status actually know how to calculate a Bayesian probability problem themselves, or are trained in this fallacy of the test so that they can accurately inform their patients. To find this out, I informally asked a variety of friends who hold MDs the same question that I asked my friends – the hypothetical senario. None of them gave the correct answer on the spot (then again, I was asking them at parties so they didn't have calculators on hand). However, several of them answered the question partially correctly – one mentioned that you have to take the overall incidence rate into question. But several gave the completely wrong answer – 99.99%! One mentioned, “I remember in medical school that there is some formula that one can use to figure this out, but I suck at statistics and have no idea how to calculate it.”

So this has gotten me thinking: Should we as a society simply mandate generalized HIV testing? Should we not at least consider the fact that people – even doctors, even professors, even statistics professors – are notoriously bad at calculating probabilities, and specifically Bayesian inverse probability problems? How can we better inform patients about their actual probabilities of having diseases?

And how vast is the problem? What about other diagnostic tests? What about in other domains in which people must make judgments?

Returning to my friends, what were their answers?

Well, on the second part, 7 said 99.99%, 1 said 99%, 1 said 90% and only one said 50%. On the first part, 2 said they would commit suicide. All but 2 said that they would get retested, and only three said they would tell their current sexual partner (I would lament about this for a while, but the fact is that this evidence isn’t fair as I didn’t acquire information as to whether any of them actually have a current sexual partner, but this is an entirely different issue that wouldn’t fit on the margin of this page.)

So let's pretend that my informal sample was a real sample (big leap of faith considering the various raffish and seedy characters I consider my friends) and say that 20% of people would commit suicide if they received a positive HIV test. Imagine we test all 300,000,000 people in the U.S. That's a lot of needles! Just by chance alone, 3,000,000 would test positive and actually be negative. Of those, 600,000 (20% of 3,000,000) would kill themselves. In 2002, 16,000 people in the U.S. perished due to the AIDS epidemic. We could in a single year increase the death rate due to AIDS by some 375.00%. Simply by having universal testing. And the really not funny thing is that 1,500,000 of these people are not even infected with HIV, but were victims of our reliance and confidence in the medical establishment.

If the CDC goes through with this plan, I hope they have better statistics professors than I teaching health care providers how to talk Bayesian with their patients.

I'm a experimentaholic

I guess the first step in any 12 step program is to admit one has a problem. And so I admit it: My name is Sean, and I am a experimentaholic.

I don't know when or where my attachment to experiments began. But slowly, my life spiraled into a digression (regression) in which I couldn't help but think about the world other than in experimental terms. I was at rock bottom when I woke up this morning with Roger E. Kirk's Experimental Design (3rd edition) open on my chest. On my nightstand, Understanding Factor Analysis. And on my kitchen table, not one, but all three volumes of Kendall's Advanced Theory of Statistics.

At that point I knew I had a problem, and I knew I had to deal with it. So I figured I would come here to move through the stages of recovery by shedding my thoughts, by getting it out in the open.

This will be my research and thinking blog. I am sure it will never receive a single hit. So it will be my journal. I will try and post all sorts of ideas - some of a statistical nature, others of a less serious nature. It will be my academic blog.

It is Stone Wall Inn for dorks. It is the 19th Amendment for geeks. I am trying to liberate mankind, and summarily bound humanity within a 99% confidence interval.

And hopefully have some fun being drunk on experiments in the process.