1 is a Large Sample Size, Actually
There's a particular conversation that plays out on the internet about 100 every day. It goes like this:
Person 1: I've noticed that something weird is going on. I have this data, and it's not the way you would normally expect it to be. Strange!
Person 2: Well, how many data points have you got?
Person 1: 20.
Person 2: Ah, well you see this is a very small sample size. We can't draw any conclusions from your data, because there's just too little of it, so more data is needed and we have no reason to think anything strange could be going on just yet.
Person 1: I see! You are very wise: 20 is a very small sample size indeed.
What do you think about this conversation? Is Person 2 very wise and prudent to see number 20 and say it's too small do draw conclusions from?
This conversation plays out over and over, countless times. Person 2 never bothers to say what sample size would be big enough for their liking, never bothers to specify what sort of model they are working under, never bothers to look at the actual data Person 1 is providing and crunch the numbers to find out the p-value. All they knows is that no matter how many data points are presented, you can always look at them and sagely declare that it's too few and there's no reason to suspect anything odd. 20? Too small. 200? Too small. 20000? Still too small.
How big of a sample size do you actually need to tell anything? Turns out, you can't answer that question in vacuum. It completely depends on your assumptions (H0) and on what sort of data you got.
Imagine that you think that the probability to look outside your window and see a polar bear is 1 in a billion. That's your null hypothesis. You conduct one single experiment: you look outside and behold - a polar bear waving its paw at you. What do you say now, do you say: "well, 1 is a very small sample size, nothing strange is going on here, no reason to suspect my assumption was stupid"? No. You say: "under my null hypothesis there's 1 in a billion chance to produce the result I obtained. So the hypothesis can be safely rejected with a high level of confidence. (p=10^-9)". You can perform more experiments, but you don't need to: with this null hypothesis and this data, 1 is a completely sufficient sample size.
Let's look at another example. Somebody hands you a die and tells you it's fair, that is equally likely to land on any number. You roll the die 5 times, and every time it produces 1. So, do you have any reason to think there might be something fishy with it, or must you say that 5 is a small sample size and it doesn't mean anything? Well, do the arithmetic. Under that hypothesis that the die is fair, the probability that it will produce data this skewed is 1/6^4 = 1/216 = 0.004. Certainly you can say that it's still reasonably high, certainly you can roll the die a couple more times if you wish to be more sure. But notice how with the sample size of 5 we can already reject the initial assumption with a 0.4% confidence. This is better that what lots of published papers can claim. Turns out, with this data 5 is a pretty healthy sample size! (It won't always be so: if you rolled the die 10 times and the results were more mixed, perhaps 10 wouldn't be enough).
The silliest real-life example of this that I've seen was when one person recorded the outcomes of the random effect in a video game. They seems weird and unfair to him, and he recorded about 150 of them. He finished the post by declaring that of course it doesn't mean much, after all 150 is a very small sample size. When we worked through the numbers, it turned out that his data (or data at least that weird) had a 1/50000 probability of occurring under the assumption that the game was fair. His sample was more than big enough, and still he insisted that it was probably too small (without crunching any numbers, of course).
Moral of the story: there's no such thing as "small sample size" in isolation from the data and the null hypothesis. Don't go around confidently declaring that some number less than a billion is a "small sample size" without doing the hard work of calculating what number would be sufficient, without looking at the data and formulating your null hypothesis. We learn statistics so that we can notice that something is weird when there is reason to, not so we can dismiss literally anything shown to us as a coincidence and "small sample size".
Comentarios
Agregar un comentario