How many responses for statistical significance

It also explains how to determine the survey statistical confidence that can be placed in results. In my conference presentations, I am inevitably asked about typical response rates and survey statistical confidence — usually before I get to that part of my presentation.

Questions range from:. I think they recalled the Central Limit Theorem from a statistics class. If only life — and statistics — were so simple.

So, we face a trade-off between cost and accuracy. How big a trade-off? That depends upon how we administered the survey — postal mail, webform, telephone, etc.

Get our Excel-based calculator. Use it to determine sample size needed for some statistical accuracy or to gauge accuracy after the survey has been completed. The statistical equations here are a bit daunting. The horizontal axis shows the population. The vertical axis show the percentage of the population from whom we have a response. This is not the response rate. The response rate is the percentage of those receiving an invitation who respond — an important critical distinction.

The chart shows seven lines that depict seven levels of survey accuracy. The first one is the horizontal line at the top. We have population parameters. Of course, that will likely never happen. Say you have a population of , and you sent a survey invitation to people. Half of those responded.

Conversely, if we have an accuracy goal for the survey project, we can use this chart to determine the number of responses needed. Find those coordinates on the chart. By applying an estimate of our response rate, we can then determine the number of survey invitations we must send out, which is our sample size.

When we actually conduct our survey and analyze the results, we will then know something about the variance in the responses. The confidence statistic incorporates the variance found in each survey question and can be calculated for each survey question.

Technically, the interval tells us where the mean of repeated survey samples would fall. If you pick a customer at random, chances are higher that they are pretty far from the average. To summarize, the important thing to understand is that the greater the variation in the underlying population, the larger the sampling error.

Redman advises that you should plot your data and make pictures like these when you analyze the data. The graphs will help you get a feel for variation, the sampling error, and, in turn, the statistical significance. The significance level is an expression of how rare your results are, under the assumption that the null hypothesis is true. Setting a target and interpreting p-values can be dauntingly complex.

Redman says it depends a lot on what you are analyzing. Then you collect your data, plot the results, and calculate statistics, including the p-value, which incorporates variation and the sample size. If you get a p-value lower than your target, then you reject the null hypothesis in favor of the alternative.

Again, this means the probability is small that your results were due solely to chance. There is also a formula in Microsoft Excel and a number of other online tools that will calculate it for you. For example, if a manager runs a pricing study to understand how best to price a new product, he will calculate the statistical significance — with the help of an analyst, most likely — so that he knows whether the findings should affect the final price.

If the p-value comes in at 0. But what if the difference were only a few cents? But even if it had a significance level of 0. The table below displays the necessary sample size for different sized populations and margin of errors. As you can see, even when a population is large, researchers can often understand the entire group with about 1, respondents. Sample size calculations tell you how many people you need to complete your survey.

What they do not tell you, however, is how many people you need to invite to your survey. To find that number, you need to consider the response rate.

All you have to do is take the number of respondents you need, divide by your expected response rate, and multiple by Sample size formulas are based on probability sampling techniques—methods that randomly select people from the population to participate in a survey. For most market surveys and academic studies, however, researchers do not use probability sampling methods.

Instead they use a mix of convenience and purposive sampling methods that we refer to as controlled sampling. When surveys and descriptive studies are based on controlled sampling methods, how should researchers calculate sample size? This often translates to a sample of about 1, to 2, people. More participants in a study will always be better, but these numbers are a useful rule of thumb for researchers seeking to find out how many participants they need to sample.

If you look online, you will find many sources with information for calculating sample size when conducting a survey, but fewer resources for calculating sample size when conducting an experiment. Experiments involve randomly assigning people to different conditions and manipulating variables in order to determine a cause-and-effect relationship.

The reason why sample size calculators for experiments are hard to find is simple: experiments are complex and sample size calculations depend on several factors. In order to begin a sample size calculation, you need to know three things.

The significance level represents how sure you want to be that your results are not due to chance. A significance level of. Statistical tests are only useful when they have enough power to detect an effect if one actually exists. The final piece of information you need is the minimum effect size, or difference between groups, you are interested in.

Determining the minimum effect size you are interested in requires some thought about your goals and the potential impact on your business. They set their significance level at.

In addition, the team determines that the minimum response rate difference between groups that they are interested in is 7.