3 Avoidable Statistical Mistakes Audrey Snowden, Ph.D.

Marketing research is grounded in the scientific method: answering questions by generating a priori hypotheses, collecting data to test hypotheses, and analyzing data to draw conclusions. Adhering to the rules of the scientific method is important to ensure that results are valid and unbiased.

 Sometimes marketing researchers are tempted to use undesirable methods, like conducting many single significance tests, performing statistical tests without hypotheses, and rerunning statistical tests until desired results are discovered. Unfortunately, engaging in these methods has unintended, detrimental consequences: namely, an increase in Type I Error. What is Type I Error? Type I Error is equivalent to a false positive. It occurs when we believe we have found a significant difference when there isn’t one. Typically, researchers set Alpha (α) to .05, limiting the chance of making a Type I Error to 5%. Often, researchers do not realize that Type I Error is influenced by the number of statistical tests conducted. Specifically, the probability of making a Type I Error increases with every additional statistical test (correlation, regression, t-tests, etc.) that is performed. This is known as the Familywise Error Rate. For example, if we conduct one significance test, we have a 5% chance of finding a significant difference when there isn’t one—because we have set α = .05. If we conduct three significance tests with an alpha equal to .05, the probability that we will make at least one Type I Error increases to 14.3%1. If we examine ten significance tests, the probability that we will find a significant difference when there isn’t one rises to 40%.

 Contact Us 604 Avenue H East Arlington, Texas 76011 United States 817-640-6166