I’m about to share the secret sauce of designing rigorous and compelling research projects. I call it a secret sauce, but it’s actually a tool that I was taught as an undergrad research assistant in psychology. Thus, it’s just one of those skills I learned through apprenticeship rather than formal coursework. The tool is using your hypotheses to iteratively improve your research design.

This post discusses additional analyses that can help you to establish conclusion validity. Conclusion validity means that your conclusions are reasonable given the evidence that you have, especially that you do not overstate the generalizability of your data (i.e., overstate what your sample says about the population). Two common tools to establish conclusion validity are interrater reliability, which ensures your data are as objective as possible, and demographic analyses, which examine differences within your data based on learner characteristics.

When using inferential statistics, we commonly use two paradigms to understand our results: null hypothesis significance testing (NHST, aka statistical significance) and effect sizes (aka the quantitative difference between groups). NHST determines whether there is a difference between groups, and effect sizes describe the difference between groups. While there is a movement away from NHST towards effect sizes, the current best practice typically includes both. This post explains how to interpret and calculate effect sizes.

The last post discussed inferential statistics for relational questions. This post will discuss inferential statistics for causal questions. Most commonly when asking a causal research question, an experimental design is appropriate. In this case, the purpose of inferential statistics is to determine whether the difference between groups is greater than what would be expected due to chance. For example, if you tested a new teaching method and found that students who received this new teaching method scored 5 points higher on a test than students who didn’t, it would matter if the standard deviation were 2 points or 20 points. If the standard deviation were 2 points, then the mean of the intervention group is 2.5 standard deviations higher than the other group, and the average student in the intervention group scored better than 98% of the other group. However, if the standard deviation were 20 points, then the difference in means likely isn’t greater than what would be expected due to chance.

The purpose of inferential statistics is to determine if the features of a sample (e.g., participants in a study) are representative of a population (i.e., all people belonging to a group). For example, the question “Do first-generation college students study more than other students?” must be answered with inferential statistics unless you can collect data from all college students. Thus, inferential statistics allow you to make some generalizations about your findings beyond your sample, and they are used to answer relational and causal research questions. The posts on inferential stats will cover many topics (at least two semesters of stats classes), so to break it up, this post will focus on only relational questions.

Descriptive statistics provide summaries of quantitative data, which can include quantified qualitative data. They are intended to describe the data as they are rather than draw conclusions beyond your sample of participants. Thus, they are used for descriptive research questions. Inferential statistics are needed to address relational or causal research questions.

In the scientific method, we collect data to support or refute hypotheses, not to prove or disprove them. We frame scientific research in this way because there might be factors that we are unaware of that affect the results. In human-subjects research, one reason we cannot prove or disprove hypotheses is that we use samples to represent populations rather than whole populations.