I’m about to share the secret sauce of designing rigorous and compelling research projects. I call it a secret sauce, but it’s actually a tool that I was taught as an undergrad research assistant in psychology. Thus, it’s just one of those skills I learned through apprenticeship rather than formal coursework. The tool is using your hypotheses to iteratively improve your research design.

This post discusses additional analyses that can help you to establish conclusion validity. Conclusion validity means that your conclusions are reasonable given the evidence that you have, especially that you do not overstate the generalizability of your data (i.e., overstate what your sample says about the population). Two common tools to establish conclusion validity are interrater reliability, which ensures your data are as objective as possible, and demographic analyses, which examine differences within your data based on learner characteristics.

When using inferential statistics, we commonly use two paradigms to understand our results: null hypothesis significance testing (NHST, aka statistical significance) and effect sizes (aka the quantitative difference between groups). NHST determines whether there is a difference between groups, and effect sizes describe the difference between groups. While there is a movement away from NHST towards effect sizes, the current best practice typically includes both. This post explains how to interpret and calculate effect sizes.

The last post discussed inferential statistics for relational questions. This post will discuss inferential statistics for causal questions. Most commonly when asking a causal research question, an experimental design is appropriate. In this case, the purpose of inferential statistics is to determine whether the difference between groups is greater than what would be expected due to chance. For example, if you tested a new teaching method and found that students who received this new teaching method scored 5 points higher on a test than students who didn’t, it would matter if the standard deviation were 2 points or 20 points. If the standard deviation were 2 points, then the mean of the intervention group is 2.5 standard deviations higher than the other group, and the average student in the intervention group scored better than 98% of the other group. However, if the standard deviation were 20 points, then the difference in means likely isn’t greater than what would be expected due to chance.

The purpose of inferential statistics is to determine if the features of a sample (e.g., participants in a study) are representative of a population (i.e., all people belonging to a group). For example, the question “Do first-generation college students study more than other students?” must be answered with inferential statistics unless you can collect data from all college students. Thus, inferential statistics allow you to make some generalizations about your findings beyond your sample, and they are used to answer relational and causal research questions. The posts on inferential stats will cover many topics (at least two semesters of stats classes), so to break it up, this post will focus on only relational questions.

Descriptive statistics provide summaries of quantitative data, which can include quantified qualitative data. They are intended to describe the data as they are rather than draw conclusions beyond your sample of participants. Thus, they are used for descriptive research questions. Inferential statistics are needed to address relational or causal research questions.

In the scientific method, we collect data to support or refute hypotheses, not to prove or disprove them. We frame scientific research in this way because there might be factors that we are unaware of that affect the results. In human-subjects research, one reason we cannot prove or disprove hypotheses is that we use samples to represent populations rather than whole populations.

When handling quantitative data, there are a number of steps that need to be completed before you can run your first test. This post describes a basic protocol for data cleaning and tools that you can use for analysis.

Creating a Data File

When you create your data file, analytic software expects that individual participants are represented on the rows and variables are represented on the columns like in the example table below.

Surveys are prominent in quantitative educational research for measuring students’ opinions or self-reported data, such as study habits. Data collected from survey research can be any level of measurement. For example, if you ask the question “How much time did you study per week?” you could collect ordinal data (e.g., “none” “some,” or “a lot”) or ratio data (e.g., number of hours). This example also illustrates the most common criticism of survey data, which is that it can be unreliable. Even if students don’t skew their answers to be socially desirable based on what they think the researcher wants to hear, they might not be able to give an accurate response from memory. For validity, it is better to have direct measurements, but in education, this is often impractical or invasive. Evaluate the trade-offs in your work.

Operationalizing Quantitative Measurements

Your measurements are your dependent variables. In educational research, one of the dependent variables is frequently learning. To measure learning, we have to define exactly what we mean. Learning could be measured by grades on assignments, such as exams or projects, performance on standardized tests, such as concept inventories, or self-report, such as feelings of learning. All of these options are possible, though some are more defensible in scientific research (more on validity later). As a researcher, you need to operationalize, or clearly articulate in a way that can be applied to your research, what you mean when you say learning, or any of your other dependent variables. You might need to operationalize your independent variables, too. For example, instead of saying you’ll measure peer-to-peer interaction, you could operationalize these interactions as number of posts on a peer-to-peer forum and number of contributions during peer-to-peer discussions in class.