Motivation 

To address a problematic “imbalance between the number of quantitative and qualitative articles published in highly ranked research journals by providing guidelines for the design, implementation, and reporting of qualitative research.” They also discuss the risks and benefits of a highly ranked research journal (Computers & Education) recommending guidelines to be used, albeit flexibly, in qualitative research.

Qualitative or Quantitative Methodology and Data

The paper starts by addressing common misconceptions about when it is appropriate to mix-and-match qualitative and quantitative. They define qualitative methodology as hermeneutic or interpretivist and based on a belief in the validity of multiple culturally-defined interpretations of multiple realities. Therefore, qualitative methodology is incompatible with quantitative methodology, which they define as objectivist or empiricist and based on a belief in the validity of one true explanation of one objective reality. Within each of these methodologies, however, data collection methods, instruments, and analysis can be both qualitative (i.e., non-numeric) or quantitative (i.e., numeric) and mixed-and-matched at will. Much more detail about these concepts and their relationships can be found at Twining’s blog post that extends their very useful Table 1.

The papers’ guidelines are for conducting and reporting on research that uses a qualitative methodology, not non-numeric data collection and analysis. Given that qualitative research’s goal is to interpret data within some of many possible, and sometimes divergent, theoretical, methodological, and value assumptions, creating a single set of criteria for evaluating the quality of qualitative research is problematic. Therefore, the authors recommend that the guidelines are not applied rigidly. Instead their goal is to remind researchers, editors, and reviewers of features that can be used to evaluate their own and other’s research.

Guidelines for Qualitative Research Design and Reporting

Theoretical stance: All research is based on theory and an ontological and epistemological position. The research’s theoretical, ontological, and epistemological positions should be made explicit, as should the alignment between these positions and the research goals, design, and data collection methods, instruments, and analyses, when appropriate.

Methodology:  The research paradigm/methodology should be clearly stated in relation to the research objectives and questions. The research objectives and questions should be informed by a review of relevant theory and literature. For research that includes digital technology in education, it is important to include up-to-date literature for areas that change quickly, such as the use of a new tool. In addition, for more stable areas, such as change management, research that is decades old might still be highly relevant.

Design: The research design should be described as should the justification for how the data collection methods and analyses are appropriate for linking the research objectives and intended conclusions. In addition, the ethical implications of the research should be explained, especially when the researchers work closely with participants.

Data collection methods and instruments: The methods should be justified and described in terms of how they address the research objectives, their feasibility, their limitations, and their implementation (including any changes that occurred during implementation). The participants’s characteristics and process for selecting them should be richly described and aligned to the research objectives (e.g., many facets of the phenomenon of interest should be represented). Likewise, the instruments used, whether non-numeric or numeric, and their appropriateness for answering the research questions should be described in detail, with verbatim copies when possible. In addition, possible implications for experimenter bias, use of devices like video cameras, and interactions with the researchers should be discussed.

Analysis: The approach to data analysis should be described and justified in terms of research objective, data collection methods, and data collected. The techniques used to interpret the data, including iterations of analysis, should be described in detail. The purpose of data analysis is not only to describe the data, but to make interpretations and draw conclusions based on theory and the research objectives. Moreover, alternative interpretations should be discussed, and examples from the data should be included for all interpretations. Connections to the literature, limitations, and an explanation of the research’s relevance should be explicit.

The authors note that qualitative research reports must include lengthier descriptions and justifications for methodology, design, data collection methods, and analysis decisions than quantitative research because many of the decisions in quantitative research are predetermined, and, thus, the justifications are assumed.

Why this is important

Many researchers learn only one type of methodology, qualitative or quantitative, based on the ontological and epistemological beliefs that are prevalent in their fields. Each methodology can provide value, but many high ranking journals have a publication bias towards quantitative research. Furthermore, many quantitative reviewers and readers are not trained to evaluate qualitative research. These guidelines highlight, for both authors and readers, how to ensure the quality of reporting on qualitative research.

Further reading with helpful recommendations in Table 1 starting p.34 and for mixed-methods in Table 3 starting p.41: Levitt, H. M., Bamberg, M., Creswell, J. W., Frost, D. M., Josselson, R., & Suárez-Orozco, C. (2018). Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 26-46.

Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C. C. (2017). Some guidance on conducting and reporting qualitative studies. Computers & Education, 106, A1-A9.

For more information about the article summary series or more article summary posts, visit the article summary series introduction.