Designing a mixed-methods research instrument and scoring rubric to investigate individuals' conceptions of plate tectonics
Designing a mixed-methods research instrument and scoring rubric to investigate individuals' conceptions of plate tectonics (in Qualitative inquiry in geoscience education research, Anthony D. Feig (editor) and Alison Stokes (editor))
Special Paper - Geological Society of America (March 2011) 474: 81-96
Research methods and underlying theories for research designs that integrate quantitative and qualitative approaches (i.e., mixed methods) are well documented in the field of education research. What is missing in the literature is a nuts-and-bolts description of the actual practice that goes into creating a good mixed-methods survey instrument for research in the science education domain. This paper will detail the steps involved in designing, implementing, and scoring a valid and reliable mixed-methods survey instrument. This survey instrument was designed to investigate experts' and novices' conceptual understanding of plate tectonics as inferred by their answers to a series of questions related to a modified version of a commonly used cross-section schematic published by the U.S. Geological Survey. Development of the instrument involved numerous revisions with iterative inputs from local and community-based experts. After integration of expert comments, the survey instrument was piloted to a physical science for nonscience majors course. This led to further revisions in the survey instrument to improve communication validity prior to widespread distribution. Development of scoring rubrics similarly required iterative modifications based on a thematic analysis of collected data. By outlining the steps involved in designing, validating, and analyzing this mixed-methods instrument, we believe that this paper can serve as a template for future survey instrument development. In particular, we hope to illustrate the iterative and time-intensive nature of mixed-methods inquiry, both in terms of pre-investigation design and postinvestigation analysis, and to offer our empirically based insights into the instrument and rubric development process.