Concept inventories are relatively new types of diagnostic instruments intended to measure student learning. Concept inventories exist for astronomy, biology, chemistry, engineering, fluid mechanics, geology, and physics. None is yet available for oceanography, and our work to construct the Introduction to Oceanography Concept Inventory Survey (IO-CIS) serves to help fill this gap. In this paper, we discuss the end members of a spectrum of test types from traditional aptitude tests to traditional achievement tests, and we suggest that concept inventory tests are hybrids of the two. An IO-CIS is constructed and validated for a specific Introduction to Oceanography course taught at the University of Colorado at Boulder. The construction of the IO-CIS is divided into development and evaluation phases, in which the development phases primarily utilize qualitative methods, and the evaluation phase uses quantitative methods. We present an efficient approach to developing a concept inventory test for a single course in the span of a single semester. The focus of this paper is on the development phases, the qualitative methods used, and the issue of validity. We discuss how classical test theory can be used to build a validity argument during test construction of the IO-CIS.