Impact of a Domain-General Module on Emergence for Learning the Concept of Diffusion Authors: Measuring In-Process LearningAuthors: Besterfield-Sacre, Mary E.;Shuman, Larry J. As engineering education moves deeper into the assessment of student learning, the focus on measuring the various processes that students use to achieve learning outcomes becomes important. Process measurements are preferred for addressing engineering student cognition and the dimensions of learning; and, in theory, are correlated with the outcome or artifact of the learning. Further , many engineering outcomes are actually more process rather than product oriented, such as team work and problem solving, or are taught as a means to provide strong resultant outcomes, such as design. We suggest that for process measurement to be rigorous , usable assessment tools aimed at capturing in-process learning must be developed – rigorous in the sense that they provide valid, rich information and usable such that they are not laborious or costly methods. In this paper we describe how we have used probability theory to “sample” in-process learning to achieve statistically similar information to 100% behavioral observation. Specifically, we discuss how we have used work sampling to measure observations of teamwork; used PDAs to prompt students as they work on open-ended problems to assess where they are in the process and the degree to which they are progressing towards solution; and used on-line structured reflective journals to map how and where students are in their design process. Each method is described along with data demonstrating how process measurement takes place. Reasoning Requisites for Problem Solving Authors: Jonassen, David Practicing engineers are hired, retained, and rewarded for solving problems, so engineering students should learn how to solve workplace problems. In the classroom, students solve story problems, which are well-structured (correct answers, known solution paths, established criteria, etc.). Workplace engineering problems, on the other hand, are substantively different from classroom problems. Workplace problems are ill structured and complex, because they possess conflicting goals, multiple solution methods, non-engineering success standards, non-engineering constraints, unanticipated problems, distributed knowledge, collaborative activity systems, the importance of experience, and multiple forms of problem representation. Some implications for designing engineering curricula and experiences that better prepare students for solving workplace problems are considered. An unresolved issue is the degree to which learning to solve classroom problems prepares engineering students to solve workplace problems. Are the same kinds of reasoning required to solve all kinds of problems? A hypothesis that we are seeking to test is that a limited set of reasoning skills is required to solve most kinds of problems. Classroom problem solving can be predicted by analogical reasoning and causal reasoning. Problems require that learners develop understanding of the concepts included in any problem. That understanding requires that learners be able to analogically compare the propositional relationships among problem examples. Being able to identify problem types based on the similarity of propositional relationships is a key to learning to solve different kinds of problems. The propositional relationship that most clearly describes the structure of problems are causal. Causal relationships require that learners understand those relationships both quantitatively and qualitatively. Describing the complex causal relationships that define propositions among concepts is also key to solving problems. Previous research showed that the major difference in reasoning required for solving ill-structured (workplace-type) problems is argumentation. We seek to validate these cognitive predictors of different kinds of problem solving. Undergraduate Engineering Students’ Information Gathering Behaviors During Design Authors: Kilgore, Deborah ;Atman, Cynthia ;Kang, Allison ;Deibel, Katherine The world is rapidly changing and as such requires “dynamism, agility, and flexibility” of engineers who solve our increasingly complex problems. Such engineers must be sensitive to the multifaceted global, societal, economic and environmental milieus in which they develop design solutions. Engineering educators have responded in a variety of ways, from developing workshops and seminars, to transforming curricula, to introducing broad program reforms. Most of these educational strategies have incorporated a broader vision of the environment in which engineers work and opportunities for students to practice applying their technical knowledge to realistic engineering problems. Rich, experiential learning opportunities are widely believed to instill in students a sense of the contexts in which they will practice, and some research has shown this to be true. Assessing engineering student learning as it relates to design-in-context has typically focused on the problem scoping phase of design, in which engineers identify a need, define the problem, and gather information. At the Center for Engineering Learning and Teaching, we are conducting extensive research to define what it means to have design-in-context expertise. In this paper, we will focus on students’ information gathering performance in engineering design. Previous research has shown that experts gather a broad range of information that speaks to both the technical details of an engineering design problem and the contextual issues and conditions surrounding the problem. During the 2005-2006 school year, we gathered data on the design behaviors of eight seniors and six first-year students. We selected these students to maximize diversity with respect to gender, race/ethnicity, and engineering sub-discipline. Thus, our sample consists of men and women; white, African American, Asian American and Hispanic students; and students intending to major in Aeronautical Engineering, Civil Engineering, Electrical Engineering, Industrial Engineering, Materials Science and Engineering, or Mechanical Engineering. We asked students to design a playground in a lab setting. As they developed their design solutions, a protocol administrator prompted them to “think aloud” about their design process and decisions. In addition, student participants were informed that they could request information and if it were available, it would be provided for their use. Students were given up to three hours to complete their designs. After transcribing the recordings from these sessions, we used techniques developed in previously conducted research to segment the verbal protocol data into distinct thought units and code them. First, we coded data with respect to the design activity in which the student was engaged. If the segment involved information gathering, we also coded for the type of information gathered, and whether it was available and provided by the administrator. Finally, we organized the types of information into two categories – design detail and design context – in order to compare individual and groups of students with regard to the breadth of their information gathering. Two researchers coded each of the transcripts, and differences were negotiated to consensus. To analyze the coded data, we tallied the times during which students were engaged in information gathering, as well as the kinds of information they asked for. We also calculated the percentage of information requests focused on design context versus design detail. In addition to these methods developed and validated in previous studies, we are analyzing the information gathering performance data to examine the ways and extent to which information gathered is used in the playground design. We will develop a coding scheme inductively, and apply it as we have in the past, with attention to replicability. Our analysis of the data will consist of both established and new methods of interpreting how broadly students think as they gather information and use it in their engineering designs. Furthermore, we will relate our findings here with those of a larger sample of students who were followed throughout their college years in the Academic Pathways Study. This paper will present rich case studies of student information gathering performance, as well as comparisons between first-year and graduating engineering students. These cases and comparisons of students at different points in their engineering education will serve as learning resources that bring into clarity how information gathering is conducted in engineering design. They will also serve to illustrate how sensitivity to context in information gathering may be observed and evaluated. Children Engineering: Lessons Learned from the Engineering is Elementary Curriculum Development ProjectAuthors: Lachapelle, Cathy P.;Cunningham, Christine M. How can you teach engineering to children? What are elementary school students ready to learn about engineering? The Engineering is Elementary project, now four years old, is reaching tens of thousands of children all over the United States. We will discuss the philosophical underpinnings of the project, the format of the curriculum, and some of our findings from classroom observations, teacher evaluations, and quantitative assessments of student learning. We will explain what we/ /are learning about what works and what does not when introducing engineering to elementary school teachers, classrooms, and students. Exploring Undergraduate Students' Understanding of Size and Scale in the Context of Nanoscience Authors: Light, Gregory The concept of “size and scale” has been widely acknowledged as a foundational idea for students’ learning of nanoscale science and technology. However, while recent research and anecdotal evidence have suggested that undergraduate students often have difficulty grasping this concept, little is known about their conceptions (and misconceptions) of “size and scale”. Two related studies aimed at answering this question are reported in this paper. The first study explored the ways students understand this key concept through think-aloud interviews, and identified a preliminary typology of student conceptions. Specifically, students’ conceptions seemed to vary along three interesting dimensions – whether objects belonging to the macro-, micro-, and nano-worlds could be represented by one continuous scale; whether the logarithmic scale, the linear scale, or a combination of the two was the most appropriate for such objects; and whether the numerical format of the chosen scale was integrated with the actual meaning it represents. Three major types of conceptions were classified based on variations along these dimensions, and possible reasons for such variations were proposed. Building upon these results, we developed four assessment items that further probe the critical dimensions and issues underlying student conceptions, and administered them to a bigger student population in the second study. Preliminary analysis of students’ responses to the items confirmed the typology, and provided richer information that could help educators better understand how undergraduate students come to grasp (or not grasp) the concept of “size and scale”. Promoting Innovative Design in Engineering Education Authors: McKenna, Ann F.;Hutchison, Mica Several recent reports stress that the competitive advantage of the U.S. lies in its role as a leader in technological innovation. These reports make statements such as “innovation will be the singlemost important factor in determining America’s success through the 21st century” (Council of Competitiveness, 2005). These reports send a resounding message that engineering education in the U.S. needs to emphasize and develop knowledge and skills that are essential to innovation in a rapidly evolving technological society. From an educational standpoint, there are many factors to consider in creating an environment that fosters and develops the ability to engage in technological innovation. In an effort to understand how educators might create the most effective opportunities for introducing students to innovative thinking, we are investigating students’ use of innovation in their development of design solutions. Guided by Schwartz, Bransford, and Sears’ adaptive expertise framework – a construct describing performance on two axes representing efficiency and innovation – we explore students’ perceptions of the problem-solving environment and the influence those perceptions have on their use of innovative approaches. Specifically, we are exploring two components of innovation: knowledge-application innovation (i.e. the ability to recognize when certain knowledge applies) and solution innovation (i.e. the range and novelty of ideas produced in the solution search). Participants in the study include students from two engineering disciplines, a science discipline, and a non-STEM discipline at two universities: a private institution and a state institution. By investigating students from this variety of learning environments we can begin to gain a better understanding of how to encourage the use of innovation in design. This includes determining how educators might modify the engineering learning environment to include components that promote the development of adaptive expertise among their students – our nation’s future engineers. Learning Science by Conversing with Animated Agents Authors: Millis, Keith ;Graesser, Art ;Halpern, Diane ;Britt, Anne In this paper, we present our work in building and testing intelligent computerized tutors that teach various aspects of science and engineering. Our intelligent tutors are fairly novel in that students can hold mixed initiated conversations with two animated pedagogical agents using natural language. The animated agents (talking heads) pose questions to the student and provide hints and prompts in order to help the student provide a fully elaborated answer. Both tutors are extensions of Auto-Tutor, a computerized tutor that teaches topics in science and technology. We will describe two of our tutors: the Critical Thinking Tutor and Project ARIES. The Critical Thinking Tutor teaches inquiry relevant to science, logical fallacies, and argumentation. Project ARIES, which is currently being built, is a serious game that teaches scientific inquiry in the social and physical sciences. We provide evidence of learning gains based on the Critical Thinking Tutor. We also describe how principles of learning are implemented in Project ARIES. Cognitive Models and Mixed Methods Assessment: A Natural CombinationAuthors: Moskal, Barbara M. Two current buzz terms in engineering education are “cognitive models” and “assessment.” The term cognitive model refers to the mental processes or models that students employ to make-sense of engineering concepts. In other fields, such as mathematics and physics, generalized cognitive models have been identified that repeatedly occur across subsets of students. Some of these models reflect correct understandings while others reflect misconceptions. Knowledge of the common models held by students with respect to a given concept can be used by educators to support the design of instruction that builds on appropriate models and challenges inappropriate ones. Assessment refers to a collection of methodologies that are used to examine what students’ know and can do. Qualitative assessment methodologies can provide detailed information concerning the conceptions and misconceptions of a small subset of individuals while quantitative methods support the generalization of assessment results to a broader population. By combining these two forms of assessment, researchers have the potential of defining individuals’ cognitive models in great detail, and then examining the extent to which these models can be generalized to the broader population. The generalization of these models has the potential of providing a powerful tool that may be used to improve engineering education. This paper will explore the natural connection that exists between mixed methods of assessment and the research that is being conducted on students’ cognitive models in engineering education. Assessing Cognitive Reasoning and Retention of Knowledge in Mechanics Authors: Papadopoulos, Chris Engineering practice is primarily concerned with design, which mingles scientific reasoning with experiential knowledge and conventional practice. Mechanics is the discipline that anchors the engineer’s scientific reasoning, and although it treats the practical behavior of structures and mechanisms – “what to think”, it is more fundamentally concerned with problem formulation and solution – “how to think”. Not coincidentally, all accredited engineering programs require mechanics courses at entry to major. Because mechanics is so centrally situated in the engineer’s intellectual training, it lends itself to the very study of how engineers learn, think, and learn how to think. In the first part of my paper, I review some of the literature from the physics and engineering communities regarding (1) student understanding and misconceptions of concepts in mechanics and corresponding methods of assessment; (2) methods to assess and accelerate student learning in mechanics; and (3) methods to assess student retention of mechanics concepts in future applications. I argue that while the literature is replete with examples of the first category, much less work has been done with respect to the second and third. I then suggest approaches to advance and systematize methods of assessing student reasoning and knowledge of mechanics (many of these methods will extend to other areas of engineering). In particular, I advocate for (1) the repeated use of existing techniques to validate the generalization of conclusions made from earlier stuides; and (2) increased attempts to measure of student retention by measuring performance in successive courses. In the second part of the paper, I review literature regarding the assessment of material in textbooks. I argue that while educators need not be bound by the presentation of material in the texts, the presentations indicate shortcomings in expectations of student reasoning. From Research to Practice: Redesigning AP Science Courses to Promote Advanced Learning and Conceptual UnderstandingAuthors: Pellegrino, James W.;Pemberton, Jeanne ;Reckase, Mark ;Eggebrecht, John ;Huff, Kristen ;Bertenthal, Meryl In 2002, the National Research Council published the results of a two-year study of the Advanced Placement (AP) and International Baccalaureate (IB) programs. The report, Learning and Understanding: Improving Advanced Study of Mathematics and Science in U.S. High Schools, calls for important improvements in advanced study in high school mathematics and science programs, and concludes that these programs’ efforts to emphasize the key concepts in each field are compromised by covering too many topics in each course. The report recommends that advanced courses not be designed solely to accelerate progress through a curriculum by replicating typical introductory college courses. Such courses rarely reflect what we know about how students learn and often cover too much content superficially, sacrificing the opportunity to build science students’ transferable conceptual understandings and inquiry skills. The NRC report suggests that the primary goal of advanced study programs “should be to help students achieve deep conceptual understanding of the content and unifying concepts of a discipline.” In July 2006 the College Board accepted these challenges and embarked on the redesign of all four AP science courses (in biology, chemistry, environmental science, and physics) with support from the NSF. In this presentation we will report on the process and progress of the redesign effort to date. This includes initial work that has focused on definition of the essential knowledge, skills, and abilities for AP courses in each of the four disciplines. Related work has also begun on the construction of models of student knowledge and learning in each of the four disciplines using an evidence-centered design approach. We will also report on plans for subsequent redesign phases, including development of new AP Exams, the development of curriculum and instructional resources to support teaching and student learning, and a professional development program to support the instructors of these courses. A Quantitative and Qualitative Assessment of Engineering Students’ Interactions with a Multimedia Learning Object in the Context of their Learning Styles and Cognitive Levels Authors: Stewart, Mary (Frankie);Zywno, Gosha This paper presents a summary of the results of a two-year research project looking into differences in interactions of engineering students with an online learning object. The object in question was a set of interactive multimedia tutorials, developed as an learning resource tool for undergraduate engineering students in Process Control. The research project investigated the effectiveness of this learning tool and identified behavior patterns of engineering students with different learning styles that may affect their learning. The paper first describes the collaborative effort involved and then focuses on the quantitative analysis of the volunteers’ results, including distributions of learning styles, pre- and post-test scores, and the breakdown of learning gains according to Bloom’s Taxonomy. The final part of the paper presents the learning style analysis of individual sessions where screen activity was videotaped and the volunteers commented aloud on their thought processes and choices as they navigated their way through the tutorials. The quantitative analysis of the study found greater achievement gains at the lower cognitive levels of Bloom’s Taxonomy (Comprehension or Application), and smaller gains at the higher levels of Bloom’s Taxonomy (Analysis and Evaluation). The investigation also looked at verbalization patterns at different cognitive levels and at access and verbalization patterns among learners with different learning styles. The analysis of verbalization patterns at different cognitive levels revealed that the participants generated significantly fewer verbalizations indicative of the higher-level cognitive processes as compared with the number of verbalizations at the lower level. It points to a pattern of superficial interactions with the module resulting in students benefiting more at the lower cognitive level. The observation of students’ greater cognitive activity and learning gains at the lower levels of Bloom’s Taxonomy is also consistent with the literature. The authors see the observation of low cognitive activity accompanying interactions with the learning object as an indication of a need for further investigations of an important question of how to encourage, and effectively support, students’ deep level learning and effective problem-solving, as opposed to a superficial understanding of the domain knowledge. Using Ontology Training to Investigate Why Some Engineering Science Concepts Are So Difficult To Learn Authors: Streveler, Ruth A.;Miller, Ronald L.;Slotta, James This paper will discuss an ongoing study which investigates development and testing of schema training strategies for helping engineering students develop more fundamentally accurate mental models of dynamic processes which occur at small length scales. Given the current interest in advances in nanotechnology (e.g.microfluidics, biotechnology, genetic engineering, nanoscale machines), new engineering graduates must have a firm grip of fundamental processes which are characterized by small-scale dynamic systems. Unfortunately, little is understood about how people learn in such conceptual domains and there is ample evidence in the literature to suggest that students of all ages (including science and engineering students) do not easily understand fundamental small-scale phenomena such as heat transfer, diffusion, fluid mechanics, and electricity. Based on the ground-breaking research of Dr. Michelene Chi and her colleagues (including co-author Slotta) the problem is more than one of confusion or misunderstanding, but rather involves fundamental and robust misconceptions about how the dynamics of small-scale processes (e.g., the random motion of molecules, atoms, or sub-atomic particles) differ from the observable, macroscopic behavior that we experience in our everyday lives. The study described in this paper tests Chi and Slotta’s theoretical framework of ontological commitments by creating effective schema training protocols and materials that help engineering students create appropriate mental models of fundamentally important dynamic processes operating at small length scales. The work represents an integration of two research lines combining cognitive psychology with direct applications to engineering education. First, the ongoing research of Miller and Streveler concerning why certain engineering processes are so difficult to learn provides an ideal context in which to research how engineering students can learn to design and operate the nanotechnologies of the future. Second, Slotta’s prior research in the use of ontological schema as training protocols for emergent processes is combined with the Miller/Streveler educational context in order to provide a powerful new paradigm for misconception research. An Assessment of Problem Solving Strategies in Undergraduate Statics Authors: Taraban, Roman ;Anderson, Edward E.;Craig, Curtis ;Fleming, Jacob ;DeFinis, Alli ;Brown, Ashlee G. Four well-articulated models that offer structured approaches to problem solving were identified in the engineering research literature. The models had been developed as a prescriptive response to students’ use of a “hodgepodge of tricks” to solve statics, dynamics, and thermodynamics problems, and they were regarded by their authors as useful to students to follow throughout their careers. Therefore the models were considered appropriate for this empirical study of problem solving by undergraduates. One research question was “To what extent did students follow the prescriptive steps of these models when solving problems?” Therefore, one goal of this study was to use these models to develop a coding system for characterizing engineering students’ behavioral and cognitive processes related to problem solving, and then to use those codes to describe students’ problem solving procedures in a detailed manner. Further, it was necessary to collect rich problem-solving data in order to test the descriptive adequacy of the models. To this end, we turned to the collection and analysis of verbal protocol (“think-aloud”) data. Verbal protocols are open-ended think-aloud reports, through which participants are asked to verbalize what they are thinking as they work through a task. A second research was “Did the behavioral and cognitive descriptions derived from the verbal protocol data correlate with objective measures of performance—i.e., course grades?” In the first part of this study, the descriptions of the models in the literature were converted into observable behaviors, like “draws a free-body diagram,” and cognitive processes, like “activates prior knowledge.” Eighteen undergraduate students who were currently enrolled in Mechanics I and four experts, who were engineering faculty, were recruited. The students solved two statics problems taken from a chapter over which they had recently had an in-class test; experts solved the same problems. Verbal protocols were collected while students and experts solved the problems. The coding table derived from the models was used to code students’ verbalizations. When the models did not provide a code for a specific verbalization, a new code was added to the coding table by the experimenters, through consensus, in order to exhaustively code the data. In order to relate students’ behavioral and cognitive problem solving behaviors to objective performance measures, students were separated into higher- and lower-grade groups on the basis of their final course grades. Experts, higher-grade, and lower-grade students were compared in terms of the cognitive and behavioral processes they engaged in during problem solving. Therefore, one of the outcomes of the study is an account of the cognitive and behavioral differences in problem solving observed in experts, higher-grade, and lower-grade students. A second outcome is a comparison of the adequacy of the four models in accounting for the cognitions and behaviors of experts and novices when solving problems. We predicted that the models would describe experts’ problem solving more comprehensively than higher-grade students’ problem solving, and higher-grade students’ problem solving more comprehensively than lower-grade students’ problem solving. We expected that the relevance of the models to undergraduate instruction would be in revealing areas of problem-solving instruction needing further development. How Do You Know Who Learned What? Learning and Engineering Expertise in Interactive Learning Environments and the Need for Alternative Assessment Practices Authors: Tonso, Karen As engineering education incorporates more and more interactive learning environments, assessment can become complicated, since how learning occurs and what students come to know often move beyond the bounds of strictly formal knowledge typical of textbook exercises. The research described in this paper diverged from the more prevalent psychological theories of learning and used anthropological learning theories that can see the depth and breadth of knowledge deployed from the vantage point of insiders in a community, instead of assuming a canonical knowledge set or intended curriculum; concentrate on what is shared among and across community members, that is see individuals as embedded in community practice, instead of isolated from it; and connect students’ everyday activity across time and space via taken-for-granted, shared (cultural) understandings. Here, findings from a three-year ethnographic study of student design teams describe two principal forms of expertise—including how they were learned, distributed across individuals, and made evident during senior design teamwork. Comparing these circumstances with assessment practices illustrates how an inadequate understanding of student learning emerged. Concluding by drawing on a literature about alternative assessment, the paper first highlights the need to match assessment both to the coursework tasks students are expected to perform and the full range of knowledge being deployed, as well as to develop assessment practices that are responsive to the kinds of work expected of practicing engineers; and second, suggests assessment practices that hold promise to better understand who learned what. Diagnosing and Enhancing Problem Solving Skills in Statics: Using the Integrated Problem Solving Model Authors: Van Meter, Peggy ;Litzinger, Thomas A. The improvement of students’ problem solving abilities is a vital concern of engineering educators. In recent years, various researchers have proposed that students’ weaknesses in analysis skills may be attributed to factors such as poor conceptual knowledge or general problem solving skills. Alternatively, we have hypothesized that, because analysis requires students to coordinate a variety of skills and knowledge, weaknesses in any one or more of these areas may be the underlying cause of poor analysis skills. Accordingly, we have proposed a multi-dimensional cognitive model of analysis, which we have labeled the Integrated Problem Solving (IPS) model. The IPS describes ideal analysis processes in a three-phase structure, in which phases correspond to the semiotic systems students must navigate during analysis (i.e., verbal, diagrammatic, and mathematical). General problem solving processes and conceptual knowledge are mapped onto this structure to describe how these factors influence analysis. Collectively, the IPS describes how students’ abilities to use and integrate semiotic systems, execute problem solving processes, and apply knowledge influence problem solving performances both independently and in coordination with one another. In this paper, we review our research relative to the IPS model, which is being conducted with students in Statics, and share our conclusions from this body of work. Specifically, we address findings from think-aloud and cluster analytic studies that were designed to address the cognitive processes and abilities that contribute to the quality of solutions when students solve free-body diagram problems. These findings are discussed in light of both the IPS model and relevant problem solving research. The implications of our work for the design and evaluation of educational interventions is also discussed. |
