Research
pasteursquadrant

Our View of Research and Evaluation:

Learning theory and faculty expertise guide the initial development of each course. As the courses are delivered, OLI researchers conduct a variety of studies to examine the effectiveness and usability of various educational innovations. The research results are used not only to improve the courses, but also to contribute to a growing understanding of effective practices in open learning environments.

We view research and applications in learning design as synergistic enterprises rather than as points at opposite ends of a spectrum, or as discrete phases of a "research leads to practice" model.  We  choose to explore theoretical quesitons about learning in contexts that really matter; and, when we work on real-world education challenges we frame them so that our work helps us make progress on fundamental theoretical issues.  We conduct our work in "Pasteur’s Quadrant."

Current Projects



Community College - OLI

Carnegie Mellon University, in collaboration with state agencies and national affinity groups, will establish a consortium of community colleges that will enact a large scale, systems-change process that increases efficiency in the way instruction is developed, delivered, evaluated, and continuously improved. The overarching goal is to demonstrate a 25% higher rate of course completion for students from vulnerable populations, with a focus on gatekeeper courses critical to graduation success. Within three years, the Community College Opening Learning Initiative (CC-OLI) will scale to 40 community college partners and will reach an additional 50-100 classrooms.

 

Read more...
 
Integrating Pedagogies and Technologies that Support Individual Learning and Group Knowledge Building

Principal Investigators: Candace Thille and Marlene Scardamalia

The OLI pedagogy and technology is designed to support and evaluate individual learning of content and  practices in complex domains. The Knowledge Forum pedagogy and technology developed by Marlene Scardamalia and Carl Bereiter is designed to support and evaluate group knowledge building (Knowledge building is a process of sustained idea improvement fostered by communities in which participants take  responsibility for the advancement of community knowledge). We will integrate the two pedagogies and technologies in an exemplar OER so that teachers and learners can participate in both environments simultaneously and so that we can collect a single data store of their use in the integrated environment. As the integrated environment is used to support teaching and learning, we will use the OLI and Institute for Knowledge Innovation and Technology (IKIT) analytic tools to evaluate the ways in which the integrated approach improves both individual learning and group knowledge building. There are many and important related research issues. For example, if learning is enhanced for a few students, do their contributions to the collaborative space enhance the work of the group; how can we assess growth and spread of ideas; can we keep ideas alive and improving in a worldwide open community?

The teachers using the integrated OER will receive fellowships for their participation in the use and evaluation research. OLI and IKIT technical staff will integrate the learning and data collection environments.

Outcomes on the specific OER level:
1. A resource that exemplifies a combined pedagogical approach of fostering individual learning and group knowledge building
2. Data on the effectiveness of the specific OER in improving teaching and learning.
3. Data on the conditions for effective travel of the OER.
4. Capacity building in the areas of new pedagogical approaches, evaluation and design for the teachers using the integrated OER.

On the network level:
1. A learning environment that facilitates the simultaneous participation of learners in individual knowledge building and group knowledge building practices.
2. Contribution to OLnet’s collection of evaluation methods, data analysis tools and design practices.
3. Contribution to OLnet’s evaluation data collection.
4. Analytic tools for visualizing both individual knowledge gain and emergent processes of knowledge building over time.

 
An Evaluation of Accelerated Learning in OLI “Logic & Proofs”

Principal Investigators:  Christian Schunn and Mellisa Patchan

The current project evaluates the OLI Logic and Proofs course ( L&P), specifically testing the hypothesis that use of L&P in a hybrid instructional mode (online instruction combined with reduced and strategically targeted face-to-face instruction) can lead to accelerated learning, either in the form of more content learning within the same amount of student time or the same content learning within a shorter time period of student time.

Before beginning the evaluation experiment, we conducted a context analysis to determine the real market of the Logic & Proofs course: what kinds of universities tended to teach this overall course, what content tended be covered, and what kinds of tools for proof construction were already being used. Overall, L&P-like content is taught in a majority of universities in the US, but with some variation by university type. Further, L&P contains some content that many courses did not cover. Thus, the applicability of the course is high, and the opportunities for acceleration via more content coverage also exist.

The evaluations are being conducted in two phases. First, two evaluations have been conducted at a large public tier-one institution, showing that face-to-face instruction produced equivalent learning outcomes to online-only instruction. In addition, online instruction appeared to result in a much lowered attrition rate, providing efficiencies for students of another form.

The current evaluation is being conducted of accelerated learning at three institutions, each representing different types of institutions, using different forms of acceleration and evaluation logics suited to the context (e.g., random assignment to condition vs. natural experiment, more content in equal time vs. equal content in less time).

 
The Open Learning Network (OLnet)

Principal Investigators:  Candace Thille and Patrick McAndrew

OLnet: Network to Support Sharing Methodologies and Evidence on the Effectiveness of OER

The aims of The William and Flora Hewlett Foundation's Open Educational Resources (OER) program in 2007 were stated as to: 1. Sponsor High-Quality Open Academic Content; 2. Break Down Barriers to Open Educational Content; and, 3. Encourage People Worldwide to Use Open Educational Resources.

The resulting OER movement has been successful in promoting the idea that knowledge is a public good, expanding the aspirations of organizations and individuals to publish OER.

The Open Learning Network  (OLnet)  aims to extend this cultural shift, to evaluate the impact of OER on teaching and learning and to facilitate transformative educational practices. Those aims are achieved by developing the links among the design, use and evaluation of OER. OLnet provides the infrastructure and builds capacity for a community concerned with improving OER design and applying methods for assessing robustness of OER.

The approach of OLnet is to build structures and activities to nurture the growing pool of OER and associated services, developing a complementary infrastructure to investigate and report on the issues around OER deployment and evaluation. OLnet takes the opportunity to go beyond isolated, individual views of OER effectiveness, by aggregating data, sharing evaluation know-how, and mediating dialogue and debate within the community.

 
The OER Effectiveness Cycle

Principal Investigators: Candace Thille and Jim Greeno.

This project seeks to establish a framework for evaluating and improving the effectiveness of Open Educational Resources (OER) in supporting improved teaching and learning in various contexts. Working with OER producers and users from the OpenCourseWare Consortium (OCWC) and other OER projects as participatory researchers, we will move several representative OER through the OER effectiveness cycle as the OER travels to a new context.

We will organize a network of OER producers and adapter/users, who will develop and use evaluation methods, document the evaluation methods, the data analysis and design practices, and the results of the evaluation showing the mechanisms through which the OER supports improved teaching and learning and the conditions and resources that can support effective travel. In at least one case, we will conduct an intensive case study, as an example of organizational learning. In the study we will work with an OER producer and with faculty at an institution that adopts the OER, studying the adopting faculty’s discussions and decisions in which they adapt, amend, and utilize the materials that the OER provide. Goals of this research (both the producer/user network and the case study or studies) include gaining information that can guide the design of resources that can be provided to assist adopters of OER in successfully adapting OER for their local use.

The OER producers/users involved in this project receive fellowships to support their participation.

Outcomes on the specific OER level:
1. Data on the effectiveness of the specific OER in supporting improved teaching and learning.
2. Data on the conditions and resources that can support effective travel of the OER.
3. Capacity building in the areas of evaluation and design for the specific OER producers and users.

On the network level:
1. A framework of progressive refinement for evaluating and improving the effectiveness of OER in supporting improvement in teaching and learning.
2. Contribution to OLnet’s collection of evaluation methods, data analysis tools and design practices.
3. Contribution to OLnet’s evaluation data collection, including an example of a network that provides infrastructure for producers and users of OER to interact and benefit from each others’ experience and learning in adapting OER in local programs and practices.

 
The Digital Dashboard for Learning (DDL)

Principal Investigators: Marsha Lovett and Ken Koedinger

The Digital Dashboard for Learning (DDL) project, funded by the Spencer Foundation, aims to develop the most useful feedback on student learning outcomes for instructors of introductory college courses. When instructors know exactly where students are struggling, they can more readily use precious class time to focus on those issues.

Read more...
 
Assessment Rich Instructional Modules for Biochemistry and Molecular Biology

Prinicipal Investigators:  Gordon Rule and Diana Bajzek

This project is directed at developing instructional modules in the general area of molecular biology and biochemistry.  These modules are unique in that not only do they portray a biological process, but they have embedded assessments with feedback to tightly couple student activities within the module with formative assessment. 

Using new developments in the Open Learning Initiative (OLI) as a platform for the delivery of these materials, we are able to monitor student responses to the assessments within these modules to return feedback to the instructor.  For development purposes these modules will be delivered as part of blended lecture/online introductory Biochemistry course to a class of ~100 chemical engineers (~50% female) at Carnegie Mellon University.

The fundamental unit of each of these modules is an animation/simulation and associated controllers that can be used to change the behavior of the animation.  These animations can be divided into three distinct groups: i) a pictorial representation of a biological process (e.g. protein synthesis), ii) a simulator that represents the molecular behavior of a system, iii) a virtual lab whose underlying mechanism is based on molecular properties.   A truly unique feature of the simulator is that the behavior of the molecules is calculated in real time based on the intrinsic, user defined, properties of the molecules.  Consequently, these simulations are numerical simulations that drive the animation that is viewed by the student.  This feature of our simulator environment allows the student to observe and modify the molecular behavior of an authentic system (Rule & Bajzek, 2005; Brown & Collins & Duguid, 1989).  Another critically important feature of these modules is the integrated delivery of formative assessments that provide immediate feedback to the student during the activity.  The course management tool that will be used for this project also reports student activities back to the instructor.  This valuable feedback can be used by the instructor to refine the simulator environment or fine-tune lecture material in blended courses.