Feedback Loops for Continuous Improvement
The most powerful feature of web-based instruction is that it allows us to embed assessment into every instructional activity and use the data from those embedded assessments to drive powerful feedback loops for continuous evaluation and improvement. As we deliver the instruction, we use technology to collect real-time interaction level data of all student use. We use this data to create four positive feedback loops.
“Feedback” in this context is the information derived from student activities that is used to influence or modify further performance.
Feedback to Students
In the case of feedback to students, we refer to corrections, suggestions and cues that are tailored to the individual’s current performance and that encourage revision and refinement. All OLI courses include frequent opportunities for students to assess their own learning and receive immediate context specific feedback. Fortunately, we benefit from inheriting some of the best work done in the area of online tutoring from Carnegie Mellon and University of Pittsburgh faculty. This approach differs from traditional computer aided instruction in that traditional instruction gives didactic feedback to students on their final answers whereas the Cognitive Tutors and “mini-tutors” provide context specific assistance during the problem solving process.
Feedback to Instructors:
The richness of the data we are collecting about student use and learning provides an unprecedented opportunity for keeping instructors in tune with the many aspects of students’ learning. OLI courses can assist instructors in addressing the challenges they confront as a result of the increasing variability in their students’ background knowledge, relevant skills and future goals. Creating an effective feedback loop to instructors using the OLI courses is our current area of research.
Feedback to Course Designers
During the design process and during use, we continuously evaluate the courses by studying data from student use and learning. With the students' permission, we digitally record interaction level detail of student learning activities in all OLI courses and labs. We analyze the student activity data to learn how students are using the material and the impact of their use patterns on learning outcomes and to iteratively refine parts of the course.
Feedback to Learning Science Researchers
Some OLI courses also serve as part of the research environment for the Pittsburgh Science of Learning Center (PSLC). Learning researchers affiliated with the PSLC can embed experimental manipulations in OLI courses to test specific learning theories. The researchers then analyze the data collected by the OLI logging service using the PSLC datashop tools. The PSLC datashop has created a number of tools specifically designed to generate meaningful displays of student learning data.
Our Learning environments both build on what we know about learning and serve as a platform in which new knowledge about human learning can be developed and further refined.
“Improvement in Post Secondary Education will require converting teaching from a ‘solo sport’ to a community based research activity.”
- Herbert Simon
Initiative Goals
We use knowledge from learning science and the affordances of the web to transform instruction, significantly improve learning outcomes and achieve significant increases in productivity in post secondary education.
- Produce exemplars of scientifically based online courses and course materials that enact instruction and support instructors
- Provide open access to these courses and materials
- Develop a community of use, research & development that contributes to the evaluation, continuous improvement, and ongoing growth of the courses and materials.
Assessment and Evaluation
Students and faculty who use the courses benefit from the opportunities for assessment and evaluation that are built into all courses. Frequent formative evaluation gives students the type of constructive and timely feedback on progress that is available from individual tutoring sessions but almost always absent from digital learning environments. Continuous evaluation of class performance gives faculty the information they need to effectively modify or supplement instruction to meet learning objectives.
Building a Community of Use
A primary objective of the initiative is to build a community of use for the courses that plays an important role in ongoing course evaluation, development and improvement. The courses are developed in a modular fashion to allow faculty at a variety of institutions to either deliver the courses as designed or to modify the content and sequence to fit the needs of their students and/or their curricular and course goals. These courses have been broadly disseminated at no cost to independent learners and at low cost to students using the materials in accredited courses.
A New Process for Developing, Evaluating and Improving Learning Environments: continuous improvement iterative model with integrated research and practice
Development is by teams composed of learning scientists, faculty content experts, human-computer interaction experts, and software engineers in order to make best use of multidisciplinary knowledge for designing effective learning environments. The team takes a research and engineering orientation toward understanding and addressing the core problems of learning in the domain under development. We have identified particularly fruitful opportunities for web-based materials that couple the evolving understanding of cognition and learning with improvements in computer technology. We know that sometimes, fundamental shifts start with small steps.
Traditional course development or educational software development is often referred to as a “waterfall” model because there is no process for the water to run back up the hill again and go for another iteration.
In contrast, the Open Learning Initiative (OLI) development process supports multiple passes with feedback and is an “iterative” model.
