Week 4: Designing and testing a prototype

Summary

In chapters 3 and 4, Frick and Bohling call for rapidly prototyping and testing a paper-based version of planned web-based instruction. The overarching purpose is to answer three essential questions: Is it effective? Are students satisfied with it? Do they find it usable?

In reviewing these chapters I’d like to extend the comparison I employed last week, when I related the creation of instructional objectives to the critical-thinking phase that opens the writing process: Where do I intend to go, and how will I know it when I see it? In the current chapters, we have an analogy to the writer’s draft that follows the collection of information needed to build the writing. Literally, we’re creating a first daft of the instruction and testing it. Like a writing draft, it allows us to ask: Have I got what I need? What’s missing? What works? What needs work? What works, in this case, is what students find easy to use and makes them willing to learn, and enables them to demonstrate mastery of the lesson content. What needs work is that which falls short of the mastery goal or results in student dissatisfaction or a lack of usability. Effective drafts always show the scope of a written piece and develop its major thematic points; they also, almost without fail, produce surprises of inspiration or gaping holes. They show the writer what must be done next to produce a usable piece. Likewise, Frick and Bohling find that prototyping almost always leads to at least one redesign in pursuit of usability.

To that end, they counsel a rapid, efficient design and development of a prototype on paper, drawing on one’s own teaching experience or consultation with others who have taught the content. The prototype should include the bulk of the lesson content (if not all of it), and should reflect what one might call the skeleton of the site, with representations of all major sections and subsections. They also call for developing several series of pages that mimic the links students (presumably) would follow to the major areas of the site, especially its deepest areas. The presumption reflects the fact that students during testing will follow a trail other than the one that we, as designers, desire for them. That done, the designer should apply Merrill’s five-star criteria (real-world problem, activation, multiple examples, extensive practice, transfer) to the prototype; develop a means of assessing student mastery that employs authentic assessment; develop a means of assessing satisfaction; and seek appraisal of the design by an expert.

 When it comes to administering the test, Frick and Bohling call for three main ingredients (pp. 40-41): authentic subjects (those who would actually use the instruction); authentic tasks (rooted in the instructional goals); and authentic conditions (the same level of support students will have). From here, the authors assume a team-based test-and-revision system, in which the team creates, pilots and follows a blueprint for the prototype test, including a script for what team members say to test subjects. The authors counsel watching for such things as test subjects giving up or doing what’s not expected, doing the right thing but with anxiety, or doing the wrong things with great confidence. (pp. 53-55) Test subjects are given a pre-assessment, a post-assessment and reactionnaire, and are debriefed afterward.

When the time arrives to analyze results, the authors call for identifying patterns in the test observation data and, from those, drawing conclusions about probable causes of design problems, with special attention to effectiveness, satisfaction and usability. From these, the team should develop a prioritized list of revisions that address ways to improve those core areas.

Critique

While much of my critique is embedded above, I would add that the authors’ assumption of a design and testing team is unrealistic in many settings, especially in a corporate context. While corporate training certainly needs greater attention to instructional prototyping aimed at improving effectiveness, satisfaction and usability, it is most likely to do so with individual practitioners consulting off-site subject matter experts for help with formulating content. Fortunately, the authors provide an otherwise sound, effective framework for prototyping and testing. The challenge in many settings is finding the time; indeed, the first “live” instruction is likely to serve as the first prototype, and improvements are likely to be made incrementally as the instruction matures. The authors would do well to address this reality directly.