|
Instructional Technology Portfolio | Learning Environments Design I | Project Login | Résumé | IT Portfolio | Home |
This evaluation report presents the results of the alpha and beta testing cycles of the two online modules that were developed for the purposes of providing an online learner performance support system.
The evaluation was originally intended to be performed with actual students in a face-to-face class at the Georgia Continuing Education Center on building computers, however, the class was cancelled in late November due to lack of enrollment. While the participants in this evaluation does not represent the true audience for the learning materials, it is hoped that the combined feedback of the various individuals who did participate in the alpha and beta testing of the product led to similar results that would have been obtained from the real-life participants.
The testers consisted of eight people, 2 females and 6 males, between the ages of 25 and 50. All testers were knowledgeable in instructional design. Two had some experience with either building computers or upgrading components in them.
The Alpha test phase was conducted during the class studio's dress rehersal session with eight individuals providing constructive feedback in response to an informal question and answer session conducted with each. The questions included the following:
Not all questions were presented to all testers, but at least three of the above questions was asked of each alpha tester. In addition to these questions, the alpha testers were observed as they attempted to peruse the exercises and complete the two tasks. Attention was paid to where they clicked and their reaction to the system's response. Notes were made as the above questions were answered or usability issues were observed.
At the time of the alpha testing, the database of components, questions, and simulation steps was still incomplete for a full lesson presentation. This was explained to the users and they were asked to focus on the lesson design and usability of the software.
In general, the testers thought the exercises were appropriately designed for the subject matter at hand and that a web-based approach was a very sound approach to presenting the subject. They also commented that both exercises were necessary and complementary to each other and led to further understanding of the subject matter.
For the component matchup exercise, the testers were not always certain what to do and did not read the instructions for performing the exercise. The lack of reading the instructions may have been because of the lab environment in which the test was conducted and because the researcher was on hand to ask questions about each exercise's purpose and directions for completing the exercise. The testers also consistently failed to see the instructions telling them to choose from the list of components below the top component. One suggested making the directions stand out more with a different font or color since it was sandwiched between the top component and the list of choices.
For the simulation exercise, the testers were quicker to grasp the nature of the exercise and move through the simulation steps. Part of this was because they intuitively understood the nature of the exercise. They were also not presented with a pre-selected component as in the first exercise in which they had to pause and think about why that component was displayed as occurred in the first exercise.
Overall, the testers had good remarks for the design and approach. Some suggested revisions included:
The Beta test phase was conducted through email and syncronous instant messaging sessions by sending participants an evaluation form to read for instructions for conducting the beta test and then fill out and return via email. The purpose for this methodology was to serve a less direct role in helping users navigate the exercises as occurred in the alpha testing phase, yet still be available to answer questions and discuss the tester's experiences as they went through the exercises. Six individuals were emailed the evaluation forms with three testers returning the evaluation form. The three failing to respond were not included in these results as they were also not reached via instant messaging to discuss their experiences. All of the respondants had never seen the online modules, but had expressed an interest in taking the class. The respondants included 2 females and one male and their age ranged from 27 to 38.
An evaluation form was emailed to each of the participants. The evaluation procedures provided background information about what the lessons were, how to log in and work through the exercises, and instructions for filling out the evaluation form and returning. Instant messaging was confined to answering questions about the evaluation form (i.e. "what are invisible attributes?" and "what are metaphors?") while they were working through the exercises and then inquiring for more information about the specific comments the respondants made on the evaluation form.
The evaluation form focused on usability issues, lesson design and layout, and supplimental materials. These issues were listed in table format and user asked to rank them as "acceptable" or "needs work" with space given to comment on each item.
Additionally, the testers were asked a series of open-ended questions focused on usefulness of the exercises and what they felt about certain aspects of the exercises. One of the beta testers did not answer any of the open-ended questions, so conclusion in this area is limited.
Overall, the beta testers found the exercises to be well written and easy to follow and understand. Unlike the alpha testers, the beta testers showed clear signs of having read the instructions on the screen and were easily able to follow the instructions. They felt that the subject matter was well covered and that the external links were very helpful. They felt that the simulation exercise was the more interactive of the two exercises, but also recognized the importance of both exercises. Since the beta testers performed the exercises without the benefit of the face-to-face class to introduce the subject matter, they expressed a difficulty with following the jargon. Even so, they felt that the technical terms that were presented were presented in an easy to grasp manner and could follow the external links for more information as needed.
The testers expressed a need for a way to see complete component specifications in the simulation exercise as displayed in the prior component matchup exercise. One tester also commented that she didn't like scrolling down the page to see all the information presented. As with the alpha testers, the beta testers also expressed a lack of obvious navigation to get them back to the start menu and suggested adding prominent buttons to allow them to restart the exercises at any point.
It was unfortunate that the face-to-face class for which this project was developed for was cancelled. As a result, I did not have the true "beta testers" I would have liked to have had. Because the beta testers were friends who at least had an interest in building computers, I felt that they made for a reasonable approximation of the actual learners that would have been in the class. Had the class been held, then one way to measure the learning effect more accurately than done here would have been by observation and measurement of the learner's confidence and efficency in ordering computer parts in the fourth class segment that would've followed. A questionaire could also be given to the learners reviewing the exercises after they had gone through the exercise of ordering components online in class. Additional measurements could also be taken by evaluating how many students ordered 100% compatable components, all components necessary, and successfully put the machine together from the components they ordered.
Also, since the class was cancelled and the client was no longer engaged in the project's direction, I did not feel compelled to draft a style manual and followed the styling cues that came from my existing website. The web designs I implement have a heavy influence from accessibility standards and style recommendations put forth by well recognized consortiums such as:
| mwlang@cybrains.net | Guest | Login | Home |