Study of Carnegie Mellon Software Institute Computer Emergency Response Team Coordination Center Website
Prepared for:
TS5140 System Usability Engineering, Analysis, and Design
Prepared by:
Capella University
Graduate School of Technology and Business
August 19, 2005
Introduction
When study Educational Psychology at the Colorado University’s Graduate School of Education, Heuristic learning defined a category of learning through observation. For example, children learned the roles of male and female through observation of parental behavior. African American children learned the peculiars of Jim Crow and Segregation by watching elders. Heuristic evaluation of user interfaces work in a similar manner.
The testing of interface is a complicated and often expensive. Unlike User Testing where outside volunteers were recruited to evaluate the current state of affairs, heuristic testing is an in-house process. Instead of naïve system user attempt to navigate at the directions of a facilitator, heuristic evaluators are domain experts with the interface (Nielsen, 1994).
Each evaluator has a preset list of criteria to evaluate the new user interface.
Criteria
1. How simple to use is the interface? Does it parallel the user's work process and encourage an appropriate task sequence? For a more complex task, how well does the interface step the user through subtasks?
2. [When graphics are involved] How clear are the meanings of graphical elements such as icons and toolbar buttons? Are they overused or underused?
3. How well is the interface organized? Are navigational aids adequate to support the organization? What feedback is provided to orient the user?
4. Are instructions or explanations presented clearly, without unnecessary complication or ambiguity? Is the language direct, simple, and non-wordy, so that users can read/hear as few words as possible to accomplish a task?
5. [For software and web sites] How effectively are analysis and/or search results presented on the screen? What window manipulations are required to view results easily?
6. What information (text, voice, or graphics) must users encounter that they don't need? What information might be missing?
7. How well does the interface assist users in recovering from problems (Tec-Ed, Inc 2005)?
Evaluation Method & Results
Each member of the development staff that was not participating in the actual development of the new web site was given first a paper prototype of the new site. Next, they received access to the working prototype. With the working prototype, they performed their evaluation. The evaluators score the new interface by answering the questions in the criteria listed above as they performed the tasks required of them when they visit the CERT website.
To illustrate the test, the questions from the criteria are repeated and the evaluators’ scores are marked in blue:
Evaluator 1
1. How simple to use is the interface? Yes, the interface is less cluttered
a. Does it parallel the user's work process and encourage an appropriate task sequence? Yes, before the search tool was the most utilized tool. New interface brings research to forefront.
b. For a more complex task, how well does the interface step the user through subtasks? Functions well. Needs to add more functionality to Archives
2. How clear are the meanings of graphical elements such as icons and toolbar buttons? Roll over buttons in contents too large
a. Are they overused or underused? Graphics used to describe special events
3. How well is the interface organized? Interface well organized needs more redundancy
a. Are navigational aids adequate to support the organization? Links in text need underline
b. What feedback is provided to orient the user? None
4. Are instructions or explanations presented clearly, without unnecessary complication or ambiguity? Help text is not need, ecommerce self-explanatory.
a. Is the language direct, simple, and non-wordy, so that users can read/hear as few words as possible to accomplish a task? Yes
5. How effectively are analysis and/or search results presented on the screen? Na, same search engine.
a. What window manipulations are required to view results easily? All windows open for text documents or PDF.
6. What information (text, voice, or graphics) must users encounter that they do not need? All relevant
a. What information might be missing? none
7. How well does the interface assist users in recovering from problems? New interface much simpler less cluttered with meaningless connections
Evaluator 2:
1. How simple to use is the interface? Interface too simple, much of information lost
a. Does it parallel the user's work process and encourage an appropriate task sequence? Yes
b. For a more complex task, how well does the interface step the user through subtasks? Functions well. However, needs more information on front page to alert user.
2. How clear are the meanings of graphical elements such as icons and toolbar buttons? links are not visible
a. Are they overused or underused? Underused, revert links to standard format.
3. How well is the interface organized? Interface well organized needs more redundancy
a. Are navigational aids adequate to support the organization? Links to other pages and text needs to be underline
b. What feedback is provided to orient the user? No feedback
4. Are instructions or explanations presented clearly, without unnecessary complication or ambiguity? No instructions on web site other than on purchase of text, membership, or classes. Then server explains error
a. Is the language direct, simple, and non-wordy, so that users can read/hear as few words as possible to accomplish a task? Yes
5. How effectively are analysis and/or search results presented on the screen? The same search engine is used as before.
a. What window manipulations are required to view results easily? None
6. What information (text, voice, or graphics) must users encounter that they do not need? None
a. What information might be missing? No missing parts
7. How well does the interface assist users in recovering from problems? System loads easily. It is not essential part of desktop
Evaluator 3
1. How simple to use is the interface? less cluttered
a. Does it parallel the user's work process and encourage an appropriate task sequence? Need more practice, simplifies the process. Less reading required
b. For a more complex task, how well does the interface step the user through subtasks? Archives are at forefront, not a major responsibility of site.
2. How clear are the meanings of graphical elements such as icons and toolbar buttons? Roll over buttons too large
a. Are they overused or underused? No
3. How well is the interface organized? Very well organized
a. Are navigational aids adequate to support the organization? Links in non-standard format put back in underline format
b. What feedback is provided to orient the user? No feedback, user utilizes about us to send email to organization
4. Are instructions or explanations presented clearly, without unnecessary complication or ambiguity? No instructions
a. Is the language direct, simple, and non-wordy, so that users can read/hear as few words as possible to accomplish a task? Good use of language
5. How effectively are analysis and/or search results presented on the screen? Same as before
a. What window manipulations are required to view results easily?
6. What information (text, voice, or graphics) must users encounter that they do not need? Same as before
a. What information might be missing? Same as before
7. How well does the interface assist users in recovering from problems? Page does not assist, page is designed to be self-sustaining, and Visited links change color.
Evaluator 4:
1. How simple to use is the interface? the interface is less cluttered
a. Does it parallel the user's work process and encourage an appropriate task sequence? Easy to use brings research to top.
b. For a more complex task, how well does the interface step the user through subtasks? Functions well. Archives not adequate
2. How clear are the meanings of graphical elements such as icons and toolbar buttons? Roll overs too large should be square
a. Are they overused or underused? Not over used on every page
3. How well is the interface organized? Only two menus and one content on pages.
a. Are navigational aids adequate to support the organization? None standard links difficult to find links without underline
b. What feedback is provided to orient the user? About us
4. Are instructions or explanations presented clearly, without unnecessary complication or ambiguity? Language good,
a. Is the language direct, simple, and non-wordy, so that users can read/hear as few words as possible to accomplish a task? News feature sets tone for website alerts novice security personnel for what to look for
5. How effectively are analysis and/or search results presented on the screen? Same as before
a. What window manipulations are required to view results easily?
6. What information (text, voice, or graphics) must users encounter that they do not need? Same as before
a. What information might be missing? No problem with search engine information, clutter was problem, results Same as before
7. How well does the interface assist users in recovering from problems? Alerts user to new problems on Internet and how to overcome security leaks.
Evaluator 5:
1. How simple to use is the interface? Yes, removes unnecessary links on first page
a. Does it parallel the user's work process and encourage an appropriate task sequence? New interface makes it easy to navigate because it has the navigation tools on every page. Making the work of updating pages more complex.
b. For a more complex task, how well does the interface step the user through subtasks? The new site places the updating squarely on administrator and staff. New articles push aside the old format of links to new issues.
2. How clear are the meanings of graphical elements such as icons and toolbar buttons? Roll over buttons should be replaced with navigation bar.
a. Are they overused or underused? Great improvement over the old format needs more pictures to liven up the place.
3. How well is the interface organized? Good interface and on every page.
a. Are navigational aids adequate to support the organization? Bring back the standard link format.
b. What feedback is provided to orient the user? About us, contact us.
4. Are instructions or explanations presented clearly, without unnecessary complication or ambiguity? Alerts vulnerabilities and fixes do not change
a. Is the language direct, simple, and non-wordy, so that users can read/hear as few words as possible to accomplish a task? Language not a problem
5. How effectively are analysis and/or search results presented on the screen? Search engine remained the same
a. What window manipulations are required to view results easily? NA
6. What information (text, voice, or graphics) must users encounter that they do not need? NA
a. What information might be missing? NA
7. How well does the interface assist users in recovering from problems? NA
Discussion
Judging from the results, the new web pages are accepted for their simplicity. However, improvements still need to be made. On every evaluator’s sheet, the issue of link format was a major concern. The traditional link method of underline in blue was replaced by the box method when the mouse rolls over. The majority of the evaluators believed standard links are less confusing than the hover box links used. Perhaps the designers can reach a compromise by combining the more aesthetic new approach with the traditional. The underline returns with a pale yellow box on roll over to emphasize the link and the forthcoming event. Second, the pale yellow box makes the links stand out; there is no confusion as to which is struck. This effect can be implemented with CSS style sheets and Java Scripts.
A second concern of evaluators was the lack of feedback on the site. Figure two in the Appendix illustrates the problem with the old interface. The contents of the web site were poorly organized and spread over the site. The new interface is centrally located accessible from any page under the menu link “About Us.” The “About Us” page contains all the information about the mission, employment, funding, sponsor, and operator in one convenient location.
The final issue each evaluator listed on the test was the size of the roll over buttons for the content. The buttons were quick roughed in when, the Macromedia Fireworks design tool lost its standard collection of buttons due to the Zotob Worm corrupted media files.
An issue not brought up on the inventory was the bottom links added through Breadcrumbs. The previous version of the site neglected to include bottom navigation clues. The only method was the browser back button.
References
Neilson, J (1994), How to conduct a Heuristic evaluation, Retrieved from useit.com http://www.useit.com/papers/heuristic/heuristic_evaluation.html
Tec-Ed, Inc (2005). Heuristic (Expert) Evaluation. Retrieved August 20, 2005 From Tec-Ed Usability Evaluation http://www.teced.com/ue-he.html
