Revamping the webapp for calibrated peer reviews
SUNY Oswego's HCI program collaborated with IBM stakeholders on an Ed-Tech course management project (the calibrated peer review tool), which required the application of software engineering and human-computer interaction concepts.
Version 1.0 was dysfunctional and riddled with usability issues.
◉ Redesigned a scalable Calibrated Peer Review Tool, which streamlined the peer review process for the students of requirements engineering at SUNY Oswego.
AFFILIATION — IBM, State University of New York at Oswego
METHODOLOGIES — Agile SCRUM, Personas, Heuristic Evaluation, Information Architecture, User Testing, Task Analysis, Empathy Mapping, Wireframing & Prototyping, Design System, Feedback Review, Design Iterations, Dev Handoff, Stakeholder Presentations
TEAM — Requirements, Engine, GUI, Database Networking, Quality Assurance, Usability
ROLE — GUI Lead, Usability Analyst
Calibrated Peer Review tool's dynamic learning approach for the requirements engineering course
Teaching the Requirements Engineering (RE) course is challenging due to the absence of absolute correct solutions for a problem. Instead, multiple compromise/alternate solutions exist for each problem.
To expose students to as many solution alternatives as possible, the CPR tool proposes a learning-by-multiple-examples process using calibrated peer review grading assignments, and a think-pair-share model for semester-long, industry-realistic, project-based milestones.
Digging deeper through heuristic evaluation & hierarchical task analysis
Upon conducting a heuristic evaluation of the previous version of the tool (CPR 1.0), the usability team discovered several issues, including inconsistent visual design, navigation structure, and accessibility concerns.
To assess CPR 1.0's functional efficiency, we conducted a hierarchical task analysis for both student & instructor roles, which gave us a head start on
-
Understanding user flows
-
Identifying problem areas
-
Anticipating stakeholder needs
Through empathy maps, the initial goals, needs, & pain points for student & instructor users were examined from a humane lens. This resulted in minor feature additions to MVP
Discovering usability issues
The primary research revealed issues including but not limited to
Connection problems (related to Google OAuth & SSL certification)
Ambiguous navigation structure
Lack of access (for unified communication channels among peers)
Manual & time consuming processes (for creating files & folders for peer reviews)
Repetitive steps (causing inefficiencies in workflow)
Solidifying Minimum Viable Features
After analyzing the usability issues for previous version of the CPR tool (1.0) — we decided to implement redesign changes.
The goals were to:
-
Increase student engagement
-
Reduce cognitive load for the instructor, &
-
Streamline the traditional CPR process
MVFs & revamp requirements were identified for a successful product relaunch
.png)
Revamping the old design system & presenting the prototype
The design process consisted of 3 major stages.
ASSESSING DESIGN FEASABILITY
through iterative collaboration with stakeholders
ESTABLISHING A DESIGN SYSTEM
to ensure consistent visuals and functionality
CREATING MOCKUPS & PROTOTYPES
validating the MVFs to begin testing the designs
The extensive design system included typography, color guide, input fields, buttons, navigations, grids, tables & charts, pop-ups & tool tips
Usability testing, quality assurance, & GUI testing
The testing focused on three main measurements: efficiency, satisfaction, and learnability.
-
Efficiency was evaluated through time on task and task success rate
-
Time on task gauged how long participants took to complete assigned tasks,
-
Success rate measured completion difficulty on a scale from zero to two
-
Satisfaction was self-reported on a Likert scale from one to five
-
Learnability, assessed through time on task and success rate
Results from qualitative & quantitative analyses
With CPR Tool's limited functionality during testing and small sample size of 25 — we removed outliers to avoid significant analysis distortion.
For the instructor roles, we found that
For the student roles, we found that
Qualitative analysis during usability testing sessions for the student & instructor roles
Design & process recommendations for product development
The usability team's reccomendations aimed to address the usability issues. The 3 major ones were:
IMPLEMENTING A SET UP WIZARD
to improve the user experience for those unfamiliar with the peer review system & help with error-reduction
SEARCH FUNCTIONALITY
to improve CPR app usability for instructors handling large data sets & help students find assignment information quickly
ADDING BETTER CONFIRMATION PROMPTS
for destructive actions, such as deleting teams or editing grades, can prevent irreversible errors and confusion for users
Building a scalable peer review & coursework tool
A tool that:
-
Empowers students & instructors to facilitate teaching & learning efficiently
-
Automates the manual think-pair-share model
-
Incorporates the learning-by-multiple- examples approach
What was achieved with the Calibrated Peer Review Tool 2.0:
-
Basic functionalities work
-
Cross-platform compatibility
-
Clear navigation
-
Bug fixes & broken links
-
Consistent visual design
-
Clear error messages
-
Confirmation screens for users
Learning the ins & outs of launching an MVP
The project provided an excellent introduction to the dynamic landscape of software design and development.
I learned to:
-
Integrate software development such as technical debt management & priority matrices concepts into my design thinking
-
Co-ordinate efficiently with developers, usability analysts & stakeholders & implement developer handoff
-
Advocate for front-end development as the team lead for GUI & hone my team-building & facilitation skills