Arduino
Measuring the gulfs of execution and evaluation for the e-commerce giant.
After leading customer satisfaction in the e-commerce space since 2010, Amazon dropped 4% to a score of 82 (out of 100) in the American Customer Satisfaction Index’s latest retail and consumer shipping report. While it is unclear just how much usability is responsible for this drop in consumer satisfaction, it is a clear reason for conducting user research.
This case study is an exercise in designing, carrying out, and delivering the results of user research for the leading ecommerce platform which generates more than 1.3 Billion unique customer experiences each day.
The Research Process
The purpose of the research memo is to begin to develop a research project to determine what changes should be made to information architecture and user interface design to improve the their online experience for both browsing and searching for products.
High level research questions
How do people navigate the site when they’re looking for something specific?How do people navigate the site when they’re browsing a high level category like “women's clothing”?
Research Justification
A cursory evaluation of Amazon.com for fundamental usability issues revealed a number of concerns. Visibility, discoverability and familiarity are all at issue for the users initial gulf of execution in which users try to figure out how to use the site to find a product.
High level research questions
How do people navigate the site when they’re looking for something specific?How do people navigate the site when they’re browsing a high level category like “women's clothing”?
Research Justification
A cursory evaluation of Amazon.com for fundamental usability issues revealed a number of concerns. Visibility, discoverability and familiarity are all at issue for the users initial gulf of execution in which users try to figure out how to use the site to find a product.
1 Research Memo
Research Highlights
- Draft a research memo
The initial research memo provides a high level overview of the business context, identifies the focus for the research (the specific problems or goals that research can help support), and proposes one or more questions that the research will attempt to answer. Finally, it addresses how the research can feasibly be carried out, given the constraints of the project or situation. - Write a research methods section
The methods section is a narrative description of appropriate and feasible research methods with citations of sources, and explanations of why they are being selected and how they will be carried out. - Write a recruitment section
The recruitment section accurately defines the target audience in terms of behaviors, technology use, and demographics. It also includes the writing of a screener which is used to determine if a potential recruit fits the audience profile. Lastly a schedule is written for the data collection sessions and management of the logistics of getting respondents to the location where the sessions will take place. - Write a Data Analysis Section
This section describes the types of data that will be collected as well as the specific variables important for capture and the expected numbers of observations to be made. The section will also explain how the data will be analysed, and how findings will be represented (e.g. tables, graphical representations, or other visualizations). The section will include data collection instruments (e.g. data collection spreadsheets, interview guides, etc.). - Conduct User research
Interviews and task scenarios, are combined, allowing the researcher to observe both what users do and say so as to quantify the experience using metrics and understand the problems in the users experience. These sessions are followed by the administration of a survey; the SUS survey, an industry standard with references in over 600 publications. - Analyze Data and Write Findings and Recommendations Section
Following the plans identified in the data analysis section data analysis is conducted to uncover insights, patterns, and other determinations. Finally these determinations are summarized to communicate key findings using charts, graphs, and other data visualizations.
Role: UX Researcher
Methods
Contextual Interviews & Task Scenarios
System Usability Scale (SUS) Survey
Skills
Generative Research
User Research
Project Planning
In person Interviewing
Information architecture analysis
Brief writing
Deliverables
User Research Project Plan
Research Instruments
Methods
Contextual Interviews & Task Scenarios
System Usability Scale (SUS) Survey
Skills
Generative Research
User Research
Project Planning
In person Interviewing
Information architecture analysis
Brief writing
Deliverables
User Research Project Plan
Research Instruments
Case Study Overview
The Problem
Drive acquisition, engagement, retention, referrals, and ultimately, revenue by discovering opportunities to improve the experience amazon users have when browsing and searching for a product.
Category navigation mixes expected product categories with amazon services, account management and unclear labels that do not appear as topics or product or services.
2 Methodologies Section
Two methodologies have been selected to build an informed understanding of opportunities for the improvement of the Amazon e-commerce shopping experience.
Task analysis is carried out with three to five participants. Results provide content for analysis with fundamental principles of interaction to account for how they participate in gulfs of execution and evaluation.
Method 1 : Contextual Interviews & Task Scenarios, Combined
Rational
The core idea behind task scenario usability testing is having real people (users) try to accomplish real tasks with software, websites, mobile apps, or devices. Through observing both what users do and say researchers are able to both quantify the experience using metrics and understand the problems in the users experience. 9 Conducting a brief interview immediately after scenario testing allows a discussion to naturally proceed from the observations made by users during the tested scenario. Separating interview discussion from the use of an interface preserves the integrity of the everyday experience users have when navigating an interface like Amazon.com. Following a scenario test with interview questions and discussion gives the user a concrete context which helps overcome the limitations of direct interviews which rely on error prone long term memory by generating immediate and similar perceptual input.
Application
Research goals have been defined to improve upon the gulfs of execution and evaluation Amazon users face when shopping online. Research sessions will begin with warm-up interview questions and the introduction of the task scenario. A task scenario is the action that you ask the participant to take on the tested interface; Amazon.com in this case. An example scenario might be “Buy a pair of shoes for less than $40”. Participants are then observed while carrying out the task. Participants will be given tasks tailored to their interests. Highly technical products (such as a wireless router, or laptop computer) will be avoided to ensure participants are informed enough to carry out the selected task. Immediately following the usability test additional interview questions will be posed. These questions will be created in advance with the understanding that other questions may come to mind during the usability test and replace the previously written questions.
Method 2 : Survey, System Usability Scale (SUS)
Rational
The System Usability Scale (SUS) is one of the most used questionnaires for measuring perceptions of usability. It consists of a 10 item questionnaire with five response options for respondents; from Strongly agree to Strongly disagree. The survey allows researchers to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites and applications. The SUS has become an industry standard, with references in over 1300 articles and publications. The noted benefits of using SUS include that it is a very easy scale to administer to participants Can be used on small sample sizes with reliable results. Is valid – it can effectively differentiate between usable and unusable systems.
Application
When a SUS is used, participants are asked to score the following 10 items with one of five responses that range from Strongly Agree to Strongly disagree. Participants will be provided a printed page with questions and their respective likert scale for scoring by hand.
Task analysis is carried out with three to five participants. Results provide content for analysis with fundamental principles of interaction to account for how they participate in gulfs of execution and evaluation.
Method 1 : Contextual Interviews & Task Scenarios, Combined
Rational
The core idea behind task scenario usability testing is having real people (users) try to accomplish real tasks with software, websites, mobile apps, or devices. Through observing both what users do and say researchers are able to both quantify the experience using metrics and understand the problems in the users experience. 9 Conducting a brief interview immediately after scenario testing allows a discussion to naturally proceed from the observations made by users during the tested scenario. Separating interview discussion from the use of an interface preserves the integrity of the everyday experience users have when navigating an interface like Amazon.com. Following a scenario test with interview questions and discussion gives the user a concrete context which helps overcome the limitations of direct interviews which rely on error prone long term memory by generating immediate and similar perceptual input.
Application
Research goals have been defined to improve upon the gulfs of execution and evaluation Amazon users face when shopping online. Research sessions will begin with warm-up interview questions and the introduction of the task scenario. A task scenario is the action that you ask the participant to take on the tested interface; Amazon.com in this case. An example scenario might be “Buy a pair of shoes for less than $40”. Participants are then observed while carrying out the task. Participants will be given tasks tailored to their interests. Highly technical products (such as a wireless router, or laptop computer) will be avoided to ensure participants are informed enough to carry out the selected task. Immediately following the usability test additional interview questions will be posed. These questions will be created in advance with the understanding that other questions may come to mind during the usability test and replace the previously written questions.
Method 2 : Survey, System Usability Scale (SUS)
Rational
The System Usability Scale (SUS) is one of the most used questionnaires for measuring perceptions of usability. It consists of a 10 item questionnaire with five response options for respondents; from Strongly agree to Strongly disagree. The survey allows researchers to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites and applications. The SUS has become an industry standard, with references in over 1300 articles and publications. The noted benefits of using SUS include that it is a very easy scale to administer to participants Can be used on small sample sizes with reliable results. Is valid – it can effectively differentiate between usable and unusable systems.
Application
When a SUS is used, participants are asked to score the following 10 items with one of five responses that range from Strongly Agree to Strongly disagree. Participants will be provided a printed page with questions and their respective likert scale for scoring by hand.
3 Recruiting Section
- I think that I would like to use this system frequently
- I found the system unnecessarily complex.
- I thought the system was easy to use.
- I think that I would need the support of a technical person to be able to use this system.
- I found the various functions in this system were well integrated.
- I thought there was too much inconsistency in this system.
- I would imagine that most people would learn to use this system very quickly.
- I found the system very cumbersome to use.I felt very confident using the system.
- I needed to learn a lot of things before I could get going with this system.
Having performed an action users are again presented with a difficult Gulf of Evaluation, where they try to figure out whether their actions got them to their goal. Results pages for both search and browse do not conform to expectations for results. It is difficult if possible at all to discern the difference between an ad and results.
Shopping Personality Styles
Analysis techniques for comparing and contrasting user motivations and shopping styles
- Distracted Shopper – When I shop online, I often like to begin the checkout process but for one reason or another, I don’t always complete the purchase. (I do that!)
- Premium Shopper – I only buy the best. I want premium name brands with the latest features, trends and the most advanced technology. (Nope)
- Determined Shopper – I always research before I buy. I will view the exact item I want multiple times, looking for a drop in price or an offer. (Yes, but *only* for more expensive stuff that I don’t necessarily need ASAP. If I need it fast, I go out and get it after doing my best researching for an hour or two)
- Active Shopper – I am always shopping online, not always buying, but always browsing. I am not looking for incentives but they can help me convert when I’m window shopping. (Yeah, I can see that)
- Free Shipping Hunter – I like to buy online, but it has to include free shipping. If I find out the store doesn’t offer free shipping, I’ll leave and find a website that does. (Haha no, but my wife does this! :))
- Thrifty Shopper – I like to buy used or refurbished items. If I can get the same product used for much less, I’ll buy it. (Yup!)
- Loyal Shopper – I only purchase if I can join a rewards club or special rewards program. This often leads to me purchasing at the same stores. (Nope!)
- Methodical Shopper – When I shop online, I like to browse for the exact product I’m looking for. I often browse using site navigation, browsing through each product until I find that perfect product. (Nope)
- Impatient Shopper – When I shop online, I search for what I’m looking for at a given site. If I don’t find it immediately, I look somewhere else. (Yup!)
- Wish List Shopper – I like to shop online—I just don’t always purchase. I add everything I am interested in to the shopping cart to see what the total price might be. (Nope)
Amazon users are relatively easy to find. As a Big Box e-commerce giant Amazon services nearly every type of user. Distribution of user across generations is nearly equal. With all of this being equal, the different approaches users take to finding and purchasing a product can be taken as the basis for target audience profiles.
Demographic Segmentation
Demographic Segmentation