Project Overview
We performed a usability test for Smartsheet’s Solution Center to assess the current and potential usability issues customers face when selecting templates and template sets within the template gallery. Our study was conducted with the goal to assist Smartsheet in the process of restructuring their template listings.
My Role & Practices Tools Project Info
UX Researcher & UX Designer
Facilitator
Script writing
Survey
Qualitative data analysis
Miro
Google Analytics
Google Spreadsheet
Zoom
Sponsored Project
Usability Study
Individual Redesign
10 weeks
Real Stakeholders
Client introduction
Smartsheet is a leading project management platform that facilitates end-to-end workflow direction. Over 90% of Fortune 100 companies use Smartsheet’s platform to transform their workflow, improve collaboration, and manage their content.
Smartsheet offers hundreds of template formats to help their customers get started quickly and efficiently. Customers have access to the Solution Center, where a gallery of pre-formatted templates allow teams to manage their projects.
Purpose & Goal
The goals of this study were to:
Assess the overall effectiveness & usability of the Solution Center for experienced and inexperienced users
Assess user satisfaction of the Solution Center
Determine whether it is easy to complete common actions and key tasks: Import, create, and share a template, and implement add-on features to a workflow
Identify obstacles to completing key tasks while using Solution Center
Participants
Our team worked with users both inside and outside of the Smartsheet client base to conduct usability tests. The varied experience levels of participants offered a comprehensive analysis of the product. We wanted to understand the issues that not only long-term clients face, but also understand how new customers struggle with using the tools offered by the Solution Center. Each user group was guaranteed to have a minimum of three participants. The Smartsheet team helped us recruit four participants from their distribution lists to fulfill the requirement for experienced users. Our team recruited three additional novice users from our personal networks. We had seven participants in total.
The criteria for the two user types are shown below:
Methods & Process
The team adopted the within-subjects design for the test by having each participant perform all tasks. The team observed participants as they performed a series of tasks interacting with the template gallery on the desktop site. To respect the participant’s time, the team designed this study to be completed in 45 minutes. The test session consisted of a pre-test questionnaire, six tasks, two post-scenario questions after each task, and a post-test questionnaire before debriefing. Details about the specific questions and tasks in the study can be found in the Appendix.
Pre-Test Questionnaire
We have two different pre-test questionnaires: one designed for novice users and one for experienced users. The pre-test questionnaire for novice users centered around their first impressions of the Solution Center on Smartsheet, and for experienced users the questionnaire was focused on their previous experiences using the Solution Center and templates.
Tasks
The moderator read the scenarios and tasks aloud to participants, and clarified tasks if participants expressed difficulty in understanding. The study included six tasks in total.
Task 1: Open a template that you can use to keep track of a budget.
Task 2: Open a template set’s sheet suitable for event registration.
Task 3: Open the template set called “Strategic Plan” and access the list of sheets, dashboards, and reports in it.
Task 4: Import any new file.
Task 5: Create a new sheet using the Task List.
Task 6: Integrate a Gmail account with a Smartsheet workflow.
Post-scenario questions
After each task ended, the moderator asked participants two follow-up questions to ascertain the participant’s perception of the ease or difficulty of the task, and also any changes they would like to make for a better experience. Post-scenario questions helped us to identify what causes the participant’s frustration and issues we needed to revisit to reduce churn risk.
Post-Test Questionnaire (System usability scale)
After completing all tasks, the moderator asked participants to complete a five point Likert-scale questionnaire to understand the system usability of Solution Center and template gallery on Smartsheet. Participants were given a scale of 1 to 5 where 1 represents “strongly disagree” and 5 represents “strongly agree” in response to ten statements.
Debriefing
In the end, a debriefing session was held to explore and review the participant's actions during the performance portion of the usability test. The moderator discussed observed problems that arose during the test, and also checked with observers in the study for any additional questions.
Data Anlaysis
Data Collection
During the test session, the assigned note-taker took notes of the observations on the data logging goole spreadsheet as the participants went through the tasks, and we collected all data (Qualitative & Quantitative) and formatted them in our shared google drive doc in the end.
Affinity Diagramming
After all testing sessions, the team gathered together and transferred all observations from the excel sheet to sticky notes on a Miro board. Then we conducted affinity mapping to find out common themes.
Severity Rating
We followed the severity scale internally used by Smartsheet company.
Critical Findings & Recommendation
1. Most users are overwhelmed while trying to use template sets because of unclear action buttons and lack of visual hierarchy in the Learn More section.
Severity Rating: 4
# The learn more page with overwhelming information.
While accessing template sets, all participants displayed confusion at the purpose of the “Learn More” button due to its wording. Furthermore after opening the “Learn More” page, no participant was able to identify the “Download” button as the call to action button and described feeling lost and overwhelmed due to an overload of new information. Overall, the “Learn More” button and “Learn More” page create an overwhelming and chaotic experience for users.
Recommendations to resolve this issue:
Keep the instruction page succinct. Only include the essential information on the page; secondary information can be hidden under a “Read More” button or subpage.
Redesign the “Download” button in a more clickable way, such as using “drop-shadows” to make the download button stand out against the background.
The wording “Learn More” can be rephrased to provide hints, indicating more about the action of using the template set. Users want to use the template set; the phrase “Learn More” does not communicate that the user must open the documentation first before they can use the templates.
2. Most users are not able to locate add-ons and integrations because of a mismatch with their existing mental model of where add-ons should be in the Smartsheet website.
Severity Rating: 4
All participants struggled to find the appropriate add-on and use it to complete the task. Overall, the placement of add-ons is not intuitive and users were unable to find it. The majority of participants tried multiple paths within the site to integrate Smartsheet with Gmail, including using personal settings, launchers, WorkApps, help pages, and resources.
Recommendations to resolve this issue:
Place the “add-ons” component in a distinct section away from templates and template sets
Provide more context for the purpose of add-ons
Signal to users that Add-Ons are an option within the Solution Center by increasing the section’s visibility
3. Users feel cognitively burdened while browsing in the Solution Center because of information overload and lack of visual contrast between different categories of templates/template sets
Severity Rating: 3
Most participants described feeling overwhelmed and frustrated at the first glance of the template gallery due to the homogenous composition of template and template set icons. A few of them struggled to identify different categories of templates, with two participants failing to identify the right category of the template to complete the usability task.
Recommendations to resolve this issue:
Prioritize information that helps users identify categories of templates, for e.g., Name, Description, Type etc.
Reduce the number of options for the users to choose from in the Browse All Solutions page and highlight sort and search functionalities.
Enhance visual contrast in the grid layout between similar-looking template and template set icons.
4. Users are unable to differentiate between templates and template sets while browsing because of lack of contextual information and visual contrast
Severity Rating: 3
Most novice participants and one experienced participant were unsure of the difference between the templates and template sets.The Solution Center has different types of assets to tailor various tasks. However, it is not clearly communicating what they are, what they do, and what the differences are between these assets
Recommendations to resolve this issue:
Add more descriptive explanations for each asset type
Guide novice users with a short tour on the different asset types
Add short videos on the differences between a template and template set
Redesign for Smartsheet 📊
Based on previous findings in the usability study, I decided to redesign parts of the Smartsheet template gallery where high-severity issues prevent users from completing tasks.
First, I redesigned to help users tell the difference between “templates” and “template sets” more clearly, especially for novice users. Second, I redesigned a smooth experience for users to open and use template sets, making the entry for initiating the use of “template sets” more intuitive and the following instruction page more pleasing to browse.
Indistinguishable Templates & Template Sets ❎
Most users don’t know the difference between templates and template sets at their first glance, and they could mistakenly open up “templates” when the tester asked them to open up “template sets”.
01. Using information icon to deliver details ✅
A. Information icon
B. A clear explanation
I designed the information icon next to the “templates” and “template sets”, prividing users especially novice users an extra bit of insights or knowledge. The information icon has a strong identity as the place to go when users want to know something. When users move their cursor to the information icon, a clear explanation of “template” or “template sets” pops up.
A overwhelming and confusing “Learn More” Page ❎
This is the page with most severe usability issues which brought both confusion and frustration to participants.
The key button on this page, the green download button, doesn’t look like a clickable thing at all to users.
All participants complained about the information overload they’ve encountered.
01. Make the download button more “clickable” ✅
Before
The original download button doesn’t look like a clickable thing for some users. They assume this as a non-interactive thing, and were hesitant to click on it.
After
Fixed the drop shadow and outside stroke to make it looks like a real clickable button.
02.minimize Users’ cognitive workload ✅
A. Accordion
B. 10 need-to-know for “template set” setup&customize
C.Accordion (after clicking)
The original setup & customize on this instruction page was extremely overwhelming. Shown in the video above, the condense and massive amount of text really scared all users away. Instead, I flexibly designed accordion to organize extensive content in a list of 10 need-to-know, minimized users’ cognitive load and helped them to read this instruction page more pleasingly.
Take Away
Give priority in usability issue with high severity level when you do redesign, but also make balance among all sorts of uncertainties when a sponsor get involved in. Some issues with higher severity level might not be the areas the sponsor want to address instantly due to certain constraints (eg, business strategy, product logistic, low frequency of usage)
