COMPANY
Intuit Mailchimp
YEAR
2023
ROLE
Senior UX Researcher
KEY SKILLS
Usability Testing
Synthesizing Data
Research Findings
Delivering Insights
Product Recommendations
Mailchimp: Navigation Redesign
Goal
Objective: To better understand if/how existing and prospective users were impacted by future state changes to the left navigation with the least associated risk. Our initial goal was to capture potential risks or impacts on key task flows and settle on our best first guess at a left navigation structure to build into a beta version in-app.
Hypothesis: We assume that the enhancements made to the left navigation will result in key payoff moments. Customers think of automations as an integral layer to their marketing strategy, not as a separate bucket of functionality within their navigation, so rather than highlighting this feature, placing it separately in the navigation creates more confusion about what it is.
Method
To mitigate risk, we decided to usability test before, during, and after the navigation redesign process, starting with usability testing 2 design concepts.
Here are some potential risks we are looking to avoid and ways to quantify those risks in dollars:
With the help of Chat GPT4*
1. DECREASED USER ENGAGEMENT
Risk: Poorly executed navigation redesign can lead to confusion, making it harder for users to find what they're looking for, potentially decreasing user engagement.
Quantification: If engagement drops by 10% and the average revenue per user (ARPU) is $15/month, with 1 million active users, the monthly revenue impact could be -$1.5 million (10% of $15 million).
2. INCREASED CUSTOMER SUPPORT COSTS
Risk: Users struggling with the new navigation might significantly increase support inquiries, driving up support costs.
Quantification: If customer support inquiries increase by 20% and each inquiry costs $10 to resolve, with an average of 10,000 inquiries per month, the additional monthly cost could be $20,000.
3. USER CHURN
Risk: Frustration due to navigation changes could lead to increased churn rates.
Quantification: If the churn rate increases by 2% from a base of 1 million users, with an ARPU of $15/month, the lost revenue from the churned users would be $300,000 monthly (2% of 1 million users * $15).
4. CONVERSION RATE IMPACT
Risk: New or modified navigation paths may inadvertently lengthen the journey to conversion or make it less intuitive, affecting sign-ups or sales.
Quantification: Assuming a 5% decrease in conversion rate from 2% to 1.9% on a baseline of 100,000 monthly sign-ups, with an average lifetime value (LTV) of $200 per customer, the impact would be -$20,000 ($200 * (1,000 fewer conversions).
Note: The dollar values provided are hypothetical and illustrate potential financial impacts.
A screenshot from my deck.
Key Questions
If we move ‘Automations' under a larger marketing bucket, do we slow users down on core tasks related to the customer journey builder?
If, instead, ‘Automations’ is a separate bucket, what types of L1s would they expect to see under this section? Why?
Where do users expect to see surveys, forms, and landing pages? How do they categorize and describe the purpose of each tool?
How do they think these features differ from one another? Do they think they have anything in common (if so, explain)?
Which bucket would they place each of these into?
Where do users expect to see transactional emails?
Do they consider this a “channel” or an “automation” type?
A screenshot from my deck.
Findings
We slow customers down trying to create marketing automations with ‘Customer Journeys’ bucketed under ‘Marketing.’
Customers want to see metrics for active marketing automations and audience growth.
Customers compare where landing pages, forms, and surveys live inside of Mailchimp to other marketing tools they use.
Customers use terminology like “trigger,” “one-off email,” and “automated email” in reference to ‘Transactional emails’.
FINDING 1: STARTING POINT TO CREATE A MARKETING AUTOMATION
Q: Do we slow users down on core tasks related to customer journey builder?
A: Yes, we slow customers down when they try to create marketing automations with ‘Customer Journey’ bucketed under ‘Marketing.’
Design Implication: How might we improve the time it takes customers to locate a starting point to create a marketing automation?
Design Exploration: Reduce cognitive load for our customers by eliminating questions like “Where did they put __________?”
FINDING 2: DETAILED METRICS FOR MARKETING AUTOMATIONS
Q: If ‘Automations’ remains a separate bucket, like in production, what do customers expect to see?
A: Customers do not intuitively think of ‘Customer Journeys’ bucketed under ‘Marketing’ when creating a marketing automation. Customers also emphasized wanting to see metrics for active marketing automations and audience growth.
Design Implication: How might we improve how we communicate metrics to customers who actively use marketing automations for personalized marketing jobs?
Design Exploration: Provide access to metrics that allow customers to take actionable next steps.
FINDING 3: MENTAL MODEL FOR LANDING PAGES, FORM, AND SURVEYS
Q: Where do users expect to see surveys, forms, and landing pages? How do they categorize and describe the purpose of each of these tools?
A: Unfortunately, we did not capture enough data on customers' mental models around landing pages, forms, and surveys. We did find customers comparing where landing pages, forms, and surveys live inside of Mailchimp to other marketing tools they use.
Design Implication: How might we improve customers' thoughts and feelings about landing pages, forms, and surveys as a marketing tool?
Design Exploration: Draw inspiration from other marketing tools to better understand customers’ mental models of landing pages, forms, and surveys.
FINDING 4: TRANSACTIONAL EMAIL
Q: Where do users expect to see transactional emails? Do they consider this a “channel” or an “automation” type?
A: Customers refer to transactional emails as a type of automation. We can then assume that customers bucket transactional emails under ‘Automations.’
Design Implication: Proposing ‘Transactional Emails’ continues to live nested under ‘Automations.’ How might we improve how we show the value of transactional emails to customers in Mailchimp to increase pay-off moments?
Design Exploration: Provide an overview of a transactional email before asking customers to demo or upgrade.
QTNA: Questions That Need Answers
Insights
‘Automations’ should remain in a separate bucket.
Customers do not intuitively think of ‘Customer Journeys’ bucketed under ‘Marketing’ when creating a marketing automation.
QTNA: What types of L1s would they expect to see under ‘Automations’?
Customers want to see metrics specific to their customer journeys: sends, opens, clicks, etc.
Journey Point Blocks demoed at Design All Hands Q2 FY23.
We did not capture enough data on customers' mental models around landing pages, forms, and surveys.
QTNA: Where do users expect to see surveys, forms, and landing pages? How do they categorize and describe the purpose of each of these tools?
Customers refer to transactional emails as a type of automation.
We can assume that customers bucket transactional emails under ‘Automations.’
There does not appear to be a detailed description of a transactional email and how it can result in a key payoff moment for our customers within the web app. This information is currently available in the Help Center and Transaction Email Features Page.
Recommendations
Based on the research findings, below are some of my product recommendations:
Reconsider the placement of 'Customer Journeys' within the 'Marketing' category as it is slowing users down when creating marketing automations. It might be more intuitive to categorize it separately or under a more relevant category.
Provide more detailed metrics specific to 'Customer Journeys' such as sends, opens, clicks, etc. This would cater to the users' need for detailed insights and could aid in better decision-making.
Carry out further research to understand customers' mental models around landing pages, forms, and surveys as the current data is insufficient. This would help in making these tools more intuitive and user-friendly.
Validate the categorization of 'transactional emails' under 'Automations'. As users view 'transactional emails' as a form of automation, it might be beneficial to maintain this categorization.
Enhance the description of a 'transactional email' within the web app, explaining its benefits and potential payoff moments. This could encourage more users to utilize this feature.
Seek user input on what categories they would expect to see under 'Automations'. This feedback could be used to customize the 'Automations' category to align with user expectations.
Look into how other marketing tools categorize landing pages, forms, and surveys. This could provide insights into users' expectations and help to refine the categorization of these tools in the product.
Consider incorporating user terminology like “trigger,” “one-off email,” and “automated email” in the design and labeling of features related to 'Transactional emails'. This could make these features more relatable and easier for users to understand.
Results
In business, pursuing one problem can lead to solving another. These insights further validated Mailchimp’s investment in providing more sophisticated marketing analytics to its users. These product enhancements were shipped in late 2023, resulting in improved user flow and decreased time to create marketing automations by 20%.