AB Test Cases AB Test Cases
  • About
  • Support
  • All cases
  • E-commerce
  • Landing pages
  • Lead generation
  • SaaS

Changing error messages (split account or pasword message) reduces failed logins

Mailchimp
Login page
October 2012
-66%
failed logins
Control and variation

-66%

A
B
Mailchimp is a popular email marketing service with over 7 million users. When reviewing their analytics they found that they had 340,591 failed login attempts in 1 month (from April 12 to May 12, 2012). That’s the total number of times someone couldn’t remember their username and/or password, or simply mistyped.

The team watched several login attempts and thought that the error messaging could be improved.  The old message “Your username or password is incorrect” would leave many options for the user to try. Suppose a users has 4 possible usernames and 4 possible passwords – that would be 16 options in total. A more specific error message ‘We could not find an account with that user name” would leave only 3 options.

Hypothesis

 Improved error handling – splitting password vs account name error – will reduce the number of failed logins.

Results

The rest ran from June 12-July 12 and the failure rate reduced around 60%.

Learnings

Error messaging can have a big impact on form completion. Note that there is a security tradeoff. Generic messages also makes is harder for hackers to guess usernames and passwords.


Source:  Mailchimp

Test details

Test element Copy
Page type Login page
Test date October 2012

Website information

Company Mailchimp
Market Marketing, Software (SAAS)
Test result
Test element Copy
Page Login page
Result -66% failed logins

Test Cases from Mailchimp

Adding social login options to the signup form
Mailchimp
+3.4% revenue
Changing error messages (split account or pasword message) reduces failed logins
Mailchimp
-66% failed logins

Leave a Reply

Click here to cancel reply.

You must be logged in to post a comment.

© 2025 AB Test Cases | Made with in Amsterdam
Privacy | Terms of Use | Cookies

Create your free account

Sharing experiments and learning about optimization
First Name
Last Name
Email
Password

  • Explore 100's of experiments for free
  • Filter by industry or type of page
  • Search all test cases
  • Share your experience
Already have an account? Login here