CRO experiment - "Just booked" A/B test
PRODUCT
I am currently working on a two-sided marketplace in the long-term rental property market, where students who are planning to study abroad can get in touch with landlords in foreign countries and book their properties.
PROBLEM
To book a property a user has to go through the usual funnel:
Search Results Page > Product Page > Booking form > Thank you Page
Like most marketplaces, we are constantly looking for ways to improve the click-through rate from one step to the next and the conversion rate overall.
Booking a property on this platform is not a trivial decision, nor a cheap one - it may cost several hundred euros to make a booking and to a certain extent it qualifies as a "life" decision. For a lot of our users it's the first time they will be studying and living abroad, so they want to make sure that they'll be living in the right place.
The consequence for us is that some of our users can be a bit indecisive and take their time to book a property - sometimes too much time - and the booking ends up not happening (because someone else booked the property or it was booked externally on another platform).
This is an outcome we obviously want to avoid, so we were looking for ways to "spur" indecisive users into action.
IDEA
One CRO "trick" you see from time to time is to tell the user that someone else - someone they can identify with - has completed the action you want them to complete, in the hope that they will follow suit.
To pursuade indecisive users to book, I wondered whether we couldn't apply this method. My first thoughts were to show users a pop-up message every time someone else made a booking. Something like:
This would rest on the following principles:
- the scarcity principle => if so many people are booking, I'd better hurry up and book myself before it's too late
- social validation => if someone else has booked, I guess it's safe for me to book as well
- excitement => a reminder of the exciting times ahead of them where they'll meet international students
After realising that we probably couldn't use the photo and name of our users so easily, I changed the proposal into this :
EXPERIMENT
To decide whether this was a good idea or not, I planned the following experiment. During three months, twenty percent of our users would be shown this message at random intervals on the product and checkout pages. If the relative uplift in conversion was more than 10% I would keep on experimenting further and maybe involve the development team. Otherwise I would abandon the idea.
Control: users would see the page as it was with no changes
Target: users would be shown at random intervals a "Just Booked" message
Traffic: 100% of web traffic would be split into 80% control and 20% target
Metric: conversions, booking of a property
Duration: 3 months, enough to get the views necessary for the experiment
Target: users would be shown at random intervals a "Just Booked" message
Traffic: 100% of web traffic would be split into 80% control and 20% target
Metric: conversions, booking of a property
Duration: 3 months, enough to get the views necessary for the experiment
As I was using an external testing tool (Google Optimize), to avoid making calls to the database to listen to the latest booking events I decided to use fake but realistic data instead. I took some of the bookings from the previous days and used their data as static data to populate the messages.
This felt a bit like black-hat ux as users wouldn't actually be told about real bookings, but I rationalized it by thinking that if this experiment failed, the equivalent with more realistic data would also fail.
VISUAL SOLUTION
The final implementation in Google Optimize looks like this. It's triggered in javascript after a random delay after pageload. Animation is css only.
RESULTS
After 3 months the results came in:
Control had 219997 pageviews with 960 conversions
Version A had 54826 pageviews with 237 conversions
Chi-squared = 0.2008682373 and p=0.65 hence "Not significant"
As the pop-up message has not generated a statiscally significant increase in conversion I decided not to pursue it further.
