A floating laptop with an insurance marketing product on the screen.

Insurance Funnels A/B Testing

2019-2021
QuoteWizard by LendingTree

Progress Indicator

My role
Design, testing, front-end development.
The team
Funnel Optimization Director
Data Analyst

Problem

When entering the primary insurance product funnels, people said they were unsure of what the time commitment of the process would be. Drop-off increased with each additional question even when asking a simple yes/no question and we hypothesized it was due to frustration with not knowing where they currently were in the process.

CHALLENGES

  • The exact number of questions is not known at form load due to several conditional questions. For example, current insurance provider is only asked if someone answers that they are currently insured.
  • Each of the five major product funnels have ten to twenty question trees for different sources, sub-products, or experience types.
  • Future tests may change the order of questions or question groups.
  • 100+ brands white-labeling our form with different colors and backgrounds.

Process

After research on progress indicators I determined that there were two types that would work within our technical limitations and provided clear visual feedback as users moved through the form. The first was a stepped indicator that grouped the form questions into content-based groups—for auto insurance those groups were "vehicles," "drivers," and "contact." The second was a slim line showing estimated percentage complete that lengthened with each question submitted. Both were flexible enough to work across multiple products and could be defined within the JSON that controlled the order of questions.

A variety of sketches showing different layouts and types of progress indicators.
Sketches for possible progress bar styles and placements.

Each approach had clear pros and cons. The stepped indicator provided a very clear sense of place and allowed users to get an idea of what was to come, but did not change until the entire group of questions was complete which felt discouraging. The slim linear indicator provided visual feedback after every question but did not provide any contextual information on what the process would entail.

A mobile screen with a three stepped progress indicator.
A mobile screen with a thin linear progress bar.
Mockups for the two styles of progress bar for mobile devices.
A desktop screen showing a product with a three stepped progress indicator.
A desktop screen showing a product with a thin linear progress bar.
Mockups for the two styles of progress bar for desktop and tablet browsers.

TESTING

Only testing could prove which was the appropriate solution for our product. A single iteration of each type was chosen and we designed a multivariate test that would test how each one affected question completion rate and full-form conversion rate. We started the experiment on 15% of all auto traffic, split evenly between stepped, slim, and the control with no progress indicator.

Variations test
Auto Insurance Funnel
Stepped Style
+0.54%
Form conversion rate
Slim Style
+1.27%
Form conversion rate

The slim style won by a wide margin. In order to validate our results, we then removed the stepped option from the test and the increase in conversion rate held. The next phase of testing upped the eligible population to 100% of all auto traffic across all brands with half seeing the slim bar and half seeing no progress indicator. After auto, we slowly phased the testing in across all the full funnel products.

Validation test
All Product Funnels
Auto
+1.31%
Form conversion rate
+$2.4M
Estimated annual revenue
Home
+5.89%
Form conversion rate
+$1.6M
Estimated annual revenue
Health
+3.27%
Form conversion rate
+$1.1M
Estimated annual revenue

We felt the data further supported our original hypothesis, and planned several further iterations on the progress bar in the future in order to provide more clarity about where the user stood in the entire insurance comparison process.

Top Listing Emphasis

My role
Design, testing, front-end development.
The team
Funnel Optimization Director
Data Analyst

PROBLEM

Although the primary purpose of the product was to display multiple insurance providers so users could compare them, there was a not insignificant percentage of cases where only one of our advertisers were able to write a policy for that person. Users were not typically happy with this result and often did not progress further to the provider’s site. Single advertiser results averaged a 20% click rate as opposed to a 35% click rate on the first position advertiser when multiple options were presented.

CHALLENGES

  • Only one template for listings existed as a string template with little flexibility.
  • Advertisers uploaded their own logos to the ad portal so resolutions and sizes varied wildly.

PROCESS

The horizontal listing layout worked well when users were presented with three or more potential insurance providers. When only option was present, people were sometimes confused and a couple thought they needed to wait for more to load until they saw the "1 match found" copy at the top of the page. My primary goal was to make it as clear as possible that this was the only provider they were matched with through our service, without implying that it was the only provider possible for them. After several iterations going through rapid user testing using employees not working on the funnel, it appeared that "best match" had the highest positive reaction. The simplest change to the layout also yielded the best result. Stacking the content vertically took up more of the empty space and was visually consistent with the current listing experience. This proposed visual change would also be simple to implement and could be replicated on other products as the A/B test progressed.

An ad with the advertiser logo and buttons next to each other on a phone.
Original single listing result
An ad with the advertiser logo and buttons stacked vertically on a phone.
Proposed single listing result

TESTING

This test was created to increase advertiser click rate for people who made it all the way through the form and only matched with one advertiser, which was only about 3% of total traffic. Our standard process would create a test population too small to draw reliable conclusions from, so we decided to run the test at a 50-50 split for 100% of those eligible users.

Validation test
P&C Products
Auto
+0.71%
Click rate on single result
+$2.5M
Estimated annual revenue
Home
+0.43%
Click rate on single result
+$0.8M
Estimated annual revenue

Once this test was complete, we decided to next test using this layout for the top listing when multiple providers were presented. That test was still in progress but the preliminary results were positive at the time I left the Funnels Optimization team.

Progressive Rate Display

My role
Requirement definition, design, testing, front-end development.
The team
Funnel Optimization Director
Engineering Manager
Full-stack Developer
Data Analyst

Problem

Progressive had been our largest advertiser for auto and home insurance for years, but had never permitted us to show their rates. After the advertiser team built up the relationship over several years, they finally agreed to test displaying the user's estimated rate at the end of our funnel before they clicked through to Progressive's site. We knew that not displaying rates caused a lot of confusion for our users, so we were excited to finally test providing that information.

CHALLENGES

  • This functionality could be used in the future for other advertisers and products.
  • The current layout of the listings at the end of the funnel did not support additional information outside of provider and a link out to their site.
  • Progressive's rate estimation API had an average response time of 20 seconds.

PROCESS

Once we had final confirmation from Progressive on what information they would be providing us via their API and what we would need to display in order to be compliant with both legal departments, I worked with the engineering team on nailing down how we would pass the data back and forth with Progressive in order to create as smooth an experience as possible. The trickiest part was agreeing on how we could delay the bulk of the load since we would have to wait for the response from Progressive and at that time we did not show the final listings page until all information was returned from our advertiser API. We did not want to leave the user in a loading limbo with no feedback for more than a couple seconds, much less twenty, and we could not change the interstitial content without affecting other experiences and products as it was a shared state.

A diagram of the data flow between funnel, advertiser, and Progressive.
Diagram of the proposed data flow between the funnel, our ad API, and Progressive's rate API.

The engineers got to work building a new rate-fetching microservice, and I started on sketching what the end state of the funnel would look like. I had to juggle what people using our product wanted, what our advertiser Progressive wanted, and what the legal and compliance teams required. After sketching, I presented the team and Progressive with several wireframe variations for each possible result.

Wireframes of showing rates and buttons to apply for home insurance.
The first round of proposed wireframes for showing insurance rates on the listings page.

Once everyone agreed on a direction, I worked on a set of high-fidelity mockups for further approval as well as built a simple interactive prototype to use in testing. The biggest question was—how long would people stand waiting for the rate to be calculated? 

TESTING

During a quick round of user testing with the interactive prototype, multiple people thought the prototype was broken when the load time was so lengthy and said they would likely have abandoned the site. While this particular part of the project was out of our control, we brought this up to the advertiser in the hopes that they might prioritize optimizing the speed of their API. Outside of load times, we heard much of the same feedback as we did during the single listing emphasis tests. QuoteWizard's products claimed to help people easily and quickly compare insurance companies and yet they were only seeing one provider option. People felt it was a bait and switch, and even though they were pleased to see a potential rate it wasn't what they had originally set out to accomplish.

An animation of the loading screen before showing an insurance rate and buttons to view the plan.
A recording of the prototype with the average load time expected before showing rates.

The development of this project took a good bit longer than anticipated due to authentication and security concerns, but once complete and fully tested in the stage environment we began the A/B test between the rate display experience and the standard single listing experience. The population for this experience was very small, so we started with a 50-50 split on eligible users. The click rate on the Progressive ads was trending upwards in the test group and we were waiting on updates from Progressive on the click-to-bind rate when I left QuoteWizard.