Small Changes, Big Impact: A Recent Success Story with Bella+Duke
This is the most significant difference we’ve observed between two variants during an A/B test, resulting from a remarkably small change.
Sometimes, achieving significantly better results doesn’t require major changes — just a few smart adjustments. That’s exactly the story behind a recent use case we concluded with one of our clients, Bella+Duke.
Bella+Duke is the UK’s #1 rated pet food subscription business. Founded in 2017, they create preservative-free recipes tailored to each pet’s individual needs, with a strong focus on improving pet health through natural nutrition.
The use case we’re discussing today began with a fairly common request: to create a new email collection banner for the homepage.
We received a Figma file with the final design from the client and implemented it through Bloomreach Engagement.
There were several elements we really liked about the initial design. Not only were we collecting valuable zero-party data (specifically, whether the customer had a cat or a dog), but we also implemented a “sticky email collection button.” This feature allowed users to easily return to the email pop-up, even if they initially chose to close it.
However, we quickly encountered a problem. Despite these features, the email collection strategy generated a very low number of signups—only about 0.19% of anonymous homepage visitors converted to email subscribers. This result was significantly below expectations. The client asked us to investigate potential issues, and we responded by designing a simple A/B test that was ready within just 15 minutes.
The email pop-up banner was not displayed automatically. Instead, users had to manually click the “Get Free Treats” button located on the left side of the screen to access the email sign-up form.
This led us to a simple hypothesis:
What if we display the pop-up automatically after, say, five seconds? Would that improve conversion rates?
In addition to testing this change in behavior, we also decided to experiment with different copy within the pop-up window. Ultimately, we set up three variants:
Variant A (Control Group): Users had to click the "Get Free Treats" button to open the pop-up with the original copy.
Variant B – The email collection pop-up automatically launched after five seconds if the website visitor did not click the “Get Free Treats” button. The pop-up also featured updated copy.
Variant C – Users had to click the “Get Free Treats” button to access the pop-up, which featured updated copy.
To be honest, we expected the email pop-up banner to improve signup rates — but we didn’t anticipate the difference would be this dramatic.
We quickly reached statistical significance with the following result:
Automatically triggering the email pop-up after five seconds (Variant B) generated 41 times more email subscribers compared to the identical version that required users to manually click the "Get Free Treats" button (Variant C).
Variant A (the original version) and Variant C (which differed only in the copy) did not produce significantly different results. At least, we did not reach statistical significance to confidently conclude that a difference existed.
Do Pop-Ups Negatively Affect Conversion Rates?
One common concern with pop-ups is that they might frustrate customers, causing them to leave the website and ultimately harming the overall conversion funnel.
In the context of Bella+Duke, a crucial step toward conversion is the creation of a “Nutrition Plan,” where customers receive a personalized recommendation for their pet food subscription based on their pet’s specific characteristics.
Before reaching the nutrition plan, visitors must complete a nine-step questionnaire that gathers detailed information about their pets.
What surprised us was that visitors who saw the email collection pop-up triggered automatically were actually more likely to complete the full questionnaire compared to those who interacted with Variant A or Variant C.
Although this difference has not yet reached statistical significance, it’s important to highlight that we observed no negative impact on Bella+Duke’s overall conversion funnel — which was the primary concern we set out to monitor.
Continuous A/B Testing
We view A/B testing as an ongoing process for continually improving the eCommerce digital experience. After learning that the automatic pop-up banner delivered significantly better results for Bella+Duke, we were excited to see that the client is eager to continue testing and further optimize the experience.
The next A/B test we will be launching focuses on the creative of the email pop-up banner. Both variants will still trigger automatically after five seconds; however, this time, the visual and messaging of the pop-up will differ.
Which version do you think will win?
Thank You, Bella+Duke Team!
We would like to sincerely thank the Bella+Duke team for their trust and openness in exploring different use cases.
In particular, we want to express our gratitude to Sam Norris, an outstanding CRM Manager. It’s truly a pleasure to collaborate with Sam on brainstorming and testing new ways to personalize the customer experience for Bella+Duke — all made possible through Bloomreach Engagement.
I would also like to thank our Datacop teammate, Samuel Babic, for his outstanding work on this project.
Thank you, and we look forward to launching many more exciting use cases together!
If you found this post valuable…
We hope this article was valuable to you. If you enjoyed it, don’t miss out on future articles—subscribe for free and have our latest content delivered straight to your inbox.
If you'd like to discuss any aspect of eCommerce marketing operations with us or explore potential collaboration, feel free to schedule a meeting using the link below: