Conversion Rate Optimization Case Studies: 14 CRO Cases

Conversion rate optimization case studies report a percentage, and the percentage is a claim about one of four different things. Before borrowing a fix, make sure your own rate is truly low with a noise test for a low conversion rate.

Every conversion rate case study shares that problem, A/B testing case studies from vendors included. Conversion rate optimization (CRO) is the practice of raising the share of visitors who complete a goal by testing changes against the current version. A case study only helps you if you know which of the four it is.

We traced 14 named cases back to the pages that published them. Most of the lifts hold up as ideas to test. None of them holds up as a number to forecast.

If you want the how-to rather than the evidence, our Shopify conversion rate optimization guide covers the process. This post is for the moment before you put a borrowed tactic into your backlog.

In this guide:

• What the percentage on a case study can measure, with 10 original cases audited
• What Walmart Canada and Venture Harbour, the two cases an AI Overview shows first, actually report
• What four original pages leave out of the headline
• Why lifts shrink when a test is repeated
• A six-check list for deciding whether to copy, test first, or ignore

A conversion rate optimization case study's headline lift measures one of four things: a single test, a program total, a rebuild result or a calendar comparison. Even the single test lift transfers only as a hypothesis, never as a size.

Here is how the four kinds differ:

A single test lift compares one variation with a control, like BluTV's 19-day test with 89,290 visitors. A program total adds many tests together: Flos USA ran 30 campaigns over 17 months.

A rebuild result bundles changes, as Walmart Canada's did. A calendar comparison sets one period against another, like Going's 104% month-over-month rise, which Unbounce's page reports for a two-variant CTA test without an arm-to-arm result. For the first kind, a direction is worth testing. For the other three, you get a hypothesis and no size.

To see what it measures in practice, we read the 10 original VWO case studies on VWO's own pages. Every conversion case study there is a customer story, so the table holds what each headline says, what the figure covers and what the page states about duration and visitors:

Download CSV (CC BY 4.0)

The counts are plain. Of the 10 pages, 5 headline a single test (ArchiveSocial, Bear Mattress, BluTV, Hubstaff, Paltalk). Four headline a program: Flos USA and Restroworks by their text, plus RuneScape and Truckstop, whose page titles say "structured CRO program" although each figure is one test's result. One, IMB Bank, attaches its figure to a product that no described test touched.

On the evidence side, 6 of the 10 pages state a duration for the test behind the number. Flos USA and Restroworks give program spans, so we did not count them. Visitors appear on 2 pages, BluTV and Hubstaff. A text search of all 10 pages finds no significance level, which says what the page text contains and nothing about whether the vendor computed one.

So 4 of the 10 pages give no test duration at all. For the wider argument about which CRO tactics have solid evidence, see our post on CRO best practices. This page stays with the case pages themselves.

Walmart Canada's own June 2014 release gives a conversion rate of more than 20 percent in the first four weeks after launch, which is a rate and not a 20% lift, and a 98% rise in mobile orders. Venture Harbour's 743% is almost five years of changes on one form.

Google's AI Overview for this query leads with the Walmart example:

Boosted overall conversions by 20% and mobile orders by 98%.

Walmart Canada's release says something different: "Walmart.ca drove a customer conversion rate of more than 20 percent in the first four weeks after the launch." That sentence names a rate. It states no baseline and no definition of the rate, so it cannot be read as a 20% improvement.

The 98% has the same problem. The release counts "a 98% increase in mobile orders" since going live in October, and orders are not conversion. The same sentence also reports a 36 percent decrease in average page load times and a three point lift in customer satisfaction.

That makes the rebuild a bundle. The release lists a unified cart, localization and live inventory beside the redesign. It mentions no A/B test, no removed product details, no sample and no duration.

What can you copy? A hypothesis: hiding the action on unavailable items should reduce friction. The size of that effect is unknown, because nothing on the page isolates it.

Venture Harbour is the cleaner number and the longer window. Its enquiry form went from 0.96% to 8.1%, which is 7.14 percentage points, across almost five years of form analytics and ongoing changes. The company says it still runs CRO tests on the form.

The only short-window figure is the more than 300% rise in the first week after adding a multi-step form. That form tool, Leadformly, is discontinued, and the page keeps the case as a historical example. So the copyable line is the multi-step form as a hypothesis. The 743% belongs to the calendar.

The original pages carry detail the headline drops: Hubstaff reports a 7% trial-to-paid drop and then says paid subscriptions were not affected, IMB Bank's 87% is a home loan rate, Paltalk headlines 11% rather than 57.32%, and Truckstop's baseline was 64.96%.

Read the denominator and the funnel stage before the number, then check who measured each stage.

Hubstaff: a downstream drop the page itself disputes. The homepage test ran three months across about 259k visitors on two pages and won with a 49% increase in sign-ups. Then the page says Wingify reporting showed a 7% drop in free trial-to-paid conversion.

Because the in-app experience was the same for both homepage versions, the page says the drop could not be attributed to the new design alone. Additional analytics tools corroborated that paid subscriptions were not affected.

The only figure the page attributes to the test is the 49%. Presenting the 7% as the redesign's effect would contradict the page. A downstream metric one tool reports can be unconfirmed by another tool on the same page, so define the event that closes each stage before you compare rates, as conversion funnel optimization does.

IMB Bank: the 87% belongs to a different product. The page lists a home loan conversion rate up 87% and a personal loan figure of 36%.

Both described tests ran on the personal loan form, one on its first page and one as an exit pop-up. The test results the page gives are 9% more users completing the form and 52% more users saving the application. No listed test result equals 87%, and no described test touched home loans.

Paltalk and Bear Mattress: two honest metrics, one headline. Paltalk's page headlines an 11% conversion lift. Its test 3, which ran 30 days, shows 11.32% conversion and a 57.32% rise in average revenue per user (ARPU). Bear Mattress reports 24.18% more successful purchases and 16.21% more revenue from a 19-day, 50-50 split test. In both cases the smaller metric is the one to plan around.

Truckstop: a big percentage on a high base. The broker pop-up test took conversion from 64.96% to 81.71%. That is 16.75 percentage points, or 25.8% relative (16.75 divided by 64.96), which the page rounds to a 26% uplift. A pop-up that already converts nearly two in three visitors gives you a very different test than one starting at 2%.

Published A/B lifts can shrink on replication because results get picked after they reach significance and because small tests exaggerate effects. A 2026 replication of 8 tests on 4 UI patterns found only 2 significant in the expected direction.

The first mechanism is selection. A 2021 KDD paper on post-selection bias notes that when we interpret A/B tests we typically focus only on the significant results and take them at face value. That practice inflates point estimates, a pattern often called the winner's curse. A case study is the extreme version, because only the winner gets written up.

The second is power. A lift from a test with too little traffic can only reach significance when it overshoots the true effect.

The 2026 KDD paper by Lukas Vermeer and co-authors puts numbers on it. It tested these patterns:

• Rounded buttons
• Page performance
• A coupon-code field
• A sticky call-to-action

Each test had a median of 2.4M users and 80% power. Across all eight replications, estimated effects were substantially smaller than previously claimed. Two were significant in the expected direction and one was significant in the opposite direction.

Its authors believe it is more likely that many prior estimates came from underpowered experiments (power below 50%), which exaggerate treatment effects. Keep the scope in mind. Even at this scale the team lacked power for revenue per user and purchase conversion, so it used surrogate metrics: click-through rate, add-to-cart rate and capped add-to-carts. The paper says nothing about any of the 14 cases here.

What it gives you is a rule. When a page never states the power of its test, take the direction of the lift and discard its size. Treat the number as the upper end of a range. For the base-rate argument behind that stance, our post on conversion rate optimization best practices already makes the case.

A lift that survives checking looks like Bing's. An ad headline test raised revenue by 12%, which triggered a "too good to be true" alert within hours, and the idea was one of hundreds the program managers had deemed low priority. Microsoft's HBR article on online experiments describes the alert, and the result was investigated before anyone believed it.

Google Optimize case studies cannot be reproduced in the tool, which ended on September 30, 2023, and Unbounce case studies are customer stories written around its own builder, so the tool is part of the claim.

Google's support page states that Optimize is no longer available and that any experiment still active on that date ended. Anything you read from an Optimize-era test is a hypothesis you rebuild elsewhere. The same applies to Venture Harbour's Leadformly case above.

For Unbounce case studies, the customer story is also a vendor's page. Going's 104% month-over-month rise in premium trial start rates after a CTA text change came from a test of two button labels, but Unbounce reports it month over month, not as one arm against the other. The tactic, a shorter and clearer button label, is worth testing. The 104% tells you nothing about your own page.

Here is the decision rule:

1. Check tool status. If the tool is gone, keep the hypothesis and drop the tool.
2. Check the vendor. A vendor's customer story is evidence that the product works, and the lift is a secondary claim.
3. Rebuild in a current tool. Our roundup of conversion rate optimization tools compares what you can run the test in today.

A case study earns a place in your backlog when it passes six checks: the lift's kind, a baseline in points, visitors and duration stated, a significance level stated, the downstream metric reported and the tool still available.

Run them in order:

1. Kind. Is the headline a single test lift, a program total, a rebuild result or a calendar comparison? Only a single test lift can earn more than a hypothesis.
2. Baseline in points. Convert the headline to percentage points on a stated base. Truckstop's 26% is 16.75 points on 64.96%.
3. Visitors and duration. Does the page state both? Only 2 of the 10 pages state visitors. Compare the visitors against your own traffic with a minimum detectable effect calculation before you plan the same test.
4. Significance level. None of the 10 pages states one. Without it you cannot tell a clean win from a noisy one, and you cannot check split integrity such as sample ratio mismatch.
5. Downstream metric. Does the page report the next funnel stage? Hubstaff reports it and then disputes it, which is better than silence.
6. Tool. Does the tool still exist, and is the story written by a vendor?

Before you start, ask who published it. A vendor's or agency's case page is marketing for the vendor, so every lift on it was chosen from a larger set.

The checks end in one of three verdicts:

No case earns "copy the number". The direction of a change can transfer, and its size cannot.

A case study also has a place in a CRO marketing strategy: it is an input to your hypothesis backlog, written down without the borrowed number. Research, prioritization and test cadence belong to the CRO guide, not to the case.

When the test you copied is a page change, treat it as a landing page optimization task: change one element, measure it against your own baseline, and ignore the original's size.

The next step is concrete. Pick one case study from your own backlog and run all six checks on it today. If it fails them, you still have a hypothesis, and that is what a case study is good for.