Mission Growth

Painted door test: how many users before a click counts

A painted door test counts clicks on a feature that does not exist yet. Learn the six steps, how many users you need, and when a click is not enough.

Published Updated 22 min read
Painted door test as a door outline on a flat glass wall with click tokens at its foot, an emerald capsule lying across the base
On this page

A painted door test offers a feature, product or plan that does not exist yet and counts how many users act on it, so a team learns demand before it builds. The click is the cheap part. The hard part is deciding in advance what click rate means build, what means drop, and how many users have to see the door before either answer can be trusted.

What a painted door test is, and what three real tests decided

The door in a painted door test is a button, banner, menu item or landing page that promises something unbuilt; users who act on it see a short message saying the feature is coming, and the team counts the actions. Three tests from one company show what that count decided, and why a team needs its own bar.

Three names, one method

The same test travels under several names, and the placement is what separates them. A fake door test usually sits inside an existing product, so it reaches current users, who in our reading bring more intent and have more trust to lose.

ProdPad calls painted door tests the MVP of MVPs, and the order matters: a fake door checks demand before even a minimum viable version of the feature gets built.

Painted door testing usually sits outside the product, on a landing page, an ad or a mockup, and tests a standalone idea on people who do not know you yet. Talk of a smoke test product is the loosest usage, and "smoke test" also names a QA check that a new software build starts at all, which is a different job.

NameWhere the door sitsWhat it testsWhat it changes
Fake doorInside the productA new feature or toggle for existing usersHigher intent, higher trust cost
Painted doorOutside the productA standalone product or planColder audience, lower trust cost
Smoke testOften a landing pageWhether anyone shows interest at allAlso names a QA build check

In practice the terms overlap, and the reader who searches for a fake door vs painted door answer can treat them as one method with two placements.

The Solitaired hint button: over 15% clicked

Among fake door testing examples with published numbers, the Solitaired hint button is the cleanest case of a door that earned a build. Solitaired, a solitaire game site, wrote up three of its tests in a post dated 8 December 2020.

It showed a Hint button to 20% of its audience and explained, on click, that the feature was coming soon. Over 15% of that test group clicked, and the team spent two weeks building the feature.

The multiplayer button: fewer than 2% clicked

The multiplayer button is the test that saved a team from a large build. Fewer than 2% of the Solitaired user base clicked it, which the team read as a clear signal not to invest in multiplayer. Solitaired described building it as a considerable undertaking that would need a meaningful engagement gain to be worthwhile, so a low click rate and a high build cost pointed the same way.

The new-games test sits between the other two. Links to new games opened a pop-up saying the games were coming soon, 5% of users clicked to play, and the team called that material enough to build. Solitaired does not state the build cost for this one, so the 5% result shows where one team put its line while the work the games took stays unknown.

What the three results leave open

Solitaired's post states no bar. Over 15% built, 5% built, under 2% dropped, so any flat bar above 2% and up to 5% fits all three decisions (a range we derived from the three decisions, with no rate divided into another because the denominators differ: the hint test counts the 20% of the audience shown it, multiplayer the user base, games users).

The click rate separated the ideas. It did not tell the team where its own bar belonged, and that is the part the rest of this guide writes down first.

TestClick rateDecisionBuild cost named
Solitaired hint buttonOver 15% (shown to 20% of the audience)BuiltTwo weeks
Solitaired new-games links5%BuiltNot stated
Solitaired multiplayer buttonLess than 2%Not builtA considerable undertaking
Bar chart of click rates in three tests: hint over 15 percent built, new games 5 percent built, multiplayer under 2 percent dropped
Click rate in three Solitaired painted door tests and what each team did

When a painted door test fits and when it does not

The idea fits a painted door test when it can be shown as a button, the audience is large enough for the bar, and the click is a fair stand-in for the real action. Run this fit check before anything else: a demand validation test with too little traffic gives an answer that does not repeat, and Solitaired warns about exactly that.

Can the idea be a button?

The idea has to fit into one visible entry point, such as a menu item, a banner or a plan card. A feature whose value only shows up after weeks of use, or one that changes how the whole product behaves, gives a click very little to say. If you cannot describe the idea in one line a stranger would click on, test something smaller first.

Is traffic large enough for the bar?

Traffic is the first gate. Solitaired notes that a product with only a few hundred users after its first months may get a result that is not statistically significant, so a door shown to a few hundred users can produce an answer that does not repeat.

The sample table later in this guide turns that warning into a number you can plan around, and that number is a count of exposed users.

Is the click a fair proxy for the action?

A click on "Subscribe to Pro" is a weaker stand-in for paying than a click on a free export button is for using it. The further the real action sits from the click, in cost, effort or commitment, the more the click overstates demand. When the real action costs money, plan for a priced step.

Can you tell the user afterwards?

Every user who clicks has been promised something that is not there. If you cannot show a message on the first screen after the click, because the door lives in an email or a flow you do not control, the test has no honest way to end for that user.

QuestionYesNo
Can the idea be a button?ContinueUse another method
Is traffic large enough for the bar?ContinueWait until it is
Is the click a fair proxy for the action?ContinueAdd a priced step or use another method
Can you tell the user afterwards?Run itDo not run it

Three cases where you skip it

Skip it when the decision is already made and the test would only decorate it, when the door would sit on a critical path such as checkout or sign-up, and when the audience is too small to clear any bar.

In the first case you are paying a trust cost for a number nobody will act on. In the second a broken promise lands where users can least afford it. The third is the traffic gate again.

Pair the fit check with a test and learn cycle so a skipped door still ends in a learning plan.

Quadrant of traffic size and click fidelity routing to run, wait or use another method
Fit check: traffic against how fair the click is as a stand-in for the real action. Positions are illustrative

How to run a painted door test in six steps

Six steps run a painted door test: state the hypothesis, write the build bar and drop bar, size the exposure, place the door, show the debrief, and read the interval against the bar. The order matters. Write two bars before launch, a build bar and a drop bar, and read each bar against its own interval end rather than the observed rate.

The test is one kind of growth experiments entry, with the difference that its treatment is a feature that does not exist.

Step 1: State the hypothesis

Write one sentence naming the user, the action and the expected result, for example that users of an invoicing app will click a bulk-export item. One door tests one hypothesis. A door that bundles three ideas produces a click rate nobody can assign.

Step 2: Write the build bar and the drop bar

The decision card is a short page with two bars on it. The build bar is the click rate the lower end of the interval must reach before you build. The drop bar is the rate the upper end must fall below before you drop the idea. Write both before the first user sees the door, since a bar chosen after the count bends toward the result.

Step 3: Size the exposure

Use the sample table in the sections below to find how many users the bar needs, and set that number as the exposure cap. Solitaired showed its hint button to 20% of its audience, which is one way to cap it. The cap does two jobs at once: it gives the interval enough data, and it limits how many people meet a feature that does not exist.

Step 4: Place the door

Pick the placement that matches the hypothesis. A fake door in the product reaches users with intent and tests a feature. A painted door on a landing page tests a new product on a colder audience.

Keep it off the critical path, and keep the label honest enough that the click means what you think it means. Test the same label in one place only, because two placements mix two audiences into one click rate.

Step 5: Show the debrief

The first screen after the click says the feature is not built yet. A short message, a waitlist field or one survey question all work. Do not leave the user in a dead end.

Step 6: Read the interval against the bar

When the sample reaches the cap, stop and compute the 95% interval around the click rate. Build if its lower end clears the build bar. Drop if its upper end sits below the drop bar. Anything in between is no decision, and it earns a second test sized in advance.

Flow of six steps from stating the hypothesis to reading the interval against the build bar and drop bar, with both bars written at step two
The six steps in order, with the two bars written at step 2

Amplitude and Contentful already tell teams to set a pass threshold before launch, and Coursera tells them to size the test cohort. What the two-bar card adds is the pairing of each bar with one end of the interval, which the next sections explain.

Checklist of metric, build bar, drop bar, exposed users, stop date and exposure rule
The two-bar decision card fields to fill before launch

Worked example: a bulk-export button in an invoicing app

A bulk-export painted door in an invoicing app is a menu item shown once to a sampled slice of users, logged with three events, judged against a bar of 10%, and explained in one message. The product and every number below are illustrative inputs chosen for the example, not a benchmark and not a real result.

The spec on one page

Everything the team needs is decided in one page before the first user sees the door.

FieldValue
HypothesisUsers who send many invoices will click a bulk export item
PlacementA menu item labelled "Export all invoices" on the invoice list
Who sees itA random slice of active users, once each
Events loggedDoor viewed, door clicked, message dismissed
Bar10% (illustrative), read at 95% confidence
Exposure cap140 users, enough if the observed rate is 15%
Kill switchA flag that removes the item for everyone at once
Stop ruleThe sample reaches the cap, or a user complaint arrives first

At an observed 15%, 140 users give 21 clicks, and the Wilson lower end reaches the 10% bar. If the same test showed 15 clicks in 100 users, the interval would run from 9.3% to 23.3% and the lower end would sit under the bar, so the result would be no decision.

The message after the click

The exact copy is part of the spec: "Bulk export is not available yet. We are checking whether it is worth building, and your click helped. One question: what would you export first?" It says the feature does not exist, thanks the user and asks one question, and the kill switch above is what lets a team end the test the moment the message draws complaints.

How many users before a click rate means anything

A click rate needs an interval around it, because 15 clicks in 100 exposed users could be anywhere from 9.3% to 23.3%, and the Wilson interval is the one to use at these sizes. The observed 15% looks safely above a 10% bar. Its lower bound does not.

Why the observed rate misleads

A rate from a small sample is an estimate. Agresti and Coull showed in 1998 that the common Wald interval performs poorly unless n is quite large, and that the score (Wilson) interval can be recommended for nearly all sample sizes and parameter values. A painted door lives in the small-sample range, so the Wilson interval is the right default.

We computed the figure above from 15 clicks and 100 users, which are assumed inputs for illustration, with the standard 95% Wilson formula.

CaseExposed usersClicksObserved rateAgainst a 10% bar
Small sample1001515%Interval 9.3% to 23.3%, lower end under the bar
Sized for 15%1402115%Lower end at or above the bar
Sized for 12%87510512%Lower end at or above the bar
Sized for 20%35720%Lower end at or above the bar

Download CSV (CC BY 4.0)

The lower bound is the number to judge

Judge the lower bound of the interval against the build bar. The observed rate is a point inside a range, and building on a point that the range puts under your bar is a bet you did not agree to. For the interval arithmetic on store conversion, the low conversion rate guide covers the Wilson basics and the compare-the-end rule.

Sample size for a single proportion against a bar is a different question from the two-arm effect in a minimum detectable effect calculation, because a painted door has no control group.

The go/no-go table: set the bar before you count

A go/no-go rule compares the interval with two written bars: build when the lower end clears the build bar, drop when the upper end falls below the drop bar, and between them make no decision and plan a second test. Against a 10% bar an observed 12% needs 875 users, 15% needs 140 and 20% needs 35.

Reading the interval against two bars

Interval positionReads asAction
Lower end at or above the build barDemand clears the barBuild
Upper end below the drop barDemand misses the barDrop
Interval straddles the barsThe test could not separate themNo decision; plan a second test

The middle row is where teams cheat. A result that straddles the bar gets a second test sized in advance, and the first test is never extended. Peeking at a growing sample until the interval clears the bar breaks the interval's meaning, and the sequential testing guide explains why for A/B tests. A painted door has no control to compare against, so the interval is read once at the pre-sized sample.

Users needed against a 10% bar

Observed click rateExposed users neededClicks at that size
12%875105
15%14021
20%357

Download CSV (CC BY 4.0)

These are computed from the assumed 10% bar and the 95% Wilson formula: for each rate, the smallest n at which that rate is a whole click count and the Wilson lower end is at least 10%. They are illustrative arithmetic on assumed inputs.

A stronger click rate needs far fewer users, which is why a door on a big idea can finish quickly and a door on a modest one needs a lot of exposure.

Bar chart of exposed users needed to clear a 10 percent bar: 875 users at an observed 12 percent, 140 at 15 percent and 35 at 20 percent
Users needed to clear a 10% bar at three observed rates, whole click counts

A click is the weakest signal: climb the ladder

A click is the cheapest signal a door can collect, an email costs a little more, a clicked price costs a choice, and a payment costs money, and the decision should name the rung it needs. Pick the rung by the cost of being wrong. This signal ladder also answers whether a waitlist is a strong validation signal: it sits on the second rung.

RungWhat the user givesWhat it can decide
ClickOne tap on a labelWhether the idea draws curiosity
EmailAn address for a launch noticeWhether interest survives one more step
Priced plan clickA choice among pricesWhich plan to build first
PaymentMoneyWhether to commit to the build
Table of four signals from a click to a payment, showing what each costs the user and what decision each can support
Four signals, what each costs the user and what it can decide

Rung one: the click

A click costs the user almost nothing, so it carries the least information. It is the right rung for a small feature with a cheap build, where being wrong costs a sprint. For anything larger, treat the click as a screen that decides which ideas earn a second step.

Rung two: the email

An email asks the user to hand over an address for a launch notice. It costs a little more than a click, and users give it for many reasons, including politeness. Count it as a stronger screen than a click, and still short of a commitment.

Rung three: the priced plan

Buffer ran this rung as a priced plan step. In a post dated 16 February 2011 its founder described a landing page, then a pricing page added in between, then an email form. The extra page was one more click before the email, and it tested the pricing by detecting which plan was clicked, and the demand, since users had to be keen.

Buffer's result: people kept giving their emails and a small number clicked paid plans, and after that the founder started building the first minimal real version. The priced plan is the rung where a door starts to look like a purchase decision.

Rung four: the payment

A payment costs real money, which makes it the strongest signal and the one with legal limits for goods. The section on the FTC order rule below covers when you can take it. Pick this rung only when a wrong build is expensive and a priced plan click has already cleared its bar. Pricing questions beyond demand belong with pricing experiments, which cover price tests and what you can legally charge.

Why stated interest overstates demand

Stated willingness to pay runs above real payment in the literature: a mean calibration factor of 2.60 from a skewed distribution whose median is 1.35, so a free click should be read as a likely overstatement. That is hypothetical bias, and it matters here because a free click is hypothetical in the same sense: nobody paid.

What the meta-analysis found

Murphy, Allen, Stevens and Weatherhead pooled 28 studies and 83 observations in a UMass Amherst working paper from June 2003. Individuals are widely believed to overstate their valuation of a good by a factor of two or three. The median ratio of hypothetical to actual value was 1.35, and the distribution had severe positive skewness. The mean calibration factor was 2.60.

Bias also rose with the hypothetical value: the paper's regression predicts a calibration factor of 1.84 at the median hypothetical value and 2.59 at the mean, and private goods showed lower bias than public goods.

One limit applies. These studies value goods in surveys, and no click-through is among them, so the numbers describe stated payment in valuation studies and say nothing exact about a button.

What it means for your clicks

Because the bias grows with the stated value, one borrowed haircut misleads. Cutting every click rate by 2.60 would be too harsh on a small idea and too kind on a large one. Put the priced second step first on the ideas with the highest click rates, and measure your own click-to-pay ratio there.

If your ratio turns out flat across click rates, one haircut is enough; if it moves with the click rate, the paper's pattern holds in your product.

Charging money on a fake door: what the FTC order rule says

A free click raises no order-rule question, but a priced pre-order for merchandise in the United States needs a reasonable basis to ship within the stated time and a delay-or-refund offer if you cannot; the FTC also describes a dry test path with disclosures.

This section covers soliciting orders for merchandise from a United States seller. It is not legal advice, and it does not extend to free clicks or to sellers elsewhere.

Path one: take the order and be able to ship

Under the order rule in the Code of Federal Regulations, a seller may not solicit an order for merchandise by mail, internet or telephone unless, at the time of the solicitation, the seller has a reasonable basis to expect it can ship within the stated time.

If shipping slips, the seller must offer the buyer, clearly and without prior demand, a choice to consent to a delay or to cancel, with a prompt refund on cancellation.

Path two: run a dry test

The FTC's business guide describes dry testing for merchandise that may never ship. In all promotional materials the merchant must disclose all material aspects of the promotion, including the fact that the merchandise is only planned and may not be shipped. Without that disclosure the order rule applies in full.

Where the rule stops

The rule covers merchandise. The FTC business guide says it does not cover services, so a software feature door, which sells nothing yet, is a different question from a pre-order for a physical product. Check the rules that apply to your product and your market before you take any money.

Two-path flow from order to shipping or refund offer, and the dry-test disclosure path
The two paths for merchandise: ship in time with a refund path, or dry-test with disclosure

Protect user trust: exposure cap and debrief message

A painted door protects trust when each user meets it at most once, the sample stops at the size the bar needs, and the first screen after the click says the feature is not built yet. Overusing fake doors or placing them on critical paths frustrates visitors, and practitioner opinion on whether they erode confidence is mixed.

The split resolves by what the door asks for: a free click is cheap to forgive, and a taken payment brings the order rule above.

Show each user the door once

Log who has seen the door and suppress it afterwards. A user who meets the same missing feature twice has been promised it twice. One exposure per user also keeps the click count honest, since repeat clicks would inflate the rate.

Stop at the size the bar needs

The sample table sets the exposure cap, so doing the maths also limits the harm. At a 15% reading against a 10% bar the cap is 140 users, and every user beyond it adds exposure without adding information. Keep the door off the critical path as well, so a user who is trying to finish a task never meets it.

Say it is not built on the first screen

The debrief message is the first thing a user sees after the click. It names the missing feature, says the team is checking demand, thanks the user and asks one question. Keep that to one question, because a long survey after a broken promise doubles the irritation. Never let the click land on an error page, since that reads as a bug and teaches users to distrust the product.

Checklist of three trust rules for a painted door: one exposure per user, a cap at the sample size, and a debrief on the first screen
Trust rules: one exposure per user, cap at the sample the bars need, debrief on the first screen

In short, a painted door click is curiosity until you give it an interval and two bars, and the decision rests on the lower end against the build bar and the upper end against the drop bar.

Before the next door goes live, write the decision card for the one idea at the top of your backlog, set its exposure cap from the users-needed table, and put the debrief message on the first screen.

Frequently asked questions

Is a waitlist a strong fake door signal?

A waitlist email is a moderate signal. It costs the user more than a click and less than a priced choice, so it sits on the second rung of the ladder. Buffer put a pricing page before its email form, so the plan that was clicked showed willingness to pay as well as interest, and that ordering is what makes a waitlist worth reading.

Should I charge money in a fake door test?

Charge only when the decision needs the payment rung and you can meet the rules. A free click is a measurement. For merchandise in the United States, taking an order needs a reasonable basis to ship within the stated time and a delay-or-refund offer, and a dry test must disclose that the product is only planned and may not be shipped.

Is fake door testing legal?

A free click on a button raises no order-rule question. The FTC order rule covers soliciting orders for merchandise, and the FTC business guide says it does not cover services. A pre-order for a product that may not ship is where disclosure and refund duties begin, so check your own product and market.

Can a fake door test be misleading?

Yes, in two ways. A free click is a hypothetical signal that overstates demand, and an observed rate hides an interval, so 15 clicks in 100 users spans 9.3% to 23.3%. Read the lower end against a bar written beforehand, and tell the user on the first screen that the feature is not built.

How many fake doors should I build?

Build one door per hypothesis. The number of doors a site can run at once is its eligible users in the window divided by the users one door needs, which is 140 for a 15% reading against a 10% bar. A site with too few eligible users should run doors one after another and wait for each result.

Painted door test vs fake door test: what is the difference?

They are the same method with different placements. A fake door sits inside an existing product and tests a feature for current users, while a painted door sits outside it, on a landing page or an ad, and tests a new product. Either way the click rate needs an interval and a bar written beforehand.

Cite this page

Aktaş, F. (2026, November 15). Painted door test: how many users before a click counts. Mission Growth. https://missiongrowth.io/blog/painted-door-test

Download the data: 2 tables as CSV (CC BY 4.0)

Figures we made for this post are free to reuse under CC BY 4.0 with credit to Mission Growth.

Up nextPricing Experiments: How to Run One Without a Backlash
14 min read
Next step

Put these playbooks to work

Start with a free audit. See where the lift is before you commit.

How it works

  1. 01

    30-minute audit call

    We map your funnel against your goal and pull live data from your channels.

  2. 02

    Lift estimate

    You get a written estimate of where the lift is, with a 30-day plan to capture it.

  3. 03

    You decide

    Run it with us, run it in-house, or shelve it. No commitment from the audit.