At eBay, the clue was a line that went up just as another came down. The company had stopped buying search ads on versions of its own name on Yahoo and MSN. Paid clicks vanished. Traffic from the unpaid search results rose almost immediately. People had come looking for eBay. Once the sponsored link disappeared, they found another way in. 1

That is an awkward result for anyone who has to defend an advertising budget. The old reports could show a click, a purchase and the time between them. All three observations could be correct. The question for the person writing the cheque is how many purchases would disappear if the cheque stayed in the drawer.

A famous name changes the experiment

Economists Thomas Blake, Chris Nosko and Steven Tadelis examined several eBay search experiments. In the short term they found no measurable benefit from buying the brand's own search terms. Ads on searches for other words did influence new and occasional customers. Frequent customers, however, accounted for much of the expense, and the estimated average return on those ads was negative. This was a marketplace that customers already knew how to find. A new hotel down a side street has good reason to keep reading before switching anything off. 1

Later, Andrey Simonov, Chris Nosko and Justin Rao studied experiments on Bing involving thousands of brands. Without competing ads, buying a brand term produced a small gain in clicks, generally smaller for larger brands. In one group of brands that stayed out of the auction despite competition for their names, rivals captured between 18% and 42% of clicks. Those are clicks. The study cannot tell a hotel how many of those people would complete a booking, or how many would book through a reseller and cost it a commission. 2

A hotel can start with its own name. Who else appears when a traveller searches it on a phone? Where does each link take the traveller? Does the hotel have empty rooms to fill, or does it want more guests to book direct? Winning a new guest and moving an existing guest from a commissionable channel into a direct booking are both valuable. They belong on different lines in the budget.

Two figures that can both be true

Consider a fictional merchant running one campaign. Before it begins, the merchant randomly assigns 20,000 eligible people to two equal groups. The campaign can reach one group and is withheld from the other. People remain in their assigned group whether or not an ad reaches them. The merchant can count, in each group, everyone who places an order during the agreed window. For this example each buyer places one order.

During the test Campaign available Campaign withheld
People assigned 10,000 10,000
Buyers 360 300
Share who bought 3.6% 3.0%

The estimated gain is 60 buyers among the 10,000 assigned to the campaign. The difference is 0.6 percentage points, or 20% relative to the baseline of 3.0%. Suppose the platform credits itself with 180 of the orders and each order is worth €80. On €2,000 of advertising, its report shows €7.20 in attributed sales per euro spent. The experiment implies €4,800 of additional sales, or €2.40 per euro spent. The platform can recognise people who interacted with an ad. Random assignment gives the merchant a way to estimate how many purchases the availability of the ad changed.

Now bring in the cost of fulfilling an order. Suppose €30 remains on each sale after the goods, shipping subsidy, payment fees and expected returns. Sixty extra orders produce €1,800 towards the €2,000 campaign. On those first orders, the best estimate is a €200 loss. The campaign would need at least 67 additional orders at that margin to pay for itself. Repeat orders could change the calculation. Give those customers time to return before spending their future margin.

There is uncertainty even in this tidy example. A simple approximate 95% interval for the difference between these two independent purchase rates runs from about 0.1 to 1.1 percentage points. That works out to roughly 10 to 110 additional buyers in the campaign group. The interval includes the 67 needed to break even. The experiment leaves room for a profitable campaign and a costly one. Its precision matters more to the next decision than the pleasing round figure of 20% lift.

What a smaller advertiser can actually test

Start with the decision that the result could change. For a business with two equally busy locations, assigning one shop to each condition might feel scientific. A burst pipe, a local festival or a particularly good sales manager could account for the whole difference. Dividing a single shop's Monday and Saturday takings is even less informative. Ask whoever designs the test how many independent units there are and what size of gain the design could realistically detect.

Google offers Conversion Lift studies using eligible groups of users or geographic areas. Its documentation says access varies by account. For geography studies the platform generally needs more budget and advises waiting until the study ends to interpret results. Decide the outcome and the time window before spending. If bookings are the outcome, count cancellations consistently. Keep prices and stock under observation. A sold out weekend can hide the value of a perfectly good ad. [3]4

Sometimes the honest answer is that a single shop cannot buy a decisive experiment at its current volume. Randall Lewis and Justin Rao examined 25 large advertising experiments, most reaching millions. Even at that scale, the median width of the confidence intervals for return on investment exceeded 100 percentage points. 5 A small operator can still decide how much uncertainty it can afford. Separate the budget spent reaching new buyers from the cost of appearing above resellers. Check the booking ledger. Set a cash limit. Record which assumption would warrant spending more.

At the next budget meeting, put two numbers beside the reported return. How many additional orders would cover this campaign? How many does the evidence suggest it produced? In our example the answers are 67 and about 60, with considerable uncertainty. The owner can now decide what it would cost to narrow that gap, and whether the decision is worth it.