Half your visitors, two prices: what a Black Friday price test really costs
2026-11-02
Shopify can now split a discount test 50/50. The arithmetic is easy. The fairness, the feeds and the October timing are where it goes wrong.
- Ecommerce
- Shopify
- Pricing
- Black Friday
Shopify can now split a discount test 50/50. The arithmetic is easy. The fairness, the feeds and the October timing are where it goes wrong.
What Shopify changed on 2 October
On 2 October, Shopify added discounts to Rollouts, its native tool for scheduling and testing store changes. You can release an offer to part of the online store's traffic and expand it later, test it against your current offers and read the conversion results, or schedule a discount to go live with the theme and checkout changes it belongs to. A rollout comes in three types. A Launch is permanent. An Event runs for a set period and rolls back on its own. An Experiment splits eligible visitors between a control and a treatment, 50/50 by default, ends 90 days after it starts unless you change it, and needs the Grow plan or higher.
The discount mechanics are worth reading slowly. In an Experiment with a discount change, the control is simply your store's usual discount behaviour, and the treatment sees the selected discounts switched on or off. A discount keeps its own active dates alongside the rollout, so a code that is already live for everyone stays live unless the rollout switches it off for its group. Catalog changes work in a similar way: setting a catalog to active publishes its prices to the visitors inside the rollout.
The part I'd put in front of every trading team sits on a separate help page. An Experiment, or an Event that reaches less than 100% of visitors, changes the online store only. External channels receive no rollout information, so ads, shopping listings, product feeds and third-party apps carry on showing the usual product details. Shopify lists what visitors can run into as a result: a price that differs from the ad, a different discount, different shipping information. It also warns that channels such as Google and Meta can delist products or suspend an account for mismatched offers.
None of that makes the feature a bad idea. Testing an offer in October, when a wrong guess costs a few days of margin on part of the traffic, beats finding out on 27 November that the offer doesn't work. My concern is narrower. A price test is a different animal from a layout test, and the tool treats them the same way.
Why customers judge a price split by a different rule
When half the visitors see a new product gallery, no one feels cheated. When half see £20 off, and half don't, there's a victim, and that victim works it out the instant two people are comparing their baskets. Judgements about price fairness are comparative. The principle of dual entitlement was set out by Kahneman, Knetsch and Thaler in the American Economic Review back in 1986. That paper is 40 years old, so I've checked it against more recent work rather than just relying on it.
Haws and Bearden (Journal of Consumer Research, 2006, so 20 years old) tested dynamic pricing directly across three studies. Consumers judged a price unfair when it differed from what other consumers paid and they had no part in setting it, and they saw price changes within short periods as more unfair than changes spread over longer ones. A 50/50 split is the extreme case of both conditions: a different price for another shopper, at the same moment.
The newer literature points the same way. Hufnagel, Schwaiger and Weritz (Journal of Business Research, 2022, so just over four years old) ran three online experiments on personalised pricing in an ecommerce setting and found negative attitudes and behaviour from disadvantaged customers and, less obviously, from favoured ones too, driven by perceived unfairness. Pluut and Heidary (Journal of Revenue and Pricing Management, published December 2024) found that people judge personalised prices on distributive fairness, what I paid against others, and on procedural fairness, how the price was set, with consumer cynicism sharpening the reaction.
I have to be straight about a gap here. These studies test prices personalised on customer data, and I couldn't find a recent peer-reviewed study that tests a purely random A/B price split. The practical problem is that customers can't see the randomisation. From the outside, a random split and a profiled price look identical, which is exactly what happened to Amazon in September 2000. It ran random discounts of 20% to 40% on 68 DVD titles for five days. Shoppers compared prices online, many assumed the differences were tied to their purchase history, and Amazon apologised, refunded 6,896 customers an average of $3.10 and promised that any future test would give every buyer the lowest test price. That episode is 26 years old, and screenshots now travel in minutes rather than days.
The favoured half deserves a thought as well. A customer who got the lower price has just learned that your price moves. Persuasion knowledge, the working understanding shoppers build of how sellers try to influence them, makes the next full-price visit feel like the unlucky draw, and the reference price for that product quietly drops. That fits Hufnagel's finding that even the winners react badly, and it lands on full-price sell-through after the test has ended.
The ad says one thing, the page says another
The second cost is the mechanical one, and it hits much quicker. As the external channels get no rollout information, a shopper can click on a Shopping ad which is showing the standard price and end up on a page in the treatment group which is showing a different price, or get to the cart and find a different discount to the one the ad had suggested. The behavioural effect is expectation disconfirmation (the page has broken the promise of the ad). If the price on the page is higher, the customer just sees it as bait and walks away. If it's lower, then the customer is happy, but you've paid for the click under one offer and converted it under a different one, which completely messes up the read on the test.
The platforms police this. Google's Merchant Center help says Googlebot crawls landing pages and compares the price in the product data with the price on the page and in structured data, and that a product with a mismatch may be disapproved. Shopify's help page goes further and warns of delisting or account suspension on Google and Meta. Its own suggestions are blunt: run the change at 100%, pause the ads on the affected products, or take those products out of listing channels while the rollout runs.
Commercially, that is a poor trade at this time of year. In the businesses I've run, Shopping campaigns carried a big slice of peak revenue, and a disapproval on a hero product in the fortnight before Black Friday costs more than most October tests could ever earn back. Excluding advertised products from the test removes the risk, but it also removes the very products whose price response you most wanted to learn.
What an October test can't tell you about Black Friday
Suppose the test runs cleanly. It still only answers a much more limited question than most people think it does. It tells you how October visitors responded to two offers in October. Black Friday traffic is a very different audience. It's a lot more deal seekers (coming in from voucher and comparison sites), a lot more gift buyers, a lot more lapsed customers who have been woken up by email. All of them are going into a market full of competing discounts. Their price sensitivity in that mix is going to be different to your October baseline, and I wouldn't be inclined to bet on the direction.
The metric is as important as the sample. Shopify's changelog refers to looking at the conversion results, and the point is that a deeper discount will almost invariably win on conversion. The real question is what the margin per visitor is for each arm once you've taken into account the customers who would have bought at full price anyway (they're all a pure subsidy). A treatment which takes conversion from 2.0% to 2.2% and at the same time reduces gross margin per order by a fifth will actually make less money per visitor (2.2 x 0.8 = 1.76 vs 2.0) and so will lose, even though the dashboard will declare it the winner. Those numbers are purely illustrative and are not from any study at all, the point is the arithmetic. Score the test on gross profit per session (you'll almost certainly have to work that out from outside the rollout report).
Then there is pull-forward. A returning customer who receives a discount in the second week of November and buys has often just moved a purchase she was planning for Black Friday week. The treatment arm shows extra conversions while your peak week quietly loses them. If the test sample is heavy with your most engaged customers, expect the October result to flatter the offer. The Shopify help pages I read also don't say how a visitor who switches from phone to laptop is assigned, so if that matters to your design, put the question to Shopify before you start.
How I run it before 27 November
My first rule is to test offer mechanics and leave the shelf price alone. 20% off against free next day delivery or a gift with purchase against a spend threshold gives you a real commercial answer, and both groups still see the same product price in the feed and on the page. The feed mismatch issue largely goes away, and the fairness comparison becomes "they picked a different deal" rather than "they paid less for the same jumper". You'd only ever change the price itself via a Launch or an Event at 100% (Shopify say that reaches checkout, POS and the Google and YouTube feeds, as well as the store).
If you really do need to test price, then borrow Amazon's 2000 rule and agree it internally before the test starts. Anyone who buys in the more expensive arm gets the difference back when the test is over. That turns an unknown reputational risk into a known cost. With an £8 gap and 1,500 expected orders in the higher price arm, the worst case is £ 12k. That's a nice round figure you can hand to the finance director and say, " here's the cost of this test ", and then go on to talk about what the test will teach you. It also gives customer service a very clear answer when that first screenshot comes in.
The regulatory thrust is to explain price differences. In the EU, the direction is set by Article 6(1)(ea) of the Consumer Rights Directive (added by Directive (EU) 2019/2161 and applicable from 28 May 2022), which requires traders to inform consumers when a price has been personalised on the basis of automated decision-making. Recital 45 puts it in the context of profiling, so whether a completely random split would fall within that is not clear, and I'm not a lawyer. In the UK there is no equivalent disclosure obligation; the test is the general one in s225 and s226 of the Digital Markets, Competition and Consumers Act 2024 (the DMCCA), i.e. whether the practice is likely to mislead and is likely to influence the average consumer's decision. On 28 October, the US FTC issued an order to nine companies (including Walmart and Target) to explain how personal data is used to determine individual prices.
For a UK retailer that also sells into the EU, my working assumption is simple: a price difference you can't explain to a customer in one sentence is one you shouldn't be running. Rollouts is a good tool, and native discount testing removes a fair amount of app clutter. Used on offer structure, with margin per visitor as the yardstick and the feeds left alone, it will make Black Friday planning sharper. Used to put two price tags on the same product for a month, it buys a conversion number with the one thing every promotion depends on, which is customers believing your price.