01 / Write the decision before launching the test.
“Which ad wins?” is incomplete. Define what you might change, whose behavior matters, the outcome that pays for acquisition, and the difference large enough to justify a decision.
A useful hypothesis: “For people considering this bottle, the commute demonstration will acquire new customers at a lower cost than the current product tour, without worsening refunds.”
Fictional. It names an audience context, a meaningful creative change, a comparison, and a quality guardrail.
Make the handoff explicit.
- Creator
- Defines the idea, rationale, material change, proof, and variants. Supplies correctly labeled assets.
- Media buyer
- Chooses the feasible design, eligibility, optimization event, tracking, budget, attribution settings, and analysis method.
- Business owner
- Confirms acceptable acquisition economics and customer quality. Approves the decision rule and scale constraints.
Copy the test plan
Use this as a starting document. Complete it with the person responsible for the next step.
CREATIVE TEST PLAN Test ID / owner / creator / approver: Business question: Falsifiable hypothesis and reasoning: Exploration, execution refinement, randomized A/B, or lift test: Primary outcome, denominator, and measurement source: Business target and how it relates to acquisition economics: Control / challenger / material change: Constants (offer, destination, tracking, audience eligibility, schedule): Optimization event / attribution window / reporting basis: Expected conversion and customer-quality delay: Baseline rate or outcome and relevant comparison period: Minimum worthwhile effect: Sizing method / confidence / power / multiple-comparison treatment: Planned sample, budget, duration, and feasibility: Decision rule (including inconclusive) agreed before launch: Safety stop conditions (tracking, spend cap, availability, harmful error): Quality guardrails (refunds, cancellations, qualified leads, margin): Platform delivery and randomization checks: Review date after outcome maturation: What each plausible result would change in the next brief:
How much spend? How long? When can we stop?
Start with the relevant baseline, the smallest improvement worth acting on, the expected variability, and the confidence and power appropriate to the decision. Use an experiment sizing method suited to the primary outcome; a cost-per-acquisition estimate is not simply a binomial conversion rate.
Translate the required information into a feasible budget and schedule. Allow for the conversion window, reporting lag, repeat customers, refunds, or lead qualification. Cover relevant buying-cycle variation without changing the test midway.
If the available budget cannot answer the question, simplify the comparison, accept that the work is exploratory, or postpone it. Do not manufacture certainty with a universal spend multiple, three-day rule, or fixed minimum conversion count.
Agree on the analysis date and a suitable fixed-horizon or valid sequential method. Checking for tracking failures is sensible; repeatedly checking conventional significance and stopping on the first favorable result increases false-positive risk. Define safety stops separately from performance conclusions.
Method reference: Microsoft’s guidance on during-experiment trustworthiness. Its principles concern controlled experimentation; applying them to an ad platform also requires checking how delivery and outcomes are measured.
02 / Choose a design that answers the question.
| Method | Useful for | What it cannot establish by itself |
|---|---|---|
| Concept exploration | Finding promising problems, promises, arguments, or demonstrations under actual delivery. | Unequal spend and exposure do not isolate why one ad performed differently. |
| Execution refinement | Improving a promising concept: opening, proof, talent, pacing, format, or length. | Changing several elements cannot identify the contribution of each one. |
| Randomized A/B comparison | Comparing defined alternatives under a preplanned allocation and measurement method. | It does not automatically prove a universal creative effect, or the incremental effect of advertising versus no advertising. |
| Lift / holdout experiment | Estimating incremental outcomes relative to an eligible no-ad or other defined control. | It does not answer every fine-grained creative question and may require substantially more scale. |
Several ads in an ad set are a useful way to explore delivery, but they are not automatically a controlled experiment. The platform can select different people and exposure for each ad. A low-spend variant may be underexplored rather than disproven.
Make the comparison fair without pretending it is perfect
For a controlled execution test, keep the offer, destination, eligibility, schedule, event, and reporting basis as consistent as the design allows. Define the material creative change in advance and check the actual platform allocation and delivery.
Random assignment strengthens inference about the tested setup. Optimization can still produce different exposure among people within each assigned group, so describe whether the result compares complete delivery strategies or isolates a narrower creative effect. Generalize cautiously to other placements, audiences, budgets, and periods.
For concept exploration, intentional bundles of differences are acceptable: a founder explanation and a product demonstration are different concepts. Record that you compared packages; do not credit the result to the opening word alone.
Setup reference: Meta: About A/B testing. Research context: Characterizing and Minimizing Divergent Delivery in Meta Advertising Experiments (2025 preprint). Platform tests and pure message-response experiments need not estimate the same thing.
Check launch readiness.
- The right conversion event fires, is deduplicated where needed, and can be reconciled with business records.
- Every cell has the intended asset, copy, offer, destination, and placement treatment.
- Eligibility, allocation, attribution window, dates, budget cap, and safety stops match the plan.
- Inventory, booking capacity, and landing-page availability support the test.
- The creator and buyer can explain what result would change the next brief.
03 / Read the business result, then investigate.
| Layer | Look at | Useful question |
|---|---|---|
| Delivery | Spend, impressions, reach, placement mix, allocation. | Was there enough relevant exposure to interpret anything? |
| Attention | Relevant video-start, retention, or completion measures. | Where might the story lose viewers? Do not compare incompatible format metrics. |
| Interest | Outbound clicks and landing-page views. | Did the ad motivate the intended visit, and did the destination load? |
| Acquisition | Purchases or qualified bookings; cost per agreed primary outcome. | Did it acquire the kind of customer the business needs at an acceptable cost? |
| Quality / economics | New-customer mix, order value, contribution, refunds, lead quality, cancellations. | Does the apparent result remain useful after downstream outcomes mature? |
Define denominators. For example, outbound CTR is outbound clicks divided by impressions; landing-page purchase rate is purchases divided by landing-page views from the same defined measurement basis. Do not mix all-click CTR, platform-attributed purchases, and unmatched analytics sessions into a causal explanation.
Common ways to misread a test
- Different attribution
- A click-only report and a click-plus-view report count different outcomes. Record the window and measurement source; distinguish event date from exposure or acquisition cohort.
- Conversion delay
- Recent spend can look weak before purchases or qualified leads arrive. Compare mature periods and avoid declaring a result while one cohort is still incomplete.
- Low volume and many variants
- A large percentage difference can come from a few events. Account for uncertainty and multiple comparisons. “Inconclusive” is a valid result, not proof of equality.
- A better hook, a worse customer
- More curiosity can create cheap visits that do not convert or qualify. Read attention alongside the primary outcome and quality guardrails.
- Fatigue
- A declining result may reflect repeated exposure, audience saturation, competition, seasonality, tracking, or a changed offer. Check those explanations before diagnosing the creative.
- A winner that does not repeat
- A selected winner can exaggerate its true advantage. Revalidate promising work with fresh evidence and monitor performance as spend and delivery change.
Copy the results-log schema
Use this as a starting document. Complete it with the person responsible for the next step.
test_id,asset_id,design,measurement_source,optimization_event,attribution_window,cohort_start,cohort_end,outcome_maturity_date,spend,impressions,outbound_clicks,landing_page_views,primary_outcomes,new_customers,qualified_leads,revenue,refunds_or_cancellations,primary_cost,uncertainty_or_test_readout,quality_guardrail,decision,next_brief,owner
Leave unavailable fields empty rather than inventing values. Add the platform experiment readout or uncertainty assessment to the log; the spreadsheet does not calculate significance for you.
A worked readout: more clicks is not the decision.
Fictional exploration · not a benchmark or significance claimTwo bottle concepts receive the same spend. The owner’s illustrative acquisition-cost target is $35. Outcomes use the same reporting window and have matured. This is observational exploration, not a randomized creative-effect estimate.
| Observed result | A / Commute demonstration | B / Curiosity reveal |
|---|---|---|
| Spend | $1,000 | $1,000 |
| Impressions | 20,000 | 20,000 |
| Outbound clicks | 200 | 400 |
| Outbound CTR | 1% | 2% |
| New-customer purchases | 25 | 20 |
| Observed cost per new customer | $40 | $50 |
Readout: B earned more clicks. A had the lower observed acquisition cost. Neither met the stated target, and this table does not establish a statistically reliable causal winner.
Decision: do not scale either on the strength of CTR. Reconcile customer quality and measurement, inspect the ad-to-page promise, and decide whether A warrants a more informative test.
Next brief: keep A’s commute concept and product presentation. Create E02 that answers the main verified buyer objection with relevant evidence, keeping the offer and destination consistent. The buyer defines a feasible validation plan before production. Do not claim the numbers alone proved which objection caused the result.
04 / Turn the result into the next assignment.
| Decision | When it makes sense | Next action |
|---|---|---|
| Scale | The business outcome and guardrails support more investment at the agreed level of confidence. | Increase investment deliberately; monitor whether economics hold as delivery changes. |
| Refine | The argument has promise and a specific unresolved weakness worth investigating. | Change a named element and write a new hypothesis. |
| Retest | Evidence is incomplete, inconsistent, confounded, or not repeatable. | Resolve the measurement or design problem; avoid a cosmetic rerun. |
| Retire | Sufficient evidence or a fundamental offer/proof problem no longer justifies the concept. | Record the context and reason so the same idea is not rediscovered without learning. |
Leave a useful learning note.
We expected… We changed… We observed… We can conclude… We cannot conclude… Therefore the next brief is…
Acquisition now, brand effects over time
Keep recognizable branding even while optimizing acquisition. Immediate attributed response and longer-term brand effects are different measurement questions; a short conversion window cannot describe all future demand.
Binet and Field’s IPA research examines that distinction. It is not a reason to apply a universal budget ratio here. Give acquisition tests their agreed business outcome, and use a separate plan when the question concerns brand memory or longer-term incremental growth.