How to Create an A/B Test

Think of it this way: An A/B Test is a controlled experiment where you compare two or more variations to determine which performs better. You test different approaches (pricing, promotions, messaging, displays) and let data guide your decisions.

The Big Picture

A/B Test (Controlled Experiment)
├── Test Setup
│   ├── Test Name
│   ├── Hypothesis (what you're testing and why)
│   ├── Test Type (Pricing, Promotion, Messaging, Product Placement)
│   └── Status (Planning, Running, Completed, Inconclusive)
├── Test Variants
│   ├── Variant A (Control or Option 1)
│   │   ├── Variant Name
│   │   ├── Description
│   │   ├── Traffic % (how many see this version)
│   │   └── Campaign Link (optional)
│   └── Variant B (Option 2)
│       └── [same fields as Variant A]
├── Test Period
│   ├── Start Date & Time
│   └── End Date & Time
└── Results (Auto-populated)
    ├── Per-Variant Metrics
    │   ├── Conversion Rate
    │   ├── Total Sales
    │   ├── Customer Count
    │   ├── Average Order Value
    │   └── Confidence Level
    └── Winning Variant (determined by results)

What is A/B Testing?

A/B Testing (split testing) is a scientific method for comparing options:

The Process:

Create hypothesis: "Changing X will improve Y"
Design variants: Version A vs. Version B (vs. C, D, etc.)
Split traffic: Direct percentage of customers to each variant
Run test: Collect data over time period
Analyze results: Determine statistical winner
Implement winner: Roll out best-performing variant

Common Uses:

Pricing: $49.99 vs. $45.00 vs. $50.00
Promotions: "20% off" vs. "Buy 2 Get 1 Free"
Messaging: "Free Shipping" vs. "Fast Delivery"
Product Placement: Endcap A vs. Endcap B vs. Feature Wall

Step 1: Test Setup

Test Name

Clear, descriptive name for this experiment.

Good Examples:

"Pricing Test - Premium Sneakers $120 vs $99"
"Promo Comparison - Percentage Off vs BOGO"
"Messaging Test - Free Shipping vs Fast Delivery"
"Display Location - Endcap vs Feature Wall"
"New Product Launch - Price Point Test"

Avoid:

"Test 1" (not descriptive)
"Price test" (which products? what prices?)
"June experiment" (testing what?)

Naming Conventions:

Start with test type: "Pricing Test", "Promo Test", "Display Test"
Include what you're testing: Product, message, location
Include variants briefly: "$99 vs $120", "20% vs BOGO"
Be searchable: Someone should understand it at a glance

Hypothesis

What you're testing and what you expect to happen.

Format: "If we [change], then [expected result] because [reasoning]"

Good Examples:

"If we price premium sneakers at $99 instead of $120, then sales volume will increase by 30% because the lower price point reduces purchase hesitation."
"If we promote 'Buy 2 Get 1 Free' instead of '20% off,' then average order value will increase because customers buy more items to qualify."
"If we display new arrivals on the feature wall instead of side endcaps, then product discovery will increase by 40% because the feature wall has 3x more foot traffic."
"If we emphasize 'Free Shipping over $50' instead of 'Fast Delivery,' then conversion rate will improve because free shipping removes a purchase barrier."

Weak Hypotheses:

"We want to see if a lower price sells more" (no specific numbers, weak reasoning)
"BOGO vs percentage off" (not a hypothesis, just a comparison)
"Testing feature wall" (no expected outcome)

Components of Strong Hypotheses:

What changes: Specific variation
Expected outcome: Metric and direction (increase/decrease)
Magnitude (if possible): By how much (%, units, $)
Reasoning: Why you believe this will happen

Test Type

Category of experiment.

Options:

Pricing

Testing different price points for products.

Examples:

$99 vs. $120 vs. $149 (three-way pricing test)
$49.99 vs. $50.00 (psychological pricing)
Premium vs. value pricing strategies

Metrics to Watch:

Units sold
Total revenue
Profit margin
Customer segments (who buys at each price)

Promotion

Testing different promotional offers.

Examples:

"20% off" vs. "Buy 2 Get 1 Free" vs. "$10 off $50"
Free shipping vs. percentage discount
Promo code usage (SAVE20 vs. FREESHIP)

Metrics to Watch:

Conversion rate
Average order value
Promo code redemption rate
Margin impact

Messaging

Testing different wording, copy, or value propositions.

Examples:

"Free Shipping" vs. "Fast Delivery" vs. "Arrives in 2 Days"
"Limited Time" vs. "While Supplies Last" vs. no urgency
Product descriptions (features vs. benefits)

Metrics to Watch:

Click-through rate
Conversion rate
Time on page
Customer feedback

Product Placement

Testing different physical locations or display methods.

Examples:

Endcap vs. Feature Wall vs. Shelf
Eye level vs. lower shelf
Front of store vs. back
Grouped by color vs. grouped by size

Metrics to Watch:

Product discovery rate
Time to purchase
Units sold per location
Foot traffic patterns

Status

Current state of the test.

Options:

Planning

Test is being designed
Variants not yet created
Hypothesis being refined
Resources being allocated

Actions: Set up test parameters, create campaigns, prepare tracking

Running

Test is live and collecting data
Customers are experiencing variants
Metrics being tracked
No changes should be made

Actions: Monitor (don't interfere!), ensure tracking is working, watch for anomalies

Completed

Test period has ended
Data collection finished
Results analyzed
Winner determined (or not)

Actions: Analyze results, document findings, implement winner, plan follow-up tests

Inconclusive

Test completed but no clear winner
Results not statistically significant
External factors contaminated results
Need to re-run or redesign

Actions: Review what went wrong, redesign test, extend time period, or abandon approach

Step 2: Test Variants

Create 2 or more variants to compare. Each variant represents one option in your test.

Variant Name

Short label for this option.

Good Examples:

"Control" (baseline/current state)
"Variant A - $99"
"Variant B - $120"
"20% Off"
"BOGO"
"Feature Wall"
"Endcap Display"

Naming Patterns:

Control + Variants: "Control (current)", "Variant A", "Variant B"
Descriptive: "Low Price Point", "High Price Point"
Option-based: "Free Shipping Emphasis", "Fast Delivery Emphasis"

Description

Detailed explanation of what this variant entails.

Good Examples:

Pricing Variant: "Premium sneakers priced at $99 with standard product description and imagery. No promotional messaging. This is the lower price point to test price sensitivity."

Promotion Variant: "Buy 2 Get 1 Free promotion applied to all apparel. Customers must add 3 items to cart to qualify. Promo code BOGO3 auto-applies. Messaging on product pages and at checkout."

Display Variant: "New arrivals displayed on main floor feature wall (Zone 12). Products arranged by color gradient, left to right. Signage: 'JUST ARRIVED' above display. Eye-level placement for hero products."

What to Include:

Specific details of the change
Context (where, when, how applied)
Any supporting elements (signage, messaging, promo codes)
Differentiation from other variants

Traffic Percentage

What percentage of customers/traffic see this variant.

Format: Number (0-100)

Common Splits:

Two-Variant (A/B):

50% / 50% (equal split, standard)
90% / 10% (safe rollout, protect against bad variant)
80% / 20% (cautious testing)

Three-Variant (A/B/C):

33% / 33% / 34% (equal three-way)
50% / 25% / 25% (control + two tests)
60% / 20% / 20% (conservative with two risky variants)

Important:

Total must equal 100%
Larger samples = faster statistical significance
Smaller splits protect revenue if variant underperforms

Example Traffic Allocation:

Test: "Pricing Test - $99 vs $120 vs $149"
Variant A ($99): 50% traffic
Variant B ($120): 30% traffic
Variant C ($149): 20% traffic

Reasoning: $99 is expected winner (gets most traffic),
$120 is current price (baseline), $149 is premium test
(smallest risk exposure)

Campaign Link (Optional)

Reference to a promotional campaign this variant is tied to.

Links to: Promotional Campaign document

Use When:

Variant includes a promo code
Testing different campaign approaches
Linking merchandising to marketing
Coordinating multi-channel tests

Example:

Variant A: "Summer Sale 2025 - 20% Off"
Variant B: "Summer Sale 2025 - BOGO"
Both link to "Summer Sale 2025" campaign

Step 3: Test Period

Start Date & Time

When the test begins collecting data.

Planning:

Warm-up period: Consider excluding first hours (system stabilization)
Start day: Avoid starting mid-week (get full week cycles)
Timing: Start early in day to capture full day's traffic

Examples:

Start of business: "2025-06-01 09:00"
Monday morning: "2025-06-05 00:00" (capture full week)
Campaign launch: "2025-07-04 06:00" (holiday sale start)

End Date & Time

When data collection stops.

Duration Guidelines:

Minimum:

Online: 1-2 weeks (depends on traffic volume)
In-Store: 2-4 weeks (slower data accumulation)
Seasonal: Full season cycle if testing seasonal displays

Factors Affecting Duration:

Traffic volume: High traffic = shorter needed
Purchase frequency: High frequency = shorter needed
Variant difference: Subtle changes need longer tests
Statistical confidence: Aim for 95%+ confidence

Examples:

High-traffic online store: 7-14 days
Medium-traffic retail: 3-4 weeks
Low-frequency purchases: 4-8 weeks
Seasonal display: Full season

Avoid:

Tests during holidays or anomalous events
Tests that span major calendar changes
Too-short tests (lack statistical significance)
Too-long tests (market conditions change)

Step 4: Test Results (Auto-Populated)

These fields are read-only and populated by your analytics system.

Per-Variant Metrics

Conversion Rate (%)

Percentage of customers who made a purchase after seeing this variant.

Formula: (Purchases / Visitors) × 100

Example:

Variant A: 1,000 visitors, 45 purchases = 4.5% conversion
Variant B: 1,000 visitors, 62 purchases = 6.2% conversion
Winner: Variant B (higher conversion)

Total Sales

Total revenue generated by this variant during test period.

Example:

Variant A ($99 price): $4,455 (45 units × $99)
Variant B ($120 price): $7,440 (62 units × $120)
Winner: Variant B (higher revenue despite lower price)

Customer Count

Number of unique customers exposed to this variant.

Used for: Understanding sample size and statistical significance

Average Order Value (AOV)

Average amount spent per transaction in this variant.

Formula: Total Sales / Number of Orders

Example:

Variant A (20% off): AOV = $68
Variant B (BOGO): AOV = $92
Winner: Variant B (customers spend more with BOGO)

Confidence Level (%)

Statistical confidence that the result is not due to chance.

Industry Standard: 95%+ confidence to declare a winner

Interpretation:

95%+: Strong confidence, safe to implement
90-95%: Moderate confidence, consider extending test
< 90%: Insufficient data, inconclusive

Winning Variant

Auto-determined based on your success metric (conversion, revenue, AOV, etc.).

Displayed as: Variant name that performed best

Example: "Variant B - $120"

Common Questions

"How many variants can I test at once?"

Technically: Unlimited

Practically: 2-4 variants maximum

Why Limit:

Sample size dilution: More variants = less traffic per variant = longer test
Complexity: Harder to manage and analyze
Statistical power: Need larger sample for significance

Best Practice:

Start with 2: Control vs. one variation (classic A/B)
Then 3: Control vs. two variations (A/B/C)
Rarely 4+: Only with very high traffic

Sequential Testing Alternative: Test A vs. B first, then winner vs. C, then winner vs. D.

"How long should I run a test?"

Depends on:

Traffic volume (high traffic = shorter test)
Purchase frequency (frequent = shorter)
Margin of difference (large differences detected faster)
Statistical confidence needed (95%+)

Minimum Sample Sizes:

At least 100 conversions per variant
At least 1,000 visitors per variant
Ideally 300-400 conversions for 95% confidence

Typical Durations:

High traffic e-commerce: 1-2 weeks
Medium traffic retail: 2-4 weeks
Low traffic or in-store: 4-8 weeks

Don't Stop Early! Even if one variant looks like it's winning, run the full planned duration.

"What if external factors affect my test?"

Common Contaminants:

Holidays or events
Weather changes (retail)
Viral social media posts
Competitor actions
Supply chain issues

Actions:

Minor impact: Note it in results, proceed with caution
Major impact: Mark test "Inconclusive", re-run later
Known event: Plan around it (avoid testing during Black Friday)

Best Practice:

Control for external factors when designing test
Test during "normal" business periods
Document anomalies

"Can I change the test while it's running?"

No! Changes invalidate results.

Never Change:

Variant configurations
Traffic split percentages
Test duration mid-flight
Success metrics

Can Adjust:

Status to "Completed" (early termination if needed)
Add notes/observations (don't change variants)

If You Must Change:

Stop current test (mark Inconclusive)
Start new test with changes

"What makes a result 'statistically significant'?"

Statistical Significance means you're confident the difference isn't due to random chance.

Confidence Levels:

95%: Standard for business decisions
99%: Extra cautious (medical, safety)
90%: Acceptable for low-risk decisions

What Affects Significance:

Sample size: Larger = more significant
Effect size: Bigger difference = faster significance
Variance: Consistent results = faster significance

Tools: Use A/B test calculators to determine significance

Example:

Variant A: 1,000 visitors, 40 conversions (4.0%)
Variant B: 1,000 visitors, 60 conversions (6.0%)

Confidence: 96.8% that B is better
Action: Implement Variant B

"What's the difference between A/B testing pricing vs. promotions?"

Pricing Test:

Changes base price of product
Example: $99 vs. $120
Goal: Find optimal price point
Metrics: Revenue, units sold, profit margin

Promotion Test:

Changes discount offer (price stays same)
Example: "20% off" vs. "BOGO"
Goal: Find best promotional strategy
Metrics: Conversion rate, AOV, promo code usage

Can Combine: Test price points AND promotions in a matrix:

Variant A: $120 base, no promo
Variant B: $120 base, 10% off
Variant C: $99 base, no promo
Variant D: $99 base, BOGO

(4-way test, requires high traffic)

Naming Best Practices Summary

Test Names

Start with test type
Include what's being tested
Briefly describe variants
Be searchable and descriptive
Don't use generic numbers
Don't be vague

Hypotheses

State specific change
Predict expected outcome
Include magnitude if possible
Provide reasoning
Don't be vague or unmeasurable

Variant Names

Short and clear
Differentiate from other variants
Include key detail (price, offer, location)
Don't be ambiguous

Quick Checklist

Before running an A/B test, verify:

A/B Test Examples

Pricing Test

Test Name: "Premium Sneaker Pricing - $99 vs $120 vs $149"
Hypothesis: "If we price at $120 instead of $149, sales volume will
increase by 40% with only 20% revenue decrease, resulting in net
revenue gain because the $120 price point aligns better with target
customer willingness to pay."

Test Type: Pricing
Status: Running

Variants:
  1. Name: "Variant A - $99"
     Description: "Premium sneakers at $99. Lower price point targeting
     price-sensitive customers. No promotional messaging."
     Traffic %: 50%

  2. Name: "Variant B - $120"
     Description: "Premium sneakers at $120. Mid-tier pricing, current
     baseline price."
     Traffic %: 30%

  3. Name: "Variant C - $149"
     Description: "Premium sneakers at $149. Premium positioning with
     'luxury materials' messaging."
     Traffic %: 20%

Test Period: June 1 - June 21, 2025 (3 weeks)

Expected Results:
  - Variant A: High volume, lower revenue
  - Variant B: Balanced volume and revenue (hypothesis)
  - Variant C: Low volume, higher per-unit profit

Promotion Test

Test Name: "Summer Promo - 20% Off vs BOGO vs Free Shipping"
Hypothesis: "BOGO will generate highest AOV and total revenue because
customers will add more items to cart to qualify, even though unit
margins decrease."

Test Type: Promotion
Status: Planning

Variants:
  1. Name: "Control - 20% Off"
     Description: "SUMMER20 promo code, 20% off entire order, no minimum."
     Traffic %: 40%
     Campaign: "Summer Sale 2025 - 20% Off"

  2. Name: "Variant A - BOGO"
     Description: "BOGOSUMMER code, Buy 2 Get 1 Free on all apparel.
     Lowest-priced item free."
     Traffic %: 40%
     Campaign: "Summer Sale 2025 - BOGO"

  3. Name: "Variant B - Free Shipping $50+"
     Description: "SUMMERSHIP code, free shipping on orders $50+.
     No product discount."
     Traffic %: 20%
     Campaign: "Summer Sale 2025 - Free Shipping"

Test Period: July 1 - July 14, 2025 (2 weeks)

Product Placement Test

Test Name: "New Arrivals Display - Feature Wall vs Endcaps"
Hypothesis: "Feature wall placement will increase discovery rate by 40%
and units sold by 25% compared to endcaps due to 3x higher foot traffic
and better visibility."

Test Type: Product Placement
Status: Completed

Variants:
  1. Name: "Control - Endcaps"
     Description: "New arrivals on 2 checkout endcaps (Zones 15 & 16).
     Standard signage. Side placement, customers pass while in checkout line."
     Traffic %: 50%
     Stores: All stores (randomized which get endcaps)

  2. Name: "Variant A - Feature Wall"
     Description: "New arrivals on main floor feature wall (Zone 12).
     Large 'JUST ARRIVED' signage. Center store, high visibility."
     Traffic %: 50%
     Stores: All stores (randomized which get feature wall)

Test Period: May 1 - May 28, 2025 (4 weeks)

Results:
  - Control (Endcaps): 4.2% discovery, 145 units sold
  - Variant A (Feature Wall): 6.8% discovery, 203 units sold
  - Confidence: 97.2%
  - Winner: Feature Wall
  - Action: Roll out feature wall placement company-wide

Tips for A/B Testing Success

Designing Strong Tests

One variable at a time: Don't test price AND promotion simultaneously
Meaningful differences: Variants should be distinct enough to matter
Clear hypothesis: Know what you expect and why
Measure what matters: Focus on business-relevant metrics

Running Tests Properly

Don't peek early: Let test run full duration
Equal experience: No variant should have technical advantages
Consistent environment: Avoid changes during test period
Random assignment: Traffic split should be random, not biased

Analyzing Results

Check significance: Don't declare winner without statistical confidence
Look beyond topline: Consider margin, customer segments, long-term effects
Document everything: Record findings, learnings, anomalies
Share insights: Communicate results to team

Building Testing Culture

Test continuously: Always have tests running
Start small: Simple tests before complex ones
Fail forward: Learn from inconclusive tests
Build library: Document all tests for institutional knowledge

Common Testing Mistakes to Avoid

Stopping test early because one variant looks good
Testing during holidays or anomalous periods
Changing test mid-flight when results don't look good
Insufficient sample size leading to false conclusions
Testing too many variants at once (dilutes traffic)
Ignoring statistical significance and going with gut feel
Not documenting results and learnings