A/B test block
Why this matters for your business
Most storefront design decisions are guesses. "Should this button be red or green?" "Should pricing show 3 plans or 4?" "Does the new hero image convert better than the old one?" Without testing, the loudest opinion wins — usually leaving significant conversion on the table.
The A/B test block is drop-in storefront experimentation. Wrap any content in two variants; the system splits visitors 50/50 (stably — same visitor sees same variant), tracks conversion per variant, reports statistical significance. Ship the winner.
The unlock isn't any single test — it's the cadence of testing. Brands that ship one test per week build a compounding edge that's invisible to competitors. The A/B test block makes that cadence cheap.
What this typically unlocks
| Outcome | Result |
|---|---|
| Conversion lift per validated test | 3-15% typical |
| Tests/quarter merchant can ship | 8-12 vs. 0-1 manually |
| Validated insights/year | 30-40 with disciplined cadence |
| Compounding annual conversion lift | +15-30% from disciplined testing |
What you actually get
| Capability | Description |
|---|---|
| Wrap any content | Hero, banner, pricing, CTA — anything in your theme |
| Stable assignment | Same visitor sees same variant (no flicker) |
| 50/50 default split | Customizable (10/90, 25/75, etc.) |
| Outcome tracking | Conversion, AOV, cart-add — pick the metric |
| Statistical significance | Live p-value + confidence interval |
| Sample-size guardrail | Won't declare winner until power is sufficient |
| Auto-ship winner | Optional: when significance reached, send 100% to winner |
How to use it
In Shopify Theme Customizer:
- Find the section you want to test (e.g. hero block)
- Add A/B test block wrapper
- Set up Variant A and Variant B (drag-drop content into each)
- Set the goal metric (cart conversion, signup, etc.)
- Save → test goes live
The system handles assignment, tracking, and reporting.
Real merchant scenarios
Scenario A — Pricing-page test
Setup. Brand wants to test 3-plan vs. 4-plan pricing display.
Result over 4 weeks (40K pricing-page visits):
| Variant | Conversions | Conversion rate | Confidence |
|---|---|---|---|
| 3-plan | 1,520 | 7.6% | — |
| 4-plan | 1,280 | 6.4% | p < 0.001 |
Decision. Shipped 3-plan. Annualized impact: ~$84K extra conversions.
Scenario B — Hero image test
Setup. Founder thought new lifestyle hero would beat product hero. Tested.
Result over 3 weeks (12K homepage visits):
| Variant | Add-to-cart rate | Confidence |
|---|---|---|
| Product hero (control) | 3.2% | — |
| Lifestyle hero | 4.1% | p = 0.02 |
Decision. Shipped lifestyle hero. Founder's instinct validated, but not before checking — the same brand had rejected an earlier hero test that also "felt right" but underperformed.
Scenario C — CTA copy test
Setup. Test CTA copy: "Shop now" vs. "See the collection" vs. "Find your fit."
Multivariate result over 2 weeks (8K visits per variant):
| Variant | CTR |
|---|---|
| "Shop now" (control) | 6.2% |
| "See the collection" | 7.1% |
| "Find your fit" | 8.4% |
Decision. Shipped "Find your fit." Lift compounded across journey through to conversion.
Scenario D — Brand catches a "feels right" but underperforms
Setup. Marketing manager wanted to add a urgency banner to product pages. Tested.
Result over 4 weeks:
- Banner variant: conversion 1.7%
- No-banner variant: conversion 1.9%
- p < 0.05
The banner hurt conversion (felt salesy on premium-brand). Without testing, would have shipped and lost ~$32K/year.
Best practices
✅ Test one variable at a time. Hero image OR CTA copy, not both — otherwise you don't know which mattered.
✅ Wait for power before declaring. Sample-size calculator shows when you have enough data; trust it.
✅ Test high-traffic pages. Need ≥1K visits/variant for meaningful results.
✅ Document hypothesis before testing. Writing the prediction makes the result mean something.
❌ Don't peek and act on early results. A test "winning by 30%" after 50 visitors flips with the next 50.
❌ Don't run too many tests at once on the same page. Variant interactions get messy.
❌ Don't keep tests running past significance. Once declared, ship the winner. Lingering tests waste opportunity.
Plan tiers
| Capability | Free | Starter | Pro | Agency | Enterprise |
|---|---|---|---|---|---|
| A/B test block | — | — | ✓ | ✓ | ✓ |
| Multivariate (3+ variants) | — | — | ✓ | ✓ | ✓ |
| Sample-size guardrails | — | — | ✓ | ✓ | ✓ |
| Auto-ship winner | — | — | ✓ | ✓ | ✓ |
| Cross-page experiments | — | — | — | ✓ | ✓ |
| Custom goal metrics | — | — | ✓ | ✓ | ✓ |
See also
- Storefront widgets overview
- Sales engine experiments + holdouts — formal experiment infrastructure