SEO Split Testing Template

Pick a Hypothesis & Test Type

Turn a vague idea ("new titles will help") into a single, falsifiable hypothesis tied to one metric.

Start with one clear hypothesis

Write it as a single sentence you can prove or disprove. Avoid testing five things at once or you will not know what moved the needle.

Hypothesis: [If we change X on this page group, then organic metric Y will improve, because Z.]

Common SEO test types

• Title tag & meta description rewrites (best for click-through rate)
• Page template changes (layout, internal links, schema)
• On-page content additions (intros, FAQs, headings)
• Indexing / technical tweaks (canonicals, structured data)

Pick the metric that matches the change: titles affect click-through rate & clicks; content and links affect impressions & position.

Primary metric: [Clicks / CTR / Avg position]
Guardrail metric (must not drop): [e.g. conversions]

Choose Test vs Control (Page-Group / Bucket Testing)

Split similar URLs into a test bucket and a control bucket, since you cannot cookie-split search users.

Why not a normal A/B test?

In SEO you cannot show Googlebot two versions of one URL, so user-level cookie splitting does not work. Instead you test at the page-group level: change one bucket of pages, leave a matched bucket unchanged, and compare organic performance over time.

Build two comparable buckets

1. Pick a set of similar pages (same template and intent, e.g. product or category pages).
2. Randomly assign each page to test or control so traffic, age, and topic are balanced.
3. Confirm both buckets have enough pages and clicks to produce a signal, not noise.

Page template under test: [Category pages]
Pages in test bucket: [# URLs]
Pages in control bucket: [# URLs]

Keep the only difference between buckets the change you are testing. Never serve Googlebot different content than users.

Set Up Tracking & Timeline

Define how long the test runs and exactly which data you will pull before you change anything.

Lock the measurement plan first

Decide your data source and timeline before launch so you cannot move the goalposts later.

• Data source: Search Console (clicks, impressions, CTR, average position), filtered by the test and control URL sets.
• Pre-period: capture a clean baseline for both buckets, ideally a full business cycle.
• Test period: long enough for Google to recrawl, re-rank, and accumulate clicks, often several weeks.

Record the baseline

Baseline window: [dates]
Test bucket baseline clicks: [#]
Control bucket baseline clicks: [#]
Planned launch date: [date]
Planned end date: [date]

Tag the launch date so you can line up before-and-after data and spot the exact moment the change took effect after recrawling.

Run the Test (Avoid Confounders)

Keep the experiment clean so an outside factor cannot be mistaken for your result.

Ship the change to the test bucket only

Apply the change to every page in the test bucket at once, and freeze the control bucket. Then leave both alone.

Watch for confounders

• Don't make other changes to either bucket mid-test (new links, redesigns, pricing).
• Track algorithm updates and core updates; note any that land during the window.
• Watch crawling & indexing so the change is actually live and seen by Google.
• Avoid seasonal spikes hitting only one bucket (e.g. a promo on test pages).

Change went live (confirmed crawled): [date]
External events logged: [updates, news, campaigns]

Resist peeking and ending early on a good day. Stopping the moment results look favorable inflates false positives. Let the planned period finish so both buckets experience the same outside conditions.

Analyze Results (Significance, Seasonality)

Compare test vs control fairly and decide whether the lift is real or noise.

Compare the buckets, not just before-and-after

Raw before-and-after numbers are misleading because traffic shifts seasonally. The control bucket absorbs those shared swings, so compare the relative change in the test bucket against the control bucket over the same dates.

1. Calculate each bucket's change from baseline to test period.
2. Subtract the control's change to isolate the effect of your edit.
3. Check whether the gap is large and consistent enough to be a real signal, not week-to-week noise.

Sanity checks

• Seasonality: did both buckets move together except for your change?
• Algorithm updates: would an update explain the swing?
• Sample size: enough pages and clicks to trust the result?

Test bucket change: [%]
Control bucket change: [%]
Net effect: [+/- %]
Verdict: [Win / Loss / No effect]

Roll Out or Roll Back & Document

Act on the result and capture the learning so the next test starts smarter.

Decide, then act

• Clear win: roll the change out to the control bucket and other matching pages.
• Clear loss: roll back the test bucket to the original version.
• No effect: revert or keep based on other reasons (UX, accessibility), but do not claim an SEO lift.

Document every test

A test you cannot find again is a test you will repeat. Log it in a shared place so the team builds a library of evidence.

Test name: [title]
Hypothesis: [statement]
Result: [win / loss / flat]
Decision: [roll out / roll back]
What we learned: [insight]
Next test idea: [follow-up]

Treat results as evidence for this site, not universal law. Re-test important wins later, since Google changes and what worked once may fade.

Why can't I run a normal A/B test for SEO like I do for conversion rate?

CRO tools cookie-split human visitors and show each person a different version of the same URL. You cannot do that with Google, because a URL must serve one version to the crawler, and showing the bot something different from users is cloaking. Instead, SEO tests work at the page-group level: you split similar pages into a test bucket and a control bucket and compare their organic performance over time.

How many pages do I need to run a reliable SEO split test?

There is no fixed number, but you need enough pages and enough clicks per bucket for the result to be a signal rather than noise. A handful of low-traffic pages will swing wildly week to week and tell you nothing. Tests on large, similar page sets (think category or product templates with steady organic traffic) give cleaner reads than tests on a few one-off pages.

How long should an SEO test run?

Long enough for Google to recrawl and re-rank the test pages and for both buckets to accumulate meaningful clicks, which often means several weeks. Running ideally through a full business cycle helps average out weekday and weekend patterns. Avoid ending the test early just because results look good on one day, since stopping on a favorable spike inflates false positives.

How do I separate my change from seasonality or an algorithm update?

That is exactly what the control bucket is for. Seasonality and core updates tend to move similar pages together, so if both buckets rise and fall in step, those are shared effects. The part you care about is how much the test bucket changed beyond the control bucket over the same dates. Always log any algorithm updates during the window so you can rule them out.

What do I measure, and where does the data come from?

Google Search Console is the primary source: clicks, impressions, click-through rate, and average position, filtered to the exact URLs in each bucket. Match the metric to the change. Title and meta description tests mainly move click-through rate and clicks, while content, internal links, and template changes show up more in impressions and average position.

What should I do when a test wins, loses, or shows no effect?

Roll a clear win out to the control bucket and other matching pages. Roll a clear loser back to the original. If there is no measurable effect, revert or keep the change based on other reasons like usability, but do not claim an SEO lift you cannot prove. In every case, document the hypothesis, result, decision, and learning so the team builds a library of evidence instead of repeating guesses.