How to Write A/B Test Hypotheses That Pass the Falsifiability Test

A hypothesis is not a prediction. It is the experiment's memory. When a stakeholder submits "IF we make X clearer, THEN conversion will increase, BECAUSE users will understand it better," they have written a goal, not a test — and six months later, when someone audits the test library to find what actually worked, that hypothesis will be useless. This guide walks through the falsifiability standard, what bad hypotheses look like, how to rewrite them, and the language that helps you ask for clarification without making the requester feel attacked.

What You'll Learn

• Why vague hypotheses break execution today and meta-analysis tomorrow
• The IF/THEN/BECAUSE structure, and what each clause is actually doing
• Ten common bad-hypothesis patterns and their better versions
• The vague-word vocabulary that almost always signals weak thinking
• A copy-paste hypothesis template and review checklist
• Suggested language for asking stakeholders to clarify without sounding academic or corrective

Quick Stats Reference

The falsifiability test for any A/B hypothesis:

>

1. Does the IF clause name a specific element, location, or flow change?

2. Could a designer build the variant from this sentence alone?

3. Could an analyst identify the primary behavior the change should move?

4. Could a future reader understand what was tested without seeing the design file?

5. Could the result come back in a way that would clearly disconfirm the belief?

The single rule: _The IF statement must name the actual thing changing, not just the desired improvement._

Why Vague Hypotheses Are Expensive

Most experimentation programs accumulate hypotheses in an intake queue. Someone sees a number they do not like, writes a one-paragraph proposal, and ships it into the backlog. The intake form has an "IF/THEN/BECAUSE" field. Most stakeholders fill it out the way they would fill out a status update — with the _theme_ of the change, not the change itself.

This feels harmless. The proposal moves forward. Design interprets the brief, analytics picks a primary metric, and the test ships. But three weeks later, when the result comes in, a different kind of cost surfaces:

• The variant won by an unclear margin, and nobody is sure _which_ part of the change drove the result.
• The variant lost, and nobody is sure whether the theory was wrong or whether the specific execution was wrong.
• Six months later, a new analyst pulls the test library and tries to find "all tests where we improved checkout clarity." They get a list of fifteen tests, all tagged "clarity," with no record of what was actually changed in each one.

This is the meta-analysis tax. A program's test history should compound into institutional knowledge. When hypotheses are written as themes instead of interventions, the history compounds into noise.

"A hypothesis is not just a prediction. It is the experiment's memory. If a future reader cannot tell what was changed and why, the test did not produce a learning — it produced a number."

The IF / THEN / BECAUSE Structure

The standard format has three clauses, and each one is doing different work.

IF [specific change to a specific element]. This is the intervention. It must be concrete enough that a designer can build the variant from this sentence alone. "Improve clarity" is a goal. "Replace the label 'Continue' with 'Review your order' on the cart-page CTA" is an intervention.

THEN [specific user behavior or metric will improve]. This is the prediction. It must be close enough to the intervention that the result is interpretable. A copy change on the cart-page CTA most directly affects clicks on that CTA — not "revenue per visitor" three steps downstream.

BECAUSE [specific reason tied to user psychology, friction, motivation, or decision-making]. This is the mechanism. It is the part most people skip or fill with platitudes. "Because users will have a better experience" is not a mechanism. "Because the revised label tells users what happens after they click, reducing uncertainty about whether they are placing the order immediately" is a mechanism.

When all three clauses are concrete, the hypothesis becomes _falsifiable_ — there is a result the test could produce that would weaken the belief. That falsifiability is what turns a test from a fishing trip into an experiment.

The Most Common Weak Pattern

Before walking through bad-to-better rewrites, it helps to name the pattern that produces 80% of weak hypotheses:

Weak pattern: _"IF we make [theme] better, THEN [downstream metric] will go up, BECAUSE [generic mechanism word]."_

Examples:

• "IF we make the page clearer, THEN conversion will increase, BECAUSE users will understand it better."
• "IF we improve the checkout flow, THEN orders will increase, BECAUSE the experience will be smoother."
• "IF we add trust messaging, THEN sign-ups will increase, BECAUSE users will feel more confident."

Each of these sounds reasonable in isolation. None of them is testable in the strict sense. The same sentence could justify ten different design treatments, three different primary metrics, and any post-hoc explanation the team wants to attach to whatever result comes back. That flexibility is exactly the problem.

The diagnostic question: _Could this same hypothesis be used to justify ten different design treatments?_ If yes, the IF clause is doing the work of a theme tag, not a hypothesis.

Two Origin Failures That Should Trigger Pushback, Not a Test

Vague wording is the most visible problem with stakeholder-submitted hypotheses. There is a deeper one underneath it: hypotheses that should not have entered the queue at all. Two patterns account for most of these. Both produce requests that may look fine after a wording rewrite, but the test itself was the wrong thing to build in the first place.

Pattern 1: "Make it better" without internal data

A stakeholder says "make the homepage feel more premium." Or "the checkout just doesn't convert as well as it should." Or "I think the layout is off." Pressed for what "premium" or "should" or "off" means, the stakeholder gives a vibe, not a definition. Pressed for data showing the current state is broken, the stakeholder gives a feeling, not a number.

This is the taste-and-opinion pattern. The hypothesis is real to the requester — they genuinely sense that something is wrong — but the underlying signal is aesthetic preference, not evidence. The "make it better" is sincere. What "better" looks like is undefined. Whether the current state is actually underperforming is uncharted.

The test that comes out of this pattern almost always fails in a confusing way. The variant is just-different-enough to be measured, but there is no theory of why the variant should outperform the control. If the variant wins, the team is unsure whether the original was actually a problem or whether any change would have produced a similar lift. If it loses, the team is unsure whether the theory was wrong or whether the wrong variant was built. Either way, the test consumes traffic without producing a learning.

The pushback question: _"What signal — quantitative or qualitative — tells us the current state is a problem worth testing?"_ Not as a gotcha. As a precondition. A test consumes traffic and time. If there is no evidence that the current surface is underperforming, the team is testing a preference, not a problem.

The right response is rarely "no." It is usually "let's confirm there's a problem first." That can be a quick funnel-drop analysis, a session-replay review, or a small qualitative study. If the data turns up a real signal, the hypothesis gets a stronger origin and is worth building. If the data turns up nothing, the team learns something important — that the surface was working fine — without spending a test cell on it.

Pattern 2: Competitor imitation without internal alignment

A stakeholder sees a competitor doing something — visible chat support on every page, a sticky urgency timer at checkout, a particular onboarding sequence, a free-tool offer in the homepage hero — and submits a request to "test the same thing on our site." There is no internal data showing the absence of that element is hurting the team's funnel. There is no analysis of whether the competitor's strategy aligns with the team's own audience, pricing, or product. Often there is not enough traffic to detect the effect at any reasonable size, but the request did not include a power calculation because the assumption is that what works for the competitor will work for the team.

This is the competitor-imitation pattern, and it produces some of the most expensive bad tests in any program. The request feels strategic on the surface — competitive parity, table stakes, defensive moves — but it is built from outside-in rather than inside-out. The competitor's strategy reflects their audience, their funnel, their pricing, their brand position, and their stage. None of those automatically transfer.

A few things that are usually missing when a competitor-imitation request lands in the queue:

• An internal data signal that the absence of the competitor's element is actually hurting the team's funnel.
• An analysis of whether the team has enough traffic on the proposed surface to detect a meaningful effect.
• A theory of why the change should work in the team's specific context, not just in the competitor's.
• An honest assessment of whether the competitor is winning _because of_ the element being copied, or _in spite of_ it.
• Recognition that what looks like a polished, intentional choice on a competitor's site may be a stale A/B test loser they have not gotten around to rolling back.

The pushback question: _"What internal signal tells us we have the same problem the competitor's pattern is solving, and do we have the traffic to detect the effect we'd care about?"_

The right answer is sometimes "let's investigate the underlying user behavior on our own surface before deciding what to test." That is not a delay tactic. It is the difference between running an experiment and running a copy-paste.

Why both patterns share a root

The taste pattern and the competitor pattern share an origin: the hypothesis came from outside the team's own data. In the taste case, it came from a single person's preference. In the competitor case, it came from another company's strategy. Neither is paired with internal evidence that the team's funnel actually has the problem the proposed intervention would address.

When the origin is internal — a funnel drop-off that exceeds expectations, qualitative data revealing a specific user misunderstanding, a segment-level pattern suggesting a specific intervention — the hypothesis tends to be both falsifiable and worth running. When the origin is external — opinion or imitation — the hypothesis usually needs to be rerouted into investigation before it becomes a test.

The origin diagnostic: _Where did this hypothesis come from — a number in our funnel, a story from our users, a sense in someone's head, or a screenshot from a competitor?_ If the answer is one of the last two, the next step is investigation, not build.

This is not about gatekeeping. It is about making sure the team is testing problems that exist, not preferences that wish they did. A program that maintains that discipline produces a test history full of real learnings. A program that doesn't produces a test history full of "inconclusive" tags and confused stakeholders six months later.

Ten Bad-to-Better Hypothesis Rewrites

Assuming the hypothesis passes the origin check — there is real internal evidence that the surface has a real problem — the remaining task is to translate vague wording into a falsifiable intervention. The examples below are intentionally generic — checkout flows, sign-up forms, homepage headlines — so the lesson is portable. The translation pattern is what matters.

1. The "improve" pattern

Bad: "IF we improve the homepage, THEN sign-ups will increase, BECAUSE users will have a better experience."

Better: "IF we replace the generic homepage headline with one that states the specific job-to-be-done the product solves, THEN more visitors will click the 'Start free trial' CTA, BECAUSE the new headline reduces the time required to evaluate relevance."

Why better: "Improve" is a category, not a change. The rewrite names the element (headline), the kind of change (job-to-be-done language), the primary behavior (CTA click), and a specific mechanism (faster relevance evaluation).

2. The "clarity" pattern

Bad: "IF we make pricing clearer, THEN purchases will increase, BECAUSE users will understand the offer."

Better: "IF we display the monthly price, annual price, and renewal date directly under each plan card, THEN more visitors will select a plan, BECAUSE the full cost is visible without an extra interaction."

Why better: "Clearer pricing" could mean a dozen things. The rewrite specifies which numbers appear, where they appear, and which interaction is being removed.

3. The "friction" pattern

Bad: "IF we reduce friction in the form, THEN more users will submit it, BECAUSE it will be easier."

Better: "IF we reduce the sign-up form from eight required fields to four required fields by moving the optional ones to a post-signup profile step, THEN more visitors who start the form will submit it, BECAUSE the first interaction requires less effort before users receive value."

Why better: "Friction" is a vague metaphor. The rewrite specifies the field count change and the reason that change should affect behavior.

4. The "stronger CTA" pattern

Bad: "IF we make the CTA stronger, THEN more users will click it, BECAUSE it will be more obvious."

Better: "IF we change the CTA copy from 'Continue' to 'See available plans' on the homepage hero, THEN more visitors will click into the product comparison page, BECAUSE the new copy describes the next step explicitly."

Why better: "Stronger" can mean color, size, placement, animation, or wording. The rewrite names which property is changing.

5. The "trust messaging" pattern

Bad: "IF we add trust messaging, THEN conversion will increase, BECAUSE users will feel more confident."

Better: "IF we add a short line stating 'Free returns within 30 days' directly beside the purchase CTA, THEN more visitors will complete checkout, BECAUSE the line addresses the specific concern of being stuck with a product that does not work for them."

Why better: "Trust messaging" is a category. The rewrite specifies which message, where it appears, and which concern it targets.

6. The "personalization" pattern

Bad: "IF we personalize the experience, THEN engagement will increase, BECAUSE users will see more relevant content."

Better: "IF returning visitors who previously viewed running shoes see a homepage hero featuring running shoes instead of the default category mix, THEN more of those visitors will click into the product listing, BECAUSE the hero matches their most recent browsing intent."

Why better: "Personalize" can mean anything. The rewrite names the audience rule, the content swap, and the targeted behavior.

7. The "simplify" pattern

Bad: "IF we simplify the checkout flow, THEN orders will increase, BECAUSE the process will be easier."

Better: "IF we remove the optional account-creation step from the main checkout path and move it to a post-purchase confirmation screen, THEN more visitors will complete a purchase, BECAUSE buyers can finish the transaction before being asked to take a secondary action."

Why better: "Simplify" is a theme. The rewrite specifies which step is removed and where it is relocated.

8. The "mobile experience" pattern

Bad: "IF we make the page more mobile-friendly, THEN mobile conversion will increase, BECAUSE the experience will be easier on mobile."

Better: "IF we make the primary CTA on the plan-detail page sticky on mobile so it remains visible while users scroll past the plan description, THEN more mobile visitors will start the enrollment flow, BECAUSE the primary action stays accessible during evaluation."

Why better: "Mobile-friendly" is a category. The rewrite specifies the exact treatment (sticky CTA) and the moment it intervenes (during scroll).

9. The "default" pattern

Bad: "IF we set a smart default, THEN more users will continue, BECAUSE they will not have to think as hard."

Better: "IF we pre-select the most common start-date option on the date-picker step, THEN more visitors will advance to the next step, BECAUSE pre-selection reduces a small decision-making cost at a step where most users were going to choose that option anyway."

Why better: "Smart default" could mean defaults on any number of fields. The rewrite names the field, the default chosen, and the underlying decision-cost theory.

10. The "in-context help" pattern

Bad: "IF we add FAQs near related content, THEN conversion will increase, BECAUSE users will have their questions answered."

Better: "IF we place a three-question FAQ block directly under the pricing comparison table addressing the most common pre-purchase concerns from support tickets, THEN more visitors will click into the plan-selection step, BECAUSE answers appear at the decision moment instead of requiring a separate navigation to a help center."

Why better: "Add FAQs near related content" is directional but vague. The rewrite names the location, the question set, and the behavioral cost being removed.

The Vague-Word Vocabulary

The same set of words shows up in almost every weak hypothesis. They are not banned. But when one of these words appears, it is a flag that the IF clause may be hiding behind a theme.

The translation rule: when one of these words appears in a hypothesis, ask "what specifically is changing on the page?" If you cannot answer that without re-reading the proposal, the hypothesis is not yet falsifiable.

The Four Components of a Strong Hypothesis

Every strong hypothesis carries four components, even if they are not always labeled.

The problem signal. What evidence justifies the test? This can be a research finding ("users in a moderated study did not understand the timing of payment"), an analytics signal ("drop-off at the date-picker step is significantly higher than at adjacent steps"), or qualitative feedback ("support tickets cluster around uncertainty about the renewal date"). Without a signal, the hypothesis is a hunch dressed in formal language.

The intervention. What exactly are you changing? Element, location, copy, flow, audience, default, hierarchy. The intervention is the part that gets written into a design spec. If the intervention is missing, Design will fill the gap by guessing, and the test ships as a different experiment than the one you thought you were running.

The behavioral mechanism. Why should the intervention affect behavior? This is the place where behavioral science earns its keep. The mechanism could be a reduction in cognitive cost, a removal of a specific decision barrier, an alignment with a known mental model (BNPL vs delayed payment, for instance), a use of social proof, or a clearer signal of progress. The mechanism is what makes the hypothesis a _theory_ rather than a guess.

The measurement. Which behavior will move first if the theory is right? Pick the metric closest to the intervention, not the metric closest to revenue. A copy change on a cart-page CTA most directly affects clicks on that CTA. Revenue-per-visitor will move only if every step downstream also responds — so a flat revenue number does not falsify a copy-clarity theory by itself.

The Strong Hypothesis Template

Standardize once. Apply forever.

PROBLEM SIGNAL:
[The data, research, analytics, or qualitative evidence that
justifies investigating this surface.]
CURRENT EXPERIENCE:
[What the user sees and does on the surface today.]
PROPOSED CHANGE:
[The specific element, location, copy, flow, or audience
being modified, and what the new version is.]
HYPOTHESIS:
IF we [specific change to a specific element],
THEN [specific user behavior or metric will improve],
BECAUSE [specific reason tied to user psychology,
friction, motivation, or decision-making].
PRIMARY METRIC:
[The behavior closest to the intervention.]
SECONDARY METRICS:
[Diagnostic behaviors that explain why the primary
metric moved — or why it did not.]
GUARDRAIL METRICS:
[Behaviors that should not get worse, even if the
primary metric moves favorably.]
LEARNING GOAL:
[What the team should know after the test, win or lose.
This is the meta-analysis line. If this is concrete,
the test will compound into institutional knowledge.]

The template takes about an afternoon to standardize across an intake form. The cost of adopting it is small. The cost of not adopting it accrues quietly as the test library grows.

A Worked Example

Here is the template applied to a generic checkout-clarity test.

Problem signal. In a recent moderated study, a majority of participants misinterpreted the "Pay after checkout" option as a buy-now-pay-later financing arrangement, similar to Affirm, Klarna, or AfterPay. The intended meaning — that the payment is still required, but is collected at a later step in the same session — was missed by most participants.

Current experience. The checkout page presents two payment-timing options with short labels and no surrounding helper text.

Proposed change. Update the label of the second option and add a one-line helper sentence directly beneath it, stating when payment is required and at which step it will be collected.

Hypothesis. IF we change the second payment-timing option's label to a phrasing that explicitly references the in-session collection step, and add a one-line helper sentence describing when payment is collected, THEN more visitors selecting that option will complete payment within the same session, BECAUSE the revised copy reduces confusion with external BNPL/installment financing mental models and surfaces the timing requirement at the moment of the choice.

Primary metric. Payment completion rate among visitors who selected the second option.

Secondary metrics. Selection rate of the second option, abandonment between the option selection and the payment step, and downstream order completion rate.

Guardrails. Overall checkout completion rate, support contacts related to payment confusion, refund and cancellation rate.

Learning goal. Establish whether mental-model alignment via copy can reduce the BNPL conflation effect, and whether the effect is large enough to justify a global rollout vs a label-only rollout.

Notice what this example does and does not include. It does not include exact participant counts, brand names, internal page names, or proprietary metric definitions. It describes a generic checkout-clarity pattern using terms a reader anywhere can map onto their own product. The mechanism — mental-model conflation with BNPL — is the part that makes it a theory rather than a guess.

Common Mistakes Beyond Vague Wording

Vague wording is the most common failure mode, but four others show up regularly and deserve explicit naming.

Mistake 1: Writing the goal as the intervention

The most frequent mistake. "Make X clearer" is a goal. The intervention is the specific element-level change you are going to ship. The goal belongs in the problem signal or the learning goal. The hypothesis IF clause should always be the change.

Mistake 2: Choosing a metric too far downstream

A copy change on a single checkout step may eventually influence final purchase completion, but the more direct behavior is continuation from that specific step. If the primary metric is "final revenue per visitor," the test may come back flat for reasons that have nothing to do with the variant — and the variant gets killed for the wrong reason. Pick the metric closest to the intervention, then add downstream metrics as diagnostics.

Mistake 3: Bundling unrelated changes

A test that "redesigns the page, simplifies the flow, updates the CTAs, and rewrites the value props" may be a valid business test. It is not a clean learning test. If it wins, you know the package worked. You do not know which component mattered. There is a place for bundled tests — they answer business questions, not behavioral ones — but they should be labeled as such and not treated as evidence about any individual change.

Mistake 4: Writing a hypothesis that cannot lose

A strong hypothesis names a result that would weaken the belief. "If we improve clarity, conversion will increase" cannot really lose — any result can be explained as "the change wasn't clear enough" or "the metric wasn't the right one." A falsifiable version specifies the change, the metric, and the mechanism such that a flat or negative result actually challenges the theory.

Mistake 5: Confusing research insight with test hypothesis

A research insight identifies the problem ("users misunderstand the timing of payment"). A test hypothesis defines the intervention ("changing the label and adding helper text will reduce misunderstanding and improve payment completion within the session"). The insight is the input. The hypothesis is the proposed solution and its expected effect. Stakeholders often submit insights as if they were hypotheses, which leaves Design and Analytics to guess at the intervention.

The Hypothesis Review Checklist

Before a hypothesis enters the build queue, run it against these questions:

1. Can a designer build the variant from the IF clause alone, without re-interviewing the requester?
2. Can analytics identify the single behavior the change is expected to move?
3. Could a future analyst, reading only the hypothesis six months from now, understand what was tested?
4. Is the BECAUSE clause more specific than "less friction," "more clarity," or "better experience"?
5. Are we testing one intervention or a bundle of unrelated changes?
6. Is the primary metric close enough to the intervention to be causally interpretable?
7. Could a result come back that would clearly weaken the belief?

If the answer to any of these is no, the hypothesis is not ready. Send it back with a specific clarifying question, not a rejection.

How to Ask for Clarification Without Sounding Academic

Most stakeholders did not intentionally write a vague hypothesis. They wrote what the intake form prompted them for, in the style of a status update. The fix is rarely a lecture about falsifiability. It is usually a single clarifying question.

Avoid:

• "This isn't falsifiable."
• "Your hypothesis is too vague."
• "You haven't told me what we're testing."

Prefer:

• "The problem statement here is strong. Could we make the IF clause a little more concrete — are we changing the label, the helper text, the confirmation screen, or some combination?"
• "The research signal makes sense. I want to make sure Design knows exactly which surface to change. Do you have a specific intervention in mind?"
• "This describes the goal clearly. Could we add one more layer specifying the UX or copy change so the team knows what to build?"
• "For the test library later, it would help if the hypothesis captured the specific intervention, not just the theme. Want me to draft a version we can iterate on?"

The framing in each case is collaborative: the requester gave us a valid signal, and we are clarifying the executable form together. That keeps the conversation about the experiment, not about the requester's writing style.

The principle: _Critique the test, not the tester. Almost every weak hypothesis represents a real insight that has not yet been fully translated. The job is to finish the translation together, not to mark the proposal incomplete._

Why This Matters for Meta-Analysis

A program's test history should compound into institutional knowledge. Six months from now, an analyst should be able to query the test library and answer questions like:

• Which copy changes have most consistently moved checkout completion?
• Have we tested the BNPL-confusion theory before, and what did we learn?
• What kinds of helper-text placements have worked, and on which surfaces?
• Which interventions have failed in this category, and why?

That kind of querying only works if each hypothesis names a specific intervention and a specific mechanism. If the test library is full of "improve clarity" and "reduce friction," the questions above cannot be answered. The institutional memory collapses into a tag cloud.

Most CRO programs underinvest in this. The team optimizes for intake speed and test velocity, both of which feel like leading indicators. The compounding asset — the searchable, mechanism-tagged library of prior results — is invisible until someone tries to use it. The teams that build that asset early tend to look qualitatively different from the teams that do not, three years in.

Quick Tips for New Analysts

• Never accept a hypothesis whose IF clause uses only theme words. Ask for the specific change.
• Always specify the primary metric closest to the intervention. Downstream metrics are diagnostics, not primaries.
• Treat the BECAUSE clause as a theory of behavior, not a platitude. It is the part that becomes useful in meta-analysis.
• Document the test in plain prose someone could read in six months. If the future-you cannot understand the past-you's hypothesis, the test produced a number, not a learning.
• Bundle tests should be labeled as bundles. Do not let a multi-change test enter the library tagged as a single learning.

Tips for CRO Managers

• Make the falsifiability checklist part of intake. A test does not enter the build queue without passing it. This is the single highest-leverage policy a new manager can adopt.
• Require an internal data signal for every intake. No test enters the queue without evidence the current surface is underperforming or has a specific identified user friction. "Make it better" and "competitor X does this" get rerouted to investigation, not build. This single rule eliminates a large fraction of the program's wasted test cells.
• Train stakeholders, not just analysts. Most weak hypotheses come from product, UX, and design stakeholders who never received explicit instruction on hypothesis structure or origin discipline. A 30-minute internal session pays for itself within a quarter.
• Audit the existing library. Pull the last 20 tests. How many have a falsifiable hypothesis? The ones that do not are candidates for retrospective rewriting based on what the team actually built and measured.
• Tag interventions, not themes, in the test library. "Label change," "field count reduction," "default pre-selection," and "in-context helper text" are interventions. "Clarity," "friction," and "trust" are themes. The library should index the former.
• Connect the hypothesis standard to the readout standard. Tests with strong hypotheses get strong readouts. Tests with weak hypotheses produce numbers that nobody can interpret three months later. The two standards reinforce each other.

FAQ

Isn't requiring a specific intervention going to slow down intake?

In the short term, slightly. In the long term, no — and the time saved on the back end is much larger than the time spent on the front end. A vague hypothesis costs at least one back-and-forth cycle during build, plus another during analysis, plus the permanent meta-analysis tax. A specific hypothesis costs five extra minutes during intake.

What if the stakeholder doesn't know exactly what change to test?

Then the proposal is at the research stage, not the test stage. That is fine. Treat it as an investigation request rather than an experiment request. Do the discovery work, then return with a falsifiable hypothesis. Trying to ship a test on top of an unresolved problem statement is what produces uninterpretable results.

How do I push back on a "competitor X does this, let's test it" request?

Treat it as an investigation request, not a test request. Acknowledge the strategic signal — that the competitor's pattern is worth understanding — and reframe the next step as research, not build. Useful language: "Before we commit a test cell to this, can we look at our own funnel data on that surface and confirm we have the same problem the competitor's pattern is solving? If we do, we can write a specific hypothesis around it. If we don't, we'll have saved the test cell and learned something about where our actual leverage points are." This framing rarely lands as resistance, because it shows you take the strategic signal seriously and are doing the work to make sure the resulting test produces a real answer. The honest reality is that some competitor patterns are stale A/B test losers that the competitor has not gotten around to rolling back — copying them blindly produces predictable failures.

How do I respond when a stakeholder wants to test something based on personal preference rather than data?

Route the underlying concern into investigation, not rejection. The stakeholder almost always has a real sense that something is off; the issue is that the sense has not been paired with evidence. Useful language: "I want to make sure we can defend whatever result comes back from this test, so let me pull the funnel data on that surface first. If the data confirms a problem, we'll write a hypothesis around the specific friction. If the data shows the surface is performing fine, we'll have learned that and can redirect to a higher-leverage surface." Most stakeholders accept this framing because what they actually want is the outcome the test is supposed to produce, not the test itself. The investigation step is small, fast, and almost always reveals more than the original preference suggested.

How do I handle hypotheses that come in as a directive from leadership?

Restate them in falsifiable form before building. "Leadership wants us to test the new homepage" is not a hypothesis. Translate it: which specific element on the new homepage are we changing first, what behavior should that move, and why? If leadership has only given a directional ask, your translation _is_ the test plan, and you should walk it back to them for confirmation before build.

Should every hypothesis include a behavioral-science citation?

No. The BECAUSE clause needs a specific mechanism, but the mechanism does not need to be a named bias from the literature. "Pre-selection reduces a small decision-making cost at a step where most users were going to make that choice anyway" is a fine mechanism without invoking "default bias" by name. The literature is helpful when it gives you shorthand, and unhelpful when it gives you the appearance of rigor without the substance.

What about hypotheses where the mechanism is genuinely unknown?

Then the hypothesis is exploratory, and you should label it as such. An exploratory test asks "does this variant move the metric?" without committing to a theory about why. That is a valid kind of test, but it produces less compounding knowledge than a theory-driven test, and it should be the minority of the program.

Can the same hypothesis be tested twice with different executions?

Yes — and this is often where compounding institutional knowledge comes from. Two label-change tests on different surfaces, both with the same BNPL-conflation mechanism, become much more informative when the meta-analysis can link them. That linkage only works if both hypotheses named the mechanism specifically.

How specific is too specific?

There is no real upper bound on intervention specificity. There is an upper bound on _scope_ specificity — a hypothesis should describe one intervention, not five. If your IF clause lists more than one change connected by "and," consider whether the test is actually a bundle that should be split or relabeled.

Standardize the Hypothesis Form This Quarter

For most experimentation programs, this is among the highest-leverage process changes available. The cost is small — an afternoon to update the intake form, a 30-minute stakeholder training, and a quarterly audit of the test library. The benefit compounds over every test the team runs from that point forward. The compounding is invisible in the first month and unmistakable in the second year.

I built GrowthLayer so the falsifiability checklist is enforced at intake by default, and every test in the library is indexed by intervention type and mechanism — not by theme. The result is a test history that compounds instead of accumulating.

For experimentation roles where this kind of rigor is the operating standard, explore open positions on Jobsolv.

Or book a consultation for help standardizing your team's intake form, retraining stakeholders on hypothesis structure, or running a retrospective audit of an existing library.