Experimentation programs I have built eventually faced the same transition: the people who started the program were not always the people running its next phase. People join. People leave. Priorities shift. Stakeholders change. A written process gives the team something durable to inspect and improve through those changes.
"You need standard processes so that no matter who comes in, they understand how we run tests. What confidence means. What power level we use. What's an acceptable MDE, what's not. When do we run tests, when do we turn them off. People come and go, but the process stays."
— Atticus Li
Standards are not bureaucracy by default. They let a growing team preserve an agreed evidence bar without every new hire reinventing the workflow from scratch.
The Scaling Wall
"As you scale up tests, you're going to hit a ceiling. It might be a tool issue, a people issue, a process issue, or a culture issue. You're going to find out it's one of those. But one of the biggest mistakes is teams running a lot of tests without focus — just running tests for the sake of running tests."
— Atticus Li
When I helped scale an experimentation program, the first constraint we encountered was informal knowledge rather than tooling or traffic. Things that "everyone knew" in a smaller group stopped being shared reliably as more people joined the workflow. This is a first-person operating observation, not a universal scaling sequence.
- A new analyst did not know what confidence threshold to apply because nobody had written it down.
- A new developer did not know which QA steps were mandatory and which were optional.
- A new CRO manager did not know how to handle a sample ratio mismatch because the convention had been verbal.
Each of these is a small gap. Together they can create inconsistent test decisions as volume rises. Track QA failures, design exceptions, missing diagnostics, and decision reversals to see whether that is happening in your own program.
The fix is not to slow down. It is to write the standards down, aggressively, before you scale — and to treat the standards as living documents that every new team member reads on day one.
What Needs to Be Standardized
Here are the decisions every experimentation program needs to make explicitly, in writing, before it can scale:
1. Confidence threshold
What significance level (or Bayesian probability) is required to call a test a winner? Is it different for low-risk vs. high-risk tests? Document the threshold, the reason, and the exceptions.
2. Statistical power
What power are you targeting in pre-test calculations, and why is that miss rate acceptable for this decision? Choose it with the rest of the error policy and document both the threshold and rationale. Do not treat an uncited industry default as a substitute for that decision.
3. Minimum detectable effect (MDE) policy
What is the smallest effect that would change the business decision? Use the page's baseline, eligible traffic, allocation, variance, outcome delay, and chosen error rates to calculate whether that effect is detectable in the relevant window. Do not infer feasibility from a generic traffic or lift cutoff.
4. Test duration policy
When do you stop a test? At significance, at time elapsed, at sample size, at a combination? Pre-determined stopping rules are the difference between rigorous testing and p-hacking.
5. Sample ratio mismatch policy
What procedure and pre-specified threshold flag a possible sample ratio mismatch? How is the cause investigated, which data may be affected, who gets notified, and who decides whether the existing readout is usable? An SRM flag should trigger diagnosis rather than automatic invalidation by an uncited universal cutoff.
6. QA checklist
Every test goes through a QA checklist before it launches. What is on the checklist? Who signs it off? What is the blocker policy if something fails?
7. Reporting format
Every test result is reported in a standard format. Same sections, same dollar-value framing, same disclosure of limitations. Consistency is how stakeholders learn to read results quickly.
8. Intake and prioritization criteria
How does an idea get from a backlog into the next test slot? What scoring framework do you use (ICE, PIE, PRISM)? Who has authority to override prioritization?
9. Holdout policy
Do you run post-launch holdouts on winners? For how long? For which kinds of tests? This is critical for validating that lifts persist beyond the test window.
10. Learning documentation
Every test, win or lose, produces a learning. Where is that learning stored? How is it searchable? How do new team members find prior learnings relevant to their current test?
Standards Are Not Dogma
One reason teams resist standards is a fear that they will calcify the program. "If we write everything down, we cannot adapt." This is the wrong concern. Good standards are versioned. They are revised. They are the starting point, not the ceiling.
"If someone new comes in with a different way of running tests, they can look at our process and give recommendations — but they'll also understand why we do the things we do. If their method is better, we take it, implement it, and learn from it."
— Atticus Li
The point of the standard is to make disagreement productive. Without a documented baseline, every new hire's preferences become personal battles. With a documented baseline, preferences become proposals to revise a process. That is healthy. It is also much faster.
Writing Standards That People Actually Read
Most experimentation standards documents fail because they are written like regulatory filings. They are long, dense, structured like legal contracts, and nobody reads them after the first day. Here is how to write standards that stick:
Make the decision path easy to scan. Put the rule, rationale, owner, exceptions, and example where a practitioner can find them quickly. Measure whether people can apply the standard correctly; page count is not a universal usability benchmark.
Use examples, not just rules. For every standard, show a concrete example of how to apply it: "Here is the error policy we chose for a reversible decision. Here is the stricter policy we chose for a costly rollout. Here are the consequences we considered."
Note the rationale. Why is the standard what it is? A team member who understands the reason can reason their way through edge cases. A team member who only knows the rule cannot.
Name an owner. Every standard has a single human who is responsible for keeping it current. If nobody owns it, it decays.
Version it. Every change to a standard is a versioned change with a date and a reason. That way you can look back and see how the program's thinking has evolved.
The Onboarding Multiplier
The value of good standards is a shorter, more observable path to independent work. Measure ramp time in your own team rather than promising a universal weeks-versus-months result.
When the process is documented, a new CRO manager can read the error policy, QA checklist, and reporting format, then demonstrate the workflow on a representative test. Measure time to an accurate, independently reviewed decision in your own team rather than promising that documentation creates competence within a day.
Onboarding effort matters more as additional people enter the workflow. Track review corrections, exception handling, and time to independent work before and after a standards change. Those local measures can show whether the documentation helped; a universal team-size or annual-cadence multiplier cannot.
The Dependencies Between Standards
One thing I have learned the hard way: standards depend on each other, and you cannot skip the dependencies.
You cannot have a real confidence threshold policy without a real MDE policy, because the MDE determines how long you have to run to hit confidence. You cannot have a real intake process without a real scoring framework, because scoring is how the intake gets triaged. You cannot have a real post-test reporting format without agreement on how realized impact is calculated.
Write your standards as a connected document, not a series of independent pages. Show the relationships explicitly. A new hire should be able to trace the flow from "idea arrives" to "result reported" and see how each standard feeds the next.
FAQ
Should standards be the same for high-traffic and low-traffic pages?
Keep the decision principles and documentation requirements consistent, with explicit design-specific choices. Duration, MDE, allocation, and feasible test type will differ with traffic, baseline, variance, outcome delay, and business consequence. Document those inputs and the reasoning.
How do you update standards without disrupting the team?
Set a review cadence that matches how quickly the program and tooling change, and add event-driven reviews after material failures or new methods. Anyone on the team can propose a revision. Proposals get discussed, versioned, and adopted or rejected through a named decision process.
What if leadership resists standardization?
Show them the cost of inconsistency. Pull examples of tests where different team members made different decisions on the same edge case. Tie those inconsistencies to lost learnings or bad outcomes. Leadership will see that standardization is how you scale without losing quality.
How do you handle cases where the standard doesn't apply?
Every standard should have an "escalation path" section. When in doubt, escalate to the owner of the standard. That way the gap gets noticed and the standard gets updated. Silent workarounds are the start of process decay.
Build the Standards That Let You Scale
Informal knowledge becomes harder to rely on as more people and handoffs enter a program. Write the standards before the resulting inconsistencies become expensive, then verify through audits that the standards are actually being used.
I built GrowthLayer specifically to encode these standards into a working tool — confidence thresholds, QA checklists, prioritization scoring, reporting templates, and a versioned learnings library. It is the system I wish I had when I was scaling enterprise programs from scratch.
If you are hiring or growing into roles that require structured program management, explore open CRO and experimentation roles on Jobsolv.
Or book a consultation and I will help you audit and build the standards that let your program scale without losing rigor.