Guardrail Metrics
Protective metrics monitored during experiments to ensure that improving the target metric doesn't come at the cost of degrading user experience, revenue, or other critical outcomes.
Guardrail metrics are the safety net of experimentation. While your experiment aims to improve a target metric (conversion rate, sign-ups, revenue), guardrail metrics ensure you're not breaking something else in the process.
Why Guardrails Are Essential
A classic example: a more aggressive checkout flow might increase conversion rate by 10% — but also increase return rates by 20% and support tickets by 30%. Without guardrails on returns and support volume, you'd ship a "winner" that actually destroys value.
Common Guardrail Metrics
Revenue guardrails: revenue per session, average order value, cancellation rate. Experience guardrails: page load time, error rate, bounce rate. Business guardrails: support ticket volume, return rate, churn rate. The specific guardrails depend on your business model and what could go wrong.
Setting Guardrail Thresholds
Guardrails typically use one-sided tests: you're not looking for improvement, you're looking for degradation beyond a threshold. Common practice: flag any guardrail that degrades by more than 2-3% with statistical significance. Some organizations use absolute thresholds (page load time must stay under 3 seconds).
Practical Application
Every experiment should have 2-4 guardrail metrics defined before launch. Include at least one revenue metric, one experience metric, and one business health metric. If any guardrail trips, the experiment should be paused and investigated — even if the primary metric is winning. A "win" that breaks a guardrail is not a real win.