A/B Test Archives: Build a Knowledge Base That Compounds

Every experimentation team I have worked with has the same problem: they run tests, learn things, and then forget what they learned. Six months later, someone proposes the exact same test that already ran and failed. Nobody remembers, because the results live in a Slack thread that has been buried under 10,000 messages.

An A/B test archive is the solution. It is a structured knowledge base that stores every experiment your team has run — the hypothesis, the setup, the results, and the learnings. Done well, it compounds institutional knowledge, prevents duplicate tests, accelerates onboarding, and transforms your experimentation program from a series of isolated events into a cumulative body of evidence about your users.

This guide covers why archives matter, what they should contain, how to structure them for maximum usefulness, and the cultural habits that keep them alive.

Why Most Teams Lose Their Experiment Learnings

The typical lifecycle of experiment knowledge goes like this: the test runs, someone checks the dashboard, results get shared in a meeting or Slack channel, a decision is made, and everyone moves on to the next test. The knowledge exists briefly in working memory and then vanishes.

Three months later, nobody can answer basic questions: "Did we test social proof on the pricing page?" "What was the conversion rate on the old checkout flow?" "Why did we switch from tabs to accordion on the FAQ page?"

This happens because experimentation culture focuses on the testing and neglects the documentation. Teams invest heavily in test velocity and statistical rigor but treat the output — the actual knowledge — as disposable.

What an Archive Prevents

A well-maintained test archive prevents four specific problems that drain experimentation programs:

1. Duplicate tests. Without an archive, teams re-run experiments that have already been conducted. This wastes traffic, engineering time, and opportunity cost — every duplicate test displaces a novel experiment that could have produced new knowledge.
2. Lost context. When someone proposes a new test, the archive shows what has already been tested in that area. This context makes the new hypothesis sharper because it builds on existing evidence rather than starting from scratch.
3. Onboarding friction. New team members spend months building intuition that already exists in the heads of people who have left the company. An archive transfers that knowledge instantly.
4. Credibility gaps. When a stakeholder asks "how do we know that works?" you need to point to evidence. An archive provides that evidence on demand, which builds trust in the experimentation program.

What Every Archive Entry Needs

Each entry in your archive should map to a single experiment and follow the structure of the experimentation process. Here are the required fields:

Test Metadata

• Test ID. A unique identifier. Use a consistent naming convention like "2024-Q1-PRICING-003" that includes timing, area, and sequence.
• Date range. When the test started and ended. This is critical for understanding what other tests were running concurrently and what external events might have influenced results.
• Owner. Who ran the test. When someone has questions six months later, they need to know who to ask.
• Page or feature. Where in the product or site this test ran. This enables filtering to see all tests on a specific page.
• Status. Running, completed (winner shipped), completed (no winner), or stopped early.

Test Design

• Hypothesis. The full hypothesis statement including the behavioral mechanism. This is the most important field because it captures the theory being tested.
• Screenshots. Visual documentation of both control and variant. Six months from now, a written description will not convey what "simplified the pricing table" actually looked like.
• Metrics. Primary metric, guardrail metrics, and secondary metrics as defined before the test launched.
• Sample size and duration. The pre-calculated required sample size and actual duration.

Results and Learnings

• Outcome. The primary metric result with confidence interval. Not just "variant won" but the specific numbers.
• Segment analysis. Key differences across segments (device, traffic source, user type).
• Decision. What was decided based on the results and why.
• Learnings. What this test taught you about user behavior. This is the compounding knowledge that makes the archive valuable beyond preventing duplicates.
• Follow-up ideas. Future tests suggested by the results. These feed back into the prioritization pipeline.

Structuring the Archive for Maximum Usefulness

The archive is only valuable if people can find things in it. Structure determines whether it gets used or becomes another graveyard of good intentions.

Searchable by Multiple Dimensions

People search for past tests in different ways. Some search by page ("what have we tested on the pricing page?"). Some search by metric ("what tests improved checkout conversion?"). Some search by theme ("what have we learned about social proof?").

Your archive needs to support all of these access patterns. Tags, categories, and full-text search are the minimum. If your archive is a spreadsheet with 200 rows and no filters, nobody will use it.

Tooling Options

The right tool depends on your team size and experimentation volume:

• Small teams (under 10 tests per quarter): A well-structured Notion database or Airtable base works fine. Low overhead, easy to maintain, good enough search.
• Medium teams (10-50 tests per quarter): Dedicated experimentation platforms like Eppo or Statsig include built-in test repositories. If your platform does not, consider a custom internal tool.
• Large teams (50+ tests per quarter): You need a dedicated experimentation hub with role-based access, automated data ingestion from your testing platform, and integration with your analysis tools.

Feeding the Archive Into Your Pipeline

The archive is not just a historical record. It is an active input into your test prioritization process. Every time you evaluate a new test idea, the archive should be consulted to answer:

• Has this been tested before? If so, what happened?
• What do we already know about this page or user flow?
• What behavioral theories have been confirmed or disproven in this area?
• What follow-up ideas were generated by previous tests that have not been pursued yet?

This creates a flywheel: tests generate learnings, learnings inform new hypotheses, new hypotheses generate better tests, and better tests generate deeper learnings. The archive is the mechanism that keeps the flywheel spinning.

Building the Documentation Habit

The archive only works if people actually use it. Here is how to build the habit:

Make it part of the workflow, not an afterthought. The archive entry should be created when the test is set up, not after it concludes. Start with the hypothesis, metrics, and screenshots. Add results when the test ends. This spreads the documentation effort across the test lifecycle instead of dumping it all at the end.

Make it required. A test is not complete until the archive entry is written. No exception. If the team knows that documentation is optional, it will not happen.

Make it useful immediately. Reference the archive in every test planning meeting. When someone proposes a test, pull up related archive entries and discuss what was learned. When people see the archive being used, they invest in maintaining it.

Keep the template simple. If the documentation template takes an hour to fill out, people will avoid it. A good template should take 15 to 20 minutes after the analysis is complete.

Quarterly Archive Reviews

Once per quarter, review the archive as a team. This is different from reviewing individual test results — it is about stepping back to see the bigger picture. Use the same rigor you apply when you analyze individual test results, but at the portfolio level.

Questions to ask in the quarterly review:

• How many tests ran this quarter? What was the win rate?
• What themes emerge across the results? Are certain types of changes consistently winning or losing?
• What areas have been under-tested? What areas have been over-tested?
• What follow-up ideas from past tests should be promoted to the active roadmap?
• Are the predicted lifts from tests matching actual production performance?

This review transforms the archive from a reference tool into a strategic asset. It surfaces patterns that individual test analyses miss and helps the team allocate future experimentation resources more effectively.

Pro Tip: The Learnings Summary

At the end of each quarter, write a one-page "Experimentation Learnings Summary" that distills the key insights from all tests into three to five bullet points. Share this widely — with product, engineering, design, and leadership.

This summary does more for experimentation culture than any individual test result. It shows the organization that testing produces cumulative knowledge, not just isolated wins. It positions the experimentation team as a source of customer insight, not just a conversion optimization service.

The archive is the foundation. The summary is the communication layer. Together, they build the kind of experimentation program that survives leadership changes, team turnover, and strategic pivots.

What to Learn Next

This article covers building and maintaining a test archive. Here is where to go from here:

• The A/B Testing Process — understand the end-to-end workflow that generates the experiments your archive will capture
• How to Prioritize A/B Tests — use archive learnings to prioritize your next experiments more effectively
• How to Set Up an A/B Test — create the test brief that becomes the starting point for your archive entry
• How to Analyze A/B Test Results — the analysis workflow that produces the results and learnings stored in your archive