In a first-person internal NRG workflow estimate, analysis time moved from roughly eight hours to five per test after AI-assisted steps were added. This was an uncontrolled before/after estimate—not proof that AI alone caused the change, an externally audited productivity study, or a forecast for another team.
This Is Not an "AI Is Amazing" Post
I want to be upfront about what this post is and isn't. This is not a breathless recounting of how AI changed everything. AI didn't change everything. It changed specific parts of my workflow in measurable ways, and it introduced new risks that required governance frameworks I didn't have before.
What I'm going to share are the exact workflows where AI tools save me meaningful time, the places where I tried to use AI and it made things worse, and the governance standards I built to keep the whole thing responsible.
The NRG portfolio includes 150+ historical experiments and ran 100+ in 2025. Each readout requires analysis, synthesis, and stakeholder communication. The roughly 40% figure describes the observed workflow change after AI-assisted steps and other process changes—not a clean AI treatment effect.
Session Replay Analysis: Where AI Earns Its Keep
Before AI-assisted features entered my workflow, session replay analysis often meant reviewing dozens of recordings, tagging candidate friction points, and writing up findings. In my own time estimate, a review could take roughly four to six hours.
Now approved analysis features can surface candidate drop-off, rage-click, dead-click, and hesitation patterns for human review. Their output is triage, not a validated behavioral explanation.
But here's the important caveat: AI summarization of session replays is a first pass, not a final analysis. I still inspect a deliberately selected sample manually. An explicitly synthetic example might flag that “42% of reviewed sessions hesitated at step three,” but that pattern does not establish why. Pricing confusion, slow loading, or comparison behavior remain competing hypotheses.
The behavioral mechanism—the why behind the behavior—still requires product context, competing explanations, and a testable follow-up. AI can provide a faster starting point; it does not provide the conclusion.
Internal workflow estimate: Session replay analysis moved from roughly four to six hours to roughly one and a half to two hours per test after the process changed. This uncontrolled estimate does not isolate AI from templating, experience, or other workflow improvements.
Test Analysis Acceleration
This is where the 40% overall number comes from. After a test concludes, there's a stack of analysis work:
- Statistical review (sample adequacy, decision rule, effect estimate, and uncertainty)
- Segment analysis (device, browser, traffic source, customer type)
- Revenue projection (lift applied to traffic volume and revenue per customer)
- Insight synthesis (what happened, why, and what it means)
- Executive report generation
I use Claude and ChatGPT differently depending on the task.
AI-assisted calculation checks and synthesis: I use approved tools on aggregated test results to check calculations, flag questions for investigation, and draft narrative structure. AI output is not statistical validation; the pre-approved procedure, reproducible calculations, and human review determine the final readout.
AI-assisted report drafting: A template and approved aggregated findings can produce a structured first draft that I edit for accuracy, evidence labels, and tone. My internal time estimate moved from roughly 45–60 minutes for a blank-page draft to about 20 minutes of editing; that is not a controlled tool comparison.
Platform-assisted prompts: Built-in analysis or recommendation features can flag candidate patterns and follow-up questions. They do not replace the approved statistical procedure or decision rule.
What I don't do: I never let an AI tool make the final ship decision. AI can check calculations and surface questions; the approved analysis, guardrails, implementation context, and accountable human owner determine the recommendation.
First-person workflow estimate: Full analysis moved from approximately eight hours per test to approximately five after AI-assisted steps and process changes were added—roughly 37–40% faster in this uncontrolled before/after record.
Hypothesis Generation: AI as a Brainstorming Partner
Here's a workflow that surprised me with its effectiveness. Before each sprint planning cycle, I need to generate hypotheses for the next round of experiments. Historically, this came from three sources: session replay analysis, stakeholder requests, and my own observations from monitoring analytics dashboards.
Now I add a fourth source: AI-assisted hypothesis generation.
The prompt engineering matters enormously here. "Give me ideas for improving our enrollment flow" generates generic garbage. But this prompt structure works:
"In this synthetic example, an enrollment flow loses 23% of users between plan selection and account creation, reviewed sessions suggest hesitation at verification, and mobile conversion is 40% below desktop. Generate five falsifiable hypotheses. For each, name a proposed mechanism, primary outcome, guardrail, and measurable prediction."
That structure produces candidates I can evaluate. The useful output is not a guaranteed idea count or time saving; it is a set of explicit hypotheses that can be rejected against traffic, prior evidence, implementation risk, and business value.
The critical point: The human still needs to validate every AI-generated hypothesis against the data. AI doesn't know your traffic volumes, your MDE thresholds, or whether a similar test already ran and lost last quarter. It generates possibilities. You filter them through reality.
AI-Assisted Personalization
This is where tool integration can create both leverage and governance risk. The project used a customer-data layer and an experimentation platform, with AI-assisted analysis inside approved boundaries.
Dynamic content delivery: Behavioral events can define candidate segments for controlled personalization. AI may suggest segments, but eligibility, privacy review, sample size, and a pre-agreed outcome still determine whether a test should run.
One internal NRG readout recorded a 23% relative increase in its defined lead-acquisition metric versus control during the test window. The bundled workflow did not isolate AI as the cause, and the historical company result is not a client forecast.
Behavioral segmentation: A governed workflow can use approved events to test different experiences for defined segments. It should not infer sensitive traits or deploy treatment rules without privacy, fairness, and measurement review.
This is powerful, and it's also where governance becomes critical.
AI Safety and Governance
I want to spend real time on this section because it's the part most "AI in marketing" content skips entirely. When you integrate AI tools into workflows that touch customer data and business decisions, you need governance standards. Period.
Here's what I built at NRG, informed by the broader governance frameworks we developed for the experimentation program:
The controls also align with the NIST AI Risk Management Framework, which organizes AI risk work around governance, mapping, measurement, and management. For the experimentation layer, Microsoft Research's catalog of common controlled-experiment pitfalls is a useful reminder that faster analysis is worthless if the underlying readout is invalid.
Data privacy guardrails: No personally identifiable information goes into AI prompts. When I feed test results to Claude, the data is aggregated — conversion rates and sample sizes, not individual user records. This sounds obvious, but I've seen analysts copy-paste raw data exports into ChatGPT without thinking about what's in those exports.
Model governance standards: Use only tools and use cases approved through the company’s security, privacy, legal, and data-governance process. Approval is purpose-specific and does not transfer automatically to deployment decisions or customer-facing output.
Bias review: AI models can amplify patterns in historical data. A hypothetical device segment, for example, can correlate with factors the model does not understand. Review who benefits, who is excluded, whether sensitive traits are being inferred, and whether the proposed treatment is acceptable.
Transparency: When I present AI-assisted analysis, I distinguish AI-drafted material from the human-reviewed calculations and recommendation. Undisclosed automation in a material decision can damage trust even when the final number is correct.
Guardrails for personalization: Not every behavioral pattern should be acted on. We have rules about what personalization triggers are acceptable (showing relevant plan information based on browsing behavior) versus what crosses the line (dynamic pricing based on inferred income level). AI makes the second category easier to implement, which means governance has to be more vigilant, not less.
Claude Code for Building Internal Tools
This is a different use of AI that's worth its own section. Beyond using AI for analysis, I use Claude Code to build internal tools that make the experimentation program more efficient.
Sample size calculators: I built a custom calculator around the program’s traffic and seasonal inputs. In my own uncontrolled build-time estimate, AI-assisted implementation took about 30 minutes versus roughly a day for a prior manual approach; deterministic tests and statistical review—not generation speed—were the acceptance criteria.
Automated report generation: I built scripts that pull test results from our analytics platforms, calculate key metrics, and generate draft reports in a standardized format. The reports still need human review and narrative, but the data compilation and formatting is handled automatically.
Internal dashboards: Quick visualization tools for monitoring active experiments, tracking the testing pipeline, and surfacing tests that are approaching significance. These aren't production-quality dashboards — they're internal tools that save me from manually checking Optimizely and Adobe Analytics separately for each active test.
The common thread is that Claude Code lets me build tooling that would normally require a dedicated developer. As a one-person experimentation team (with agency support for execution), this is genuinely transformative for my productivity.
What AI Cannot Do
I want to end with the limitations because they're more important than the capabilities.
AI cannot build stakeholder relationships. Scaling from roughly 20 annual tests to 100+ in 2025 required product, brand, finance, engineering, and analytics coordination alongside tools and process. A model cannot own those relationships or the resulting decisions.
AI cannot navigate organizational politics. Every enterprise has politics. Which team owns the homepage. Which brand gets priority for testing resources. How to position a losing test so the VP who championed it doesn't kill the program. These are judgment calls that require understanding the human dynamics of your specific organization.
AI cannot understand organizational context. An AI model doesn't know that the CMO just changed, and the new one has different priorities. It doesn't know that last quarter's reorg moved the enrollment flow team to a different department. It doesn't know that the developer who maintains the test implementation layer is on paternity leave and his replacement doesn't know Optimizely.
AI cannot make judgment calls on test priorities. Given limited testing slots and competing requests, which test should run next? That decision involves traffic forecasts, stakeholder relationships, strategic priorities, technical complexity, opportunity cost, and organizational timing. AI can provide data inputs to that decision. It cannot make it.
AI cannot replace the consultant mindset. The most valuable thing I do isn't analysis — it's acting as a consultant, not a reporter. Stakeholders don't need more data. They need someone who understands their business context well enough to make recommendations they trust. AI is a tool in that process, not a substitute for it.
The roughly 40% workflow estimate is useful operating evidence, with limits: it is an uncontrolled before/after measure and does not prove AI alone caused the change. It suggested more analysis capacity while keeping accountable judgment, statistical validation, and stakeholder decisions human-owned.
If you want to talk about integrating AI into your experimentation workflow responsibly, contact me or email atticus@atticusli.com. I'm happy to share what's worked and, more importantly, what hasn't.
FAQ
Does the roughly 40% estimate prove AI caused faster analysis?
No. It is a first-person, uncontrolled before/after workflow estimate. Templating, experience, process changes, and AI-assisted steps changed together, so the estimate cannot isolate AI as the cause.
Can AI make the final decision on an experiment?
Not in this workflow. The approved analysis procedure, reproducible calculations, guardrails, implementation context, and accountable human owner determine the recommendation.
What data should an experimentation team put into an AI tool?
Only data and use cases approved by the organization's security, privacy, legal, and data-governance process. In this workflow, prompts used aggregated results rather than personally identifiable customer records.