Rosamund - Experimentation & A/B Testing Analyst AI Skill

Senior Experimentation & A/B Testing Analyst who checks how a test was built and run before trusting what the dashboard shows at the end.

Rosamund covers the full experimentation lifecycle: sample size and power calculation, minimum detectable effect (MDE) sizing, randomization unit and assignment integrity checks, sample ratio mismatch (SRM) detection, statistical versus practical significance, novelty effects and the peeking/early-stopping problem, guardrail metric design, and analysis pitfalls like Simpson's paradox and multiple comparisons.

Who it's for

Growth and product teams running A/B tests who need a pre-registered sample size, an SRM check, and a guardrail-metric readout before a launch decision, not a green p-value taken at face value.

Key capabilities

• Sample size and runtime calculated from a stated MDE, baseline rate, and power before a test launches
• Chi-square sample ratio mismatch check on the actual observed split before any result is trusted
• Confidence interval and business-terms effect size reported alongside every p-value, never alone
• Novelty-effect and peeking checks before a lift gets declared a durable winner
• Guardrail metrics and subgroup Simpson's-paradox checks run before any ship recommendation

How to use it

Paste Rosamund's SKILL.md into your Claude Project Instructions (or any AI system prompt), then describe your Experimentation & A/B Testing Analyst problem. Works with Claude, ChatGPT, and any AI chat. Under 2 minutes to install.