Claude is a strong second reader for the parts of data science that happen before and after the model: framing the question, designing the experiment, choosing a test and saying out loud what its assumptions are, then reading the result without overselling it. It is weak where numbers have to be exact without a calculator, and it cannot know how your data was really collected. Work in four passes (design, size, test, readout), make it write assumptions down before any result, and have it run code for every number you will quote.
Su questa pagina
- What do you tell it before it designs an experiment?
- How do you get a design doc that cannot be quietly changed later?
- How do you size the test without trusting its arithmetic?
- How do you choose a test and check its assumptions?
- How do you read a result without overselling it?
- What does a useful answer look like?
- Where does Claude fail at this?
- Which skill fits which job?
- How do you install a Leila skill in Claude?
- What does All-in-one cost?
- Related guides
- Frequently asked questions
- faq_jsonld
If you want the spreadsheet-and-dashboard side, our data analyst guide covers it. This one is about the decisions that make an analysis defensible: what you randomised, what you measured, which test fits, and what you can honestly conclude.
Data Scientist AI Skill - Leila
Get Data Scientist AI Skill - Leila →What do you tell it before it designs an experiment?
A request like "design an A/B test for our new checkout" gets a tidy template and no thinking. The design hangs on facts only you have:
The decision. What will you do if the variant wins, and what if it loses? If the answer is the same in both cases, you do not need the test.
The unit of randomisation. User, session, account, store, region. If users share an account or influence each other, randomising sessions quietly breaks independence.
One primary metric, and the guardrails. Name the single metric the decision hangs on, and two or three that must not get worse. Everything else is exploratory and must be labelled that way.
The smallest effect worth acting on. Not the effect you hope for. The effect that would change the decision given the cost of shipping it.
Baseline and variance, from real data. Current conversion or the mean and standard deviation of the metric. Claude has no way to know these and will invent plausible ones if you let it.
The skill this guide is built on: Data Scientist AI Skill - Leila ($19). Every prompt below is the kind it ships with, already loaded, so you bring the facts and it holds the rules. Get it →
How do you get a design doc that cannot be quietly changed later?
The biggest threat to an experiment is the analyst, not the maths: stopping when the number looks good, switching metrics, slicing until something is significant. A short written plan, made before data arrives, removes most of that room.
Item 7 is the part teams skip. Writing "inconclusive" as an outcome in advance stops the post-hoc search for a subgroup that proves the test worked.
How do you size the test without trusting its arithmetic?
Sample size is where Claude is both most useful and most likely to embarrass you. The reasoning is reliable. The calculation, done in prose, is not. A rough check you can do in your head: for two groups, a two-sided 5% test and 80% power, you need about 16 × variance ÷ (difference)² users per group. For a conversion rate near 10%, a one-point lift needs on the order of fourteen thousand users per group, and halving the detectable lift roughly quadruples that. That rule of thumb is a sanity check, not a substitute for a proper power calculation.
Asking for a side-by-side is the trick. It shows you how sharply the cost rises as the effect shrinks, which is usually what changes the conversation with a stakeholder.
How do you choose a test and check its assumptions?
"Which test should I use?" is the wrong question. The test is a consequence of the design and the data. Give Claude both, and make it argue the choice.
Item 5 forces the honest outcome. If the unit of analysis is wrong (analysing sessions when users were randomised, say), no test choice rescues it, and a good answer says so.
How do you read a result without overselling it?
After the data is in, ask Claude to argue against you. It is better at being a sceptical reviewer than at being a confident analyst.
What does a useful answer look like?
An illustration, with invented numbers, of the shape of a good reply to the sample-size prompt:
Statistician AI Skill - Nuria
Get Statistician AI Skill - Nuria →Where does Claude fail at this?
It computes in prose and gets it slightly wrong. A sample size or a p value quoted from its head can be off by a lot with total confidence. Insist on code execution and read the code, not just the answer.
It will not catch a broken randomisation on its own. Sample ratio mismatch, a tracking bug on one arm, bots in one bucket. It checks these only if you ask and supply the counts.
It tolerates peeking. If you say "we checked daily and it hit significance on day 6", a default answer may congratulate you. Ask explicitly what peeking did to the false positive rate.
It reasons about causality from correlation too readily. On observational data, ask it to list confounders and the assumption that would be needed for a causal reading. Do not accept "the data suggests that X drives Y".
It cannot see the data-generating process. Whether users are independent, whether the event log double-counts, whether a metric was redefined mid-test: only the person who owns the pipeline knows. Write those facts into the prompt.
It over-explains significance and under-explains practical size. Always ask for the effect in business units, with an interval, before the p value.
Which skill fits which job?
| Skill | Best for | Price |
|---|---|---|
| Leila - Data Scientist | Framing, EDA, A/B design with power and guardrails, modeling, honest readouts | $19 |
| Nuria - Statistician | Test choice with assumption checks, effect sizes and intervals, multiple-comparison correction, APA-style reporting | $19 |
| All-in-one | Both, plus analyst, SQL and reporting skills | $12/mo |
How do you install a Leila skill in Claude?
With an All-in-one subscription there is nothing to download. Add KissMySkills to Claude once as a connector: open Customize → Connectors, click +, then Add custom connector, and paste https://mcp.kissmyskills.com/mcp. Sign in with your email, then ask for the skill by name, for example "load Leila". The same connector works in ChatGPT, Claude Code, Codex and Cursor; setup for each is on the connector page. Signing in is free, and your first skill call starts a 7-day free trial with every skill.
Bought the single file? You download one .skill file, which is a ZIP archive with SKILL.md inside (prompt packs arrive as a .txt file you paste in). In Claude, turn on code execution under Settings → Capabilities, then open Customize → Skills, click +, choose Upload a skill, pick the file as it is and toggle it on. In ChatGPT, Gemini or another chat, open the file as a ZIP and paste the contents of SKILL.md into custom instructions.
What does All-in-one cost?
All-in-one is $12 a month or $69 a year and includes every skill, prompt pack and agent in the catalog. Signing in is free, the first skill call starts a 7-day free trial with every skill, and there is a 30-day money-back guarantee on your first payment. See why a subscription, the Library page and the MCP connector.
Related guides
- How to Use Claude as a Data Analyst
- How to Use Claude for Conversion Rate Optimisation
- How to Use Claude for SQL Queries
- How to Use Claude for Marketing Analytics
Frequently asked questions
Is there a Claude skill for data science?
Yes. Leila - Data Scientist ($19) covers problem framing, EDA, A/B test design with power, sample size and guardrails, modeling and communicating the decision. Nuria - Statistician ($19) goes deeper on test choice, assumption checks, effect sizes and multiple-comparison correction.
Can Claude design and analyse an A/B test?
Partly. It is good at writing the pre-analysis plan, listing threats to validity and reviewing a result sceptically. Have it run code for sample size and p values rather than quoting them from text, and supply the baseline and variance from your own data.
Can Claude choose the right statistical test?
It can recommend one and state the assumptions, if you describe the design, the unit of randomisation and the shape of the outcome. Ask for a second-choice test and for the check that would make it refuse to recommend any test.
Should I trust the sample size Claude gives me?
Only if it came from executed code you can read. A rough check is that two groups at 5% two-sided and 80% power need about 16 times the variance divided by the squared difference per group. Anything far from that deserves a second look.
What is Claude bad at in data science?
Exact arithmetic in prose, spotting broken randomisation unless you give it the counts, and inferring causes from observational data. It also cannot know how your data was collected, so write that into the prompt.
faq_jsonld
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "Is there a Claude skill for data science?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. Leila - Data Scientist ($19) covers problem framing, EDA, A/B test design with power, sample size and guardrails, modeling and communicating the decision. Nuria - Statistician ($19) goes deeper on test choice, assumption checks, effect sizes and multiple-comparison correction."
}
},
{
"@type": "Question",
"name": "Can Claude design and analyse an A/B test?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Partly. It is good at writing the pre-analysis plan, listing threats to validity and reviewing a result sceptically. Have it run code for sample size and p values rather than quoting them from text, and supply the baseline and variance from your own data."
}
},
{
"@type": "Question",
"name": "Can Claude choose the right statistical test?",
"acceptedAnswer": {
"@type": "Answer",
"text": "It can recommend one and state the assumptions, if you describe the design, the unit of randomisation and the shape of the outcome. Ask for a second-choice test and for the check that would make it refuse to recommend any test."
}
},
{
"@type": "Question",
"name": "Should I trust the sample size Claude gives me?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Only if it came from executed code you can read. A rough check is that two groups at 5% two-sided and 80% power need about 16 times the variance divided by the squared difference per group. Anything far from that deserves a second look."
}
},
{
"@type": "Question",
"name": "What is Claude bad at in data science?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Exact arithmetic in prose, spotting broken randomisation unless you give it the counts, and inferring causes from observational data. It also cannot know how your data was collected, so write that into the prompt."
}
}
]
}Data Scientist AI Skill - Leila
Or get it with every other skill in All-in-one: $12/month, 7 days free, every skill.