The Statistics Every Data Scientist Actually Uses

You don't need a statistics PhD to do excellent data science, but you do need a solid grip on a handful of concepts that come up every week. These are the ideas that separate "the dashboard says conversions went up" from "conversions went up by an amount that's probably real, and here's how sure we are."

#Key takeaways

• Think in distributions, not single numbers. Report spread and uncertainty alongside averages.
• Samples vary: confidence intervals quantify how much.
• Hypothesis tests and p-values answer a narrow question. They're easy to misuse.
• Correlation isn't causation. Confounders, selection bias and reverse causality are everywhere.
• Watch for Simpson's paradox, survivorship bias and multiple comparisons.

#Descriptive statistics: beyond the average

• Mean is sensitive to outliers; median is robust. Revenue per customer is usually skewed, so report both.
• Standard deviation and interquartile range describe spread.
• Percentiles (p50, p90, p99) matter for things like latency and order values, where tails drive experience (latency numbers every engineer should know).

Always visualize distributions: histograms and box plots reveal skew, multiple modes and outliers that summary numbers hide.

#Common distributions

Knowing which shape to expect helps you choose the right models, tests and summary statistics.

#Sampling and the central limit theorem

You rarely observe a whole population, only a sample. The central limit theorem says that the average of a reasonably large random sample is approximately normally distributed around the true mean, even if the underlying data isn't normal. That's why confidence intervals and many tests work.

The standard error measures how much a sample statistic varies between samples:

textSE(mean) = s / √n

Quadrupling the sample size halves the standard error. That's why detecting small effects requires large samples (A/B testing done right).

Sampling bias matters more than sample size: a million survey responses from your most engaged users tell you little about typical users.

#Confidence intervals

A 95% confidence interval gives a range of plausible values for the true parameter:

textmean ± 1.96 × SE

Example: average order value is $52.40 with a 95% CI of [$50.10, $54.70].

Correct interpretation: if we repeated this sampling process many times, about 95% of the intervals constructed this way would contain the true value. Practically, wide intervals mean you know little; narrow intervals mean more precision. Report intervals, not just point estimates.

pythonimport numpy as np
from scipy import stats

sample = np.array(order_values)
ci = stats.t.interval(0.95, len(sample) - 1, loc=sample.mean(), scale=stats.sem(sample))

For skewed data or complex statistics (medians, ratios), bootstrapping (resampling your data many times) gives robust intervals without distributional assumptions.

#Hypothesis testing and p-values

A hypothesis test asks: if there were no real effect (the null hypothesis), how surprising would our data be?

• The p-value is the probability of observing data at least this extreme, assuming the null is true.
• A small p-value (commonly below 0.05) suggests the data is inconsistent with "no effect".

What a p-value is not:

• ❌ The probability that the null hypothesis is true.
• ❌ The probability that the result is due to chance.
• ❌ A measure of effect size or importance.

A tiny, unimportant difference becomes "significant" with enough data; a large, important difference can be "not significant" with too little data. Always report effect sizes with confidence intervals.

Common tests:

#Errors, power and sample size

• Type I error (false positive): concluding an effect exists when it doesn't. Its rate is α.
• Type II error (false negative): missing a real effect. Its rate is β.
• Power (1 − β): the probability of detecting a real effect of a given size.

Underpowered studies miss real effects and, when they do find something, tend to exaggerate its size. Plan sample sizes before collecting data.

#Multiple comparisons

Test 20 independent metrics at α = 0.05 and you expect about one false positive even when nothing changed. Slice results into many segments and you'll find "significant" effects by chance. Mitigations:

• Pre-register a primary metric and a small set of segments.
• Apply corrections (Bonferroni, or false discovery rate control like Benjamini–Hochberg).
• Treat exploratory findings as hypotheses to confirm in a new test.

#Correlation vs causation

Two variables moving together doesn't mean one causes the other:

• Confounding: a third factor drives both. Users who enable a feature retain better, but maybe power users both enable features and retain better.
• Reverse causality: engaged users contact support more; support contact doesn't cause engagement.
• Selection bias: analyzing only users who completed onboarding hides everyone who dropped out.

Ways to get at causation:

1. Randomized experiments: the gold standard.
2. Quasi-experimental methods when you can't randomize: difference-in-differences, regression discontinuity, instrumental variables, synthetic controls.
3. Careful regression that adjusts for known confounders, with humility about unknown ones.

#Regression: the Swiss army knife

Linear and logistic regression do double duty: prediction and explanation.

• Coefficients estimate how the outcome changes with each variable, holding others constant.
• Confidence intervals on coefficients show uncertainty.
• Beware multicollinearity (correlated predictors make coefficients unstable) and omitted variable bias.
• Check residuals for patterns that indicate a poor model fit.

For predictive modeling beyond regression, see machine learning explained.

#Classic traps

Simpson's paradox. A trend appears in every subgroup but reverses when groups are combined, because group sizes differ. For example, a treatment can look better within each hospital yet worse overall if it's used mostly in the hospital with the sickest patients. Always check whether aggregation hides a confounder.

Survivorship bias. Analyzing only survivors, such as successful companies or long-tenure customers, and inferring causes of success, while ignoring the failures that shared the same traits.

Regression to the mean. Extreme observations tend to be followed by more typical ones. The worst-performing stores this month will likely improve next month, with or without your intervention.

Base rate neglect. A test with 99% accuracy for a condition affecting 0.1% of people still produces mostly false positives. Ignoring base rates misleads fraud, security and medical reasoning. This connects directly to precision on imbalanced data (classification metrics explained).

#FAQ

#How much statistics does a data scientist need?

Enough to design and interpret experiments, quantify uncertainty, avoid common biases and choose appropriate models. Descriptive statistics, probability, sampling, confidence intervals, hypothesis testing and regression cover most day-to-day needs.

#What's the difference between statistical and practical significance?

Statistical significance says an effect is unlikely to be zero; practical significance asks whether it's large enough to matter. A 0.1% conversion lift can be statistically significant with huge traffic but not worth the engineering cost.

#Should I use Bayesian or frequentist statistics?

Both are legitimate. Frequentist methods are widely standardized; Bayesian methods express results as probabilities and incorporate prior knowledge. Choose based on your team's familiarity and the decisions you need to support, and apply either rigorously.

#What is bootstrapping?

A resampling technique: repeatedly sample your data with replacement, recompute the statistic each time, and use the distribution of results to estimate uncertainty. It works well for medians, ratios and other statistics without simple formulas.