Statistics for Data-Driven Developers: Making Decisions With Numbers

Central Tendency: Mean, Median, Mode

The mean (average) is the most common summary — and the most misleading when data is skewed. If 9 users pay $10/month and 1 user pays $1,000/month, the mean revenue per user is $109, which describes no one's actual experience[wheelan-statistics].

The median (middle value) is robust to outliers. In the same dataset, the median is $10 — a more honest summary of the typical user. Use median when data is skewed (revenue, response times, house prices) and mean when data is symmetric (heights, test scores, measurement errors).

The mode (most frequent value) matters for categorical data: the most common error code, the most popular feature, the most frequent user country.

Spread: Variance and Standard Deviation

Central tendency tells you where the middle is. Spread tells you how far the data ranges around that middle. Two APIs can have the same mean response time (350 ms) but wildly different consistency: one with σ = 50 ms (predictable) and one with σ = 400 ms (erratic)[nist-handbook].

Variance is the average squared deviation from the mean:

Standard deviation is the square root of variance, bringing the units back to the original data[nist-handbook]:

For the normal distribution, roughly 68% of values fall within ±1 standard deviation of the mean, 95% within ±2, and 99.7% within ±3 — the empirical rule[nist-handbook]. The Standard Deviation Calculator and Variance Calculator compute these from your raw data, including the crucial sample-vs-population distinction (divide by n-1 for samples, n for populations).

Percentiles: The Tail Tells the Story

For skewed data (response times, revenue, file sizes), percentiles are more honest than mean + standard deviation. The p50 is the median. The p95 is the value below which 95% of observations fall. The p99 captures the worst 1%[wheelan-statistics].

Why percentiles matter for monitoring:

• Mean hides the tail (the alert that cried wolf)
• p95 captures the experience of the slowest 5% without being dominated by outliers
• p99 captures the worst 1% — the users who are most likely to churn

Percentiles are not additive. The p95 of a sum is not the sum of the p95s. If API A has p95 = 200 ms and API B has p95 = 300 ms, the end-to-end p95 of calling both is not 500 ms — it's typically less, because the slowest 5% of A's calls don't always coincide with the slowest 5% of B's calls. This is why distributed tracing matters: you need to measure the end-to-end percentile, not sum individual ones[madeyski-reliability].

Distributions: The Shapes of Data

Every dataset has a shape — a distribution. The most important distributions for developers:

Normal (Gaussian) distribution is the bell curve: symmetric, mean = median = mode. Heights, measurement errors, and sums of many independent variables tend to be normal (thanks to the Central Limit Theorem, discussed below)[nist-handbook].

Log-normal distribution is skewed right: the log of the values is normal. Response times, file sizes, and revenue per user tend to be log-normal. The mean is much higher than the median. Use geometric mean or percentiles, not arithmetic mean[madeyski-reliability].

Exponential distribution models time between events: time between requests, time between failures. It has a "memoryless" property: the probability of a failure in the next minute is the same regardless of how long it's been since the last failure.

Poisson distribution models counts of events in a fixed interval: requests per second, errors per hour, support tickets per day. If events arrive independently at rate λ, the count in any interval is Poisson-distributed with mean λ[downey-thinkstats].

The Central Limit Theorem

The Central Limit Theorem (CLT) is one of the most powerful results in statistics: if you take sufficiently large random samples from ANY distribution (normal, skewed, it doesn't matter), the distribution of the sample means will be approximately normal[nist-handbook].

This is why the normal distribution appears everywhere: any metric that's the sum or average of many independent contributions (response times averaged over a window, conversion rates averaged over days) tends toward normal, even if the underlying data is wildly skewed. The CLT is also why you can use normal-based methods (z-tests, t-tests) even when your data isn't normal — as long as your sample size is large enough (typically n > 30)[wheelan-statistics].

Correlation: The Number That Gets Misused

Correlation measures the linear relationship between two variables, ranging from -1 (perfect inverse) to +1 (perfect direct). A correlation of 0 means no linear relationship[nist-handbook].

The Pearson correlation coefficient:

Correlation ≠ causation is the most violated principle in data analysis. Ice cream sales and drowning deaths are correlated — not because ice cream causes drowning, but because both increase in summer (a confounding variable). In software: server load and error rates are correlated, but the cause is a third factor (a deployment that introduced both a memory leak and increased CPU usage)[wheelan-statistics].

Spurious correlations appear by chance when you test enough pairs. If you check 100 unrelated metrics, about 5 will show "significant" correlation at p < 0.05 purely by chance. This is why pre-specifying hypotheses matters — and why data mining without theory produces nonsense[wheelan-statistics].

Confidence Intervals: The Range of Plausible Values

A confidence interval is a range that likely contains the true population parameter. A 95% confidence interval of [2.1%, 2.5%] for a conversion rate means: "if we repeated this experiment many times, 95% of the intervals we'd compute would contain the true conversion rate"[nist-handbook].

The formula for a confidence interval on a mean:

The key insight: the width shrinks with $\frac{1}{\sqrt{n}}$ — to halve the margin of error, you need 4× more data[nist-handbook]. The Confidence Interval Calculator computes this from your sample mean, standard deviation, and sample size, for any confidence level.

Why confidence intervals beat point estimates. Reporting "conversion rate is 2.3%" hides the uncertainty. Reporting "conversion rate is 2.3% ± 0.2% (95% CI)" tells you the precision of your estimate. If the CI for the difference between two variants includes zero, the difference is not statistically significant[georgiev-ab-testing].

Hypothesis Testing: Is This Real or Noise?

Hypothesis testing is the formal framework for deciding whether an observed effect is real or due to chance. The logic: assume the boring explanation (null hypothesis), compute the probability of seeing your data under that assumption (p-value), and reject the null if that probability is low enough[ncbi-pvalue].

The framework:

1. Null hypothesis (H₀): the default assumption (e.g., "the new feature does nothing")
2. Alternative hypothesis (H₁): what you want to prove (e.g., "the new feature increases conversion")
3. Test statistic: a number computed from your data (t-statistic, chi-square statistic, z-score)
4. p-value: the probability of seeing a test statistic at least as extreme as yours, assuming H₀ is true
5. Decision: if p < α (typically 0.05), reject H₀

The p-value is not the probability that H₀ is true. It is the probability of the data given H₀ — not the probability of H₀ given the data. This confusion is so pervasive that the American Statistical Association published a formal statement clarifying it[ncbi-pvalue].

The t-Test Calculator and Chi-Square Calculator compute test statistics and p-values for the two most common tests: t-test for comparing means (revenue, time, score) and chi-square for comparing proportions (conversion rates, click/no-click).

Statistical Power: The Probability You'll Detect a Real Effect

Statistical power is the probability that your test will detect an effect if one truly exists. A test with 80% power has a 20% chance of missing a real effect (Type II error)[cohen-power].

Power depends on three things:

• Effect size: larger effects are easier to detect
• Sample size: more data → more power
• Significance level: lower α → less power (stricter threshold)

The Sample Size Calculator computes how many observations you need for a desired power, given your expected effect size and α. Use it before running any experiment — an underpowered test is worse than no test at all, because it produces false confidence in a "no difference" result[cohen-power].

Monitoring and Alerting

The most common statistical mistake in monitoring is alerting on the mean. For skewed distributions (response times, queue lengths, file sizes), the mean hides the tail. Alert on percentiles instead:

• p95 for "most users are having a good experience"
• p99 for "the worst 1% are having a bad experience"
• Mean only for symmetric metrics (CPU utilization, memory usage)[madeyski-reliability].

Set thresholds using baselines, not absolutes. A response time of 500 ms might be normal for a database query but catastrophic for a cache lookup. Establish per-endpoint baselines using historical percentiles, and alert on deviations from the baseline (e.g., "p95 is 3× higher than the 7-day rolling average")[madeyski-reliability].

A/B Testing and Experimentation

A/B testing is hypothesis testing applied to product decisions. The same principles apply: pre-specify your primary metric, compute sample size in advance, run the test to completion, and report confidence intervals — not just p-values[georgiev-ab-testing].

Common mistakes:

• Peeking: checking results daily and stopping when p < 0.05 inflates false positives
• Multiple comparisons: testing 20 metrics and reporting the one that's significant is p-hacking
• Underpowered tests: running on too few users produces inconclusive results
• Ignoring practical significance: a 0.1% lift may be statistically significant with 1M users but not worth shipping

Data-Driven Development

"Data-driven" means letting evidence guide decisions — but evidence can be misleading if you don't understand the statistics. Before trusting any metric:

1. Check the distribution. Is it normal or skewed? Use mean for normal, median/percentiles for skewed.
2. Look at the spread. A mean of 350 ms with σ = 50 ms is very different from a mean of 350 ms with σ = 400 ms.
3. Compute confidence intervals. A conversion rate of "2.3%" is less useful than "2.3% ± 0.2%."
4. Beware of confounding. Correlation is not causation. A metric may move because of a third factor you're not measuring.
5. Pre-specify hypotheses. Decide what you're testing before you look at the data. Post-hoc rationalization produces false discoveries.

Capacity Planning and Forecasting

Statistics also powers capacity planning: predicting how much infrastructure you'll need. If your traffic grows 10% per month, you can extrapolate when you'll need to scale. But extrapolation requires understanding variance — if traffic is highly variable (high standard deviation), you need more headroom than the average suggests[madeyski-reliability].

The rule of thumb: plan for the p95 of expected traffic, not the mean. If your mean projected traffic is 10,000 requests per second, but the 95th percentile is 15,000, provision for 15,000. Otherwise, you'll be overloaded 5% of the time — which is roughly 3.6 hours per day[madeyski-reliability].

Regression for forecasting. Simple linear regression fits a line to historical data (traffic over time) and extrapolates. The regression calculator on this site fits a line to your data and reports the slope (growth rate) and R² (how well the line fits). Use it to answer: "At current growth rate, when will we hit our limit?" But be cautious — regression assumes the future resembles the past. A viral post, a product launch, or a pandemic breaks that assumption[nist-handbook].

Error Budgets and Reliability

Site Reliability Engineering (SRE) uses statistics to define error budgets. If your SLO (Service Level Objective) is 99.9% uptime, your error budget is 0.1% — roughly 43 minutes of downtime per month. Statistics tells you whether you're on track to meet that budget or whether you've already blown it[madeyski-reliability].

The math: if you've had 30 minutes of downtime in the first 15 days of a 30-day month, you're on pace for 60 minutes — double your budget. But if your downtime events are random (Poisson-distributed), the variance matters. A month with 30 minutes of downtime could be normal variation or the start of a trend. Statistical process control charts (which plot metrics with control limits at ±3σ) distinguish between normal variation and a genuine shift[madeyski-reliability].