P-Values

A p-value is used to decide whether hypothesis test results are statistically significant or not. Once it is calculated from analyzing test data, it is compared to the selected alpha level – if lower than the alpha level, the results are deemed to be statistically significant; if higher, the results are deemed to not be statistically significant.

This value is expressed as a number between 0 and 1.

How a Black Belt Actually Uses a P-Value

On a real project, the p-value is the number that keeps you from fooling yourself. Say you change a machine setting and defect rate drops from 4% to 3.5% over the next week. Is that the fix working, or just normal week-to-week noise the process always had? Run a hypothesis test (a two-proportion test here), calculate the p-value, and compare it to your alpha (usually 0.05). If it is below alpha, the drop is unlikely to be random noise, and you have statistical grounds to say the change worked. If it’s above alpha, you don’t have enough evidence yet, and celebrating the “fix” would be premature.

Sports version of the same question: a hitter changes his batting stance and his average climbs from .260 to .280 over the next month. Did the new stance work, or is a hot month just normal variation for any hitter? This one number answers that exact question, whether it’s a batting stance or a manufacturing process: is this change big enough, relative to the normal noise in the system, to call it real?

The Most Common P-Value Misinterpretation

A p-value of 0.03 does NOT mean “there’s a 3% chance the null hypothesis is true,” even though that’s how most people instinctively read it, and it’s a trap on certification exams. What it actually measures is the probability of seeing data this extreme (or more extreme) if the null hypothesis were true, not the probability that the null hypothesis itself is true. Those are different conditional probabilities, and mixing them up is one of the most common statistics errors, in Six Sigma work and in published research alike.

This Khan Academy video further explains how they apply to hypothesis tests, in-depth:

While the p-value is a standard method for finding the key measurement of one’s results, there is a mix of opinions on whether it’s actually the ideal solution.

Geoff Cumming, Emeritus Professor at La Trobe University in Melbourne, Australia, explains his dislike of p-values in hypothesis testing in the following video:

But then, Jeff Leek and Rafa Irizarry do not agree, and their article lists a number of useful pieces of information about this statistic. The main points (the full list, with explanations, is available in the linked article):

• They’re easy to calculate.
• They’re easy to understand.
• They have simple, universal properties.
• Their calibration is within useful error rates.
• They can be used in correlation with multiple tests.
• They’re reproducible.

Frequently Asked Questions About P-Values

What is the difference between an alpha level and a p-value? (PDF)

This is a fundamental but often confusing question in hypothesis testing. Here’s a clear breakdown of the difference between alpha level and p-value:

• Alpha Level (α): This is the threshold you set before your test; it represents the probability of making a Type I error: that is, rejecting the null hypothesis when it’s actually true. Common values are 0.05, 0.01, or 0.10.
• p-value: This is the probability that the observed data (or something more extreme) could occur under the null hypothesis. It’s calculated from your sample data after the test is conducted.

Key Difference:

• Alpha is the benchmark you compare against.
• p-value is the result you get from your experiment.

If your result is less than or equal to α, you reject the null hypothesis; otherwise, you fail to reject it.

For more detailed insights on hypothesis testing, including both of these terms, visit our article on Hypothesis Testing.

Hope this clears it up!

When you’re ready, there are a few ways I can help:

First, join 12,000+ other Six Sigma professionals by subscribing to my email newsletter. A short read every Monday to start your work week off correctly. Always free.

* * *

If you’re looking to pass your Six Sigma Green Belt or Black Belt exams, I’d recommend starting with my affordable study guide:

1)→ 🟢Pass Your Six Sigma Green Belt: a study guide organized by the exam Body of Knowledge, 1,000+ practice questions with detailed answer explanations, and 8 full-length mock exams

2)→ ⚫Pass Your Six Sigma Black Belt: the full Black Belt Body of Knowledge, 2,000+ practice questions with detailed answer explanations, and 8 full-length mock exams

You’ve spent so much effort learning Lean Six Sigma. Why leave passing your certification exam up to chance? This comprehensive study guide offers 1,000+ exam-like questions for Green Belts (2,000+ for Black Belts) with full answer walkthroughs, access to instructors, detailed study material, and more.

​ Thousands have used this system to prepare. Join them here.