TL;DR: A p-value is not the probability that a finding is true. It is the probability of observing a result at least as extreme as the one found, assuming the null hypothesis is correct. A statistically significant result (p < 0.05) does not establish that an effect is large, real, or biologically meaningful, especially in small-sample preclinical studies. To evaluate peptide research honestly, researchers need to read p-values alongside effect sizes, confidence intervals, and sample size, and understand how p-hacking and publication bias distort the published literature.
Research-Use Disclaimer: This article is for educational and research reference purposes only. The compounds referenced are research chemicals not approved by the FDA for human use. This content does not constitute medical advice, does not recommend human administration of any compound, and does not describe protocols for personal use. For adults 21+ with a research interest only.
What Is a P-Value? The Precise Definition
The p-value is one of the most cited, and most misunderstood, numbers in biomedical research. Before examining common errors in its interpretation, a precise definition is necessary.
A p-value is the probability of obtaining a result at least as extreme as the one observed in a study, if the null hypothesis were true. The null hypothesis typically holds that there is no effect, no difference, or no association between the variables under investigation. A p-value of 0.03, for example, means that if the null hypothesis were true, there would be a 3% probability of seeing a result as extreme as (or more extreme than) the one found, by chance alone.
The American Statistical Association (ASA), in its landmark 2016 statement on p-values, the first time in its 180-year history the ASA formally addressed the use of a specific statistical procedure, established six core principles for interpreting p-values in research. The statement, authored by Wasserstein and Lazar, was published in The American Statistician (DOI: 10.1080/00031305.2016.1154108). Its core message: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.”
A 2019 review by Quatto, Ripamonti, and Marasini in the Journal of Biopharmaceutical Statistics, retrieved via PubMed, directly traced this problem: “p-values have been frequently misunderstood and misused in practice, and medical research is not an exception, ” noting the debate culminated in the publication of the ASA statement in 2016 (DOI: 10.1080/10543406.2019.1632874). The review argues that the p-value should always be supplemented by complementary measures, particularly effect sizes and Bayesian indicators, to avoid over-reliance on a single threshold.
The Six Most Common Misreadings of a P-Value
As documented in a 2018 peer-reviewed editorial by Smith in the American Journal of Physical Anthropology, retrieved from PubMed, the following misinterpretations of p-values and null hypothesis significance testing (NHST) are both well-characterized in the statistical literature and persistently common across empirical research disciplines (DOI: 10.1002/ajpa.23399):
| Misinterpretation | Why It Is Wrong |
|---|---|
| 1. p < 0.05 means the finding is true | A p-value is a conditional probability under the null hypothesis. It says nothing about the probability that the alternative hypothesis is correct. |
| 2. p is the probability the null hypothesis is true | Incorrect. A small p indicates that the observed data are unlikely under the null, not that the null is false with any given probability. |
| 3. p measures the size or importance of the effect | As Smith (2018) documents: “P values are a function of both the sample size and the effect size”, a tiny effect can yield p < 0.001 with a large enough sample. P and effect magnitude are independent. |
| 4. p > 0.05 proves no effect exists | A non-significant p-value means the study failed to detect a statistically significant difference, it does not confirm the null. A small study may simply lack the power to detect a real effect. |
| 5. The 0.05 threshold is a natural or universal boundary | The 0.05 cutoff is a convention, originally suggested by Fisher as a rough heuristic. It is arbitrary. Smith (2018) notes it represents “an arbitrary dichotomization of continuous variation.” |
| 6. A replicated p < 0.05 confirms the original finding | P-values have poor replicability. The same experiment run twice under identical conditions will frequently produce different p-values, especially when sample sizes are small. |
The practical consequence for researchers reading peptide studies: a p-value alone, even a small one, cannot be read as confirmation that a compound produced a real, meaningful, or reproducible biological effect. It is a single statistical signal that must be interpreted in context.
Statistical Significance vs. Clinical (Practical) Significance
One of the most consequential distinctions in research literacy is the separation between statistical significance and clinical or practical significance. These are not synonyms. They measure entirely different things, and conflating them produces systematically misleading conclusions.
Statistical significance indicates that an observed difference is unlikely to be due to chance alone, given the size of the sample and the variability in the data. It is a function of three things: the magnitude of the effect, the variability of the measurements, and the sample size. A study with a very large number of subjects, thousands of rodents across multiple experimental cohorts, can detect a statistically significant difference that is, in absolute biological terms, extremely small.
Practical (clinical) significance asks a different question: is the magnitude of the effect large enough to matter? Is a 2% change in a biomarker reading meaningful in the context of the research model being studied? Does the observed effect in a rodent muscle injury model translate to a biologically plausible magnitude in the relevant tissue system?
The gap between these two concepts is a particularly acute problem in preclinical peptide research, where studies typically use small rodent cohorts of 6–12 animals per group. In such designs, a single outlier can drive a statistically significant result. The observed effect size, not just the p-value, is the correct unit of analysis for evaluating whether a finding is meaningful.
What Is Effect Size? Cohen’s d and Related Measures
An effect size is a quantitative measure of the magnitude of a difference or association, expressed in a standardized form that allows comparison across studies. Where a p-value tells researchers whether a difference exists (under the null hypothesis framework), an effect size tells them how large that difference is.
The most commonly reported effect size in biomedical research is Cohen’s d, which expresses the difference between two group means in units of pooled standard deviation. Jacob Cohen proposed conventional interpretive benchmarks in his foundational 1988 work Statistical Power Analysis for the Behavioral Sciences, widely cited across the biomedical literature:
| Cohen’s d Value | Conventional Label | Interpretive Note |
|---|---|---|
| ~0.2 | Small effect | Difference is present but may not be detectable without large samples |
| ~0.5 | Medium effect | Difference visible with adequate samples; practically notable in many contexts |
| ~0.8 | Large effect | Difference is substantial and likely to be detected even in moderate samples |
Cohen himself explicitly cautioned that these benchmarks are context-dependent approximations, not rigid rules. What constitutes a meaningful effect in a tendon biomechanics model differs from what constitutes a meaningful effect in a gene expression assay.
Other effect size metrics include Hedges’ g (a corrected version of Cohen’s d suitable for small samples), eta-squared (η²) and omega-squared (ω²) (used in ANOVA designs), Pearson’s r (for correlational designs), and odds ratios or risk ratios (for binary outcomes in clinical studies). Regardless of metric, the principle is the same: the size of the effect is an independent and essential piece of information beyond the p-value.
A practical point for reading preclinical peptide studies: many papers report statistically significant differences without reporting any standardized effect size measure. In such cases, researchers can estimate the effect magnitude from reported means and standard deviations, or treat the finding as incompletely characterized until replication with full reporting is available.
Confidence Intervals: What They Are and What They Are Not
A confidence interval (CI) is a range of values constructed from the study data within which the true population parameter is estimated to fall, with a specified level of confidence, typically 95%. A 95% CI of [0.2, 1.4] for a between-group difference means that if the same study were repeated many times and a CI constructed from each repetition, approximately 95% of those intervals would contain the true population value.
Confidence intervals convey information that p-values do not:
- Direction of the effect: A CI entirely above zero indicates a positive effect; one straddling zero is consistent with no effect.
- Magnitude of the effect: A narrow CI centered on a large value suggests a large and precisely estimated effect. A wide CI indicates high uncertainty.
- Practical significance: Even a CI that excludes zero (statistically significant) may include only small values, indicating a real but practically unimportant effect.
A common misreading of CIs parallels the misreading of p-values: interpreting a 95% CI as meaning “there is a 95% probability the true value lies within this specific interval.” That is incorrect. The correct interpretation is frequentist: 95% of intervals constructed by this procedure across repeated experiments would contain the true parameter. The specific interval from one study either does or does not contain the true value, the probability is not assigned to that interval specifically.
In the context of evaluating a peptide study, a CI that barely excludes zero, for example, a 95% CI of [0.01, 2.1], is a weaker result than one with a CI of [0.8, 2.1] for the same nominal p-value. The width and placement of the interval is as informative as whether it crosses zero. Vetter (2017) in Anesthesia and Analgesia notes that numerous journals, including flagship publications, “strongly encourage or require the reporting of pertinent confidence intervals” precisely because they carry information the p-value alone does not (DOI: 10.1213/ANE.0000000000002471).
Sample Size and Statistical Power: Why Underpowered Studies Are Dangerous
Statistical power is the probability that a study will detect a true effect of a given size, if that effect actually exists. It is conventionally set at 0.80, meaning an 80% probability of correctly rejecting a false null hypothesis. Power is determined by three factors: the significance threshold (α), the effect size being sought, and the sample size. For a fixed significance level and effect size, larger samples produce greater power.
The practical consequence: underpowered studies, those with insufficient sample sizes to reliably detect the effect being tested, have two interacting problems. First, they are likely to miss real effects (false negatives). Second, and less intuitively, when an underpowered study does produce a statistically significant result, that result is more likely to be inflated or unreliable than the same significant result from an adequately powered study. This phenomenon, sometimes called the “winner’s curse”, means that statistically significant findings from small-sample studies tend to overestimate the true effect magnitude on average.
The neuroscience literature documented this problem rigorously. A 2018 commentary by Algermissen and Mehler in Journal of Neurophysiology, retrieved from PubMed, explicitly engaged with the “power failure” diagnosis from Button et al.’s landmark 2013 meta-analysis and reviewed techniques to improve statistical power in neuroscience studies, concluding that publication bias and researcher bias remained active inflators of apparent power estimates (DOI: 10.1152/jn.00765.2017).
For peptide researchers: most preclinical animal studies use 6–12 subjects per group, cohort sizes determined largely by cost and ethical constraints in animal research, not by formal power calculations. These designs are often adequate for detecting large effects but may be systematically underpowered for detecting medium or small effects. A statistically significant finding in such a study warrants scrutiny of the reported effect size and the plausibility of the magnitude, not just the p-value.
The Replication Problem and P-Hacking: Why Published Results Are Not Facts
The replication crisis, the documented failure of a substantial proportion of published findings to replicate in independent experiments, is directly relevant to how peptide research should be read and weighted.
P-hacking (also called data dredging or outcome switching) refers to the practice of analyzing data in multiple ways and reporting only the analyses that produce a significant p-value. Common forms include: testing multiple outcome variables and reporting only significant ones; testing multiple subgroups and reporting the significant subgroup; stopping data collection when significance is reached rather than at a predetermined sample size; and excluding outliers selectively to achieve significance. These practices inflate the false-positive rate dramatically, a study with 20 tested outcomes that reports only the one significant at p < 0.05 has, under the null, roughly a 64% chance of having a false positive among those 20 tests.
Nissen, Magidson, Gross, and Bergstrom (eLife, 2016), retrieved from PubMed, modeled this dynamic formally: using a Markov process framework to simulate how scientific communities update their beliefs based on published results, they demonstrated that “unless a sufficient fraction of negative results are published, false claims frequently can become canonized as fact, ” and that “data-dredging, p-hacking, and similar behaviors exacerbate the problem” (DOI: 10.7554/eLife.21451). Their model documented that this failure mode is most severe in research domains where positive results are strongly preferred for publication over negative or null results, a known feature of the preclinical compound literature.
For researchers evaluating any peptide compound’s literature base, this has direct operational consequences:
- A single statistically significant study is not replication. A compound with “one positive rodent study” has not had its signal confirmed.
- Multiple positive findings from the same research group, even at different doses or in different injury models, are weaker evidence than findings from independent groups.
- The absence of published null results in a compound’s literature is not evidence that null results do not exist. It may reflect publication bias rather than uniform positive findings.
- Pre-registration of hypotheses and outcomes before data collection, increasingly standard in clinical research, remains rare in preclinical compound studies. Its absence means the line between confirmatory and exploratory analysis is often unclear.
Why Statistical Literacy Matters for Reading Peptide Studies
The research base for most peptide compounds sits at the preclinical level: rodent injury models, in vitro assays, and small-n controlled experiments. These studies are genuinely valuable as hypothesis generators, they identify mechanisms worth investigating, flag safety signals, and provide initial dose-response characterization. But they are also the study type most vulnerable to the statistical issues described in this article: small samples, multiple outcome testing, researcher degrees of freedom, and publication bias toward positive results.
Reading a peptide study rigorously requires examining all of the following, not just the p-value reported in the abstract:
- What is the sample size, and was a power calculation reported?
- Was an effect size (Cohen’s d, Hedges’ g, or equivalent) reported alongside the p-value?
- Are confidence intervals provided for the primary outcomes?
- How many outcomes were measured, and how many are reported as significant?
- Has the finding been independently replicated in a different laboratory?
- Does the study pre-register its hypotheses and primary endpoints, or is this exploratory analysis?
A study that reports p = 0.04 from a cohort of eight rodents, tests twelve outcome variables, and is from a single laboratory that has produced all prior positive findings on the same compound deserves a very different confidence weight than a study reporting p = 0.001 with a Cohen’s d of 1.2, a 95% CI of [0.8, 1.6], from a pre-registered protocol, independently replicated in two labs. The p-value in isolation provides no information about which scenario applies.
For a broader foundation in reading individual studies, see also How to Read a PubMed Abstract, What Is a Randomized Controlled Trial, and the guide to Evaluating Peptide Research Quality.
Frequently Asked Questions
What does a p-value actually mean?
A p-value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true. It is not the probability that the finding is real, not the probability the null hypothesis is true, and not a measure of effect magnitude. The American Statistical Association’s 2016 statement, the most authoritative guidance document on this topic, explicitly clarifies all of these distinctions. A p-value below 0.05 does not confirm that a compound works; it indicates only that the observed result would be unlikely under the specific assumption of no effect.
What is the difference between statistical significance and clinical significance?
Statistical significance (p < 0.05) reflects that an observed difference is unlikely to be due to chance, given sample size and data variability. Practical or clinical significance reflects whether the magnitude of that difference is biologically or therapeutically meaningful. These are independent: a trivially small effect can achieve statistical significance with a large enough sample. In peptide preclinical research, distinguishing between the two requires examining effect sizes, such as Cohen’s d, not just p-values.
What is Cohen’s d and how is it interpreted?
Cohen’s d is a standardized effect size measure expressing the difference between two group means in standard deviation units. Cohen’s conventional benchmarks treat d ≈ 0.2 as a small effect, d ≈ 0.5 as a medium effect, and d ≈ 0.8 as a large effect. These are interpretive guidelines, not absolute thresholds, the meaningful effect size in any domain depends on the research context. Reporting Cohen’s d alongside a p-value provides the information needed to assess both whether an effect was detected and how large it appears to be.
What is p-hacking and how does it affect the peptide research literature?
P-hacking refers to analytic practices, whether intentional or unconscious, that inflate the probability of obtaining a statistically significant result by chance: testing many outcomes and reporting only significant ones, analyzing subgroups selectively, or stopping data collection when significance is reached. Nissen et al. (eLife, 2016) demonstrated formally that p-hacking, combined with publication bias favoring positive results, can result in false findings becoming accepted as established science. In peptide preclinical research, where small rodent cohorts and multiple outcome measures are standard, p-hacking is a structural risk that makes independent replication essential for evaluating any finding.
For educational and research reference purposes only. Not medical advice. Not for human use.