Data-snooping bias
Also called data-mining bias
The bias that arises when many rules or parameter settings are tested on the same data and the best is reported. Some will look profitable purely by chance, so past results overstate what to expect.

On this page· 5 sections
Test enough trading rules on one stretch of price history and some will look good by luck alone. Report only the best, and the result tells you more about the search than about the rule. That is data-snooping bias.
Researchers can always find patterns, such as a profitable trading rule, that exist purely by chance in one data set and vanish in other samples[1].
How it works
Neely and Weller separate three related problems. Data snooping is choosing, consciously or not, to test rules already proven profitable on part or all of the data tested before. Data mining is testing many rules and basing the conclusion on the most successful, with negative results ignored. Publication bias is journals accepting positive results more readily than negative ones[1].
Technical rules invite the bias because they need parameters, such as a moving-average length, and manuals give little guidance beyond common practice[1]. A CFA Institute Research Foundation review gives the extreme case of an analyst testing every moving-average length from 1 to 10,000 days and picking the one that would have earned most. Because so many variables are tested, the results are not a reliable guide to future performance[2]. Quoting David Aronson, it says findings from selecting a best method among many are upwardly biased[2].
Researchers use several remedies. One is to pick rules on an in-sample period and test them on out-of-sample data. Another is to adjust significance levels for the number of rules considered, as in White’s reality check. The most certain is a true out-of-sample test long after the original study[1]. Neely and Weller add that some snooping is almost impossible to avoid, since data sets and plausible rules are limited[1].
Example
Say you test every moving-average length from 1 to 200 days on ten years of daily prices for one market. That is 200 rules. Suppose none has any real edge, and each has a 5% chance of passing a significance test by luck.
- The chance that one given rule fails the test is 95%.
- If the 200 tests were independent, the chance that all fail is 0.95^200, about 0.0035%.
- So at least one rule passes with a probability near 99.99%.
Neighboring lengths are correlated, so the tests are not independent and the true figure is lower. The direction is the same: the best of many tests looks better than a single test would.
Watch out
The bias needs no dishonesty. Rules that were profitable on past data are the ones that get studied, which can leave a small set of rules that happened to work over one period[1].
It does not prove that every result is false. In a true out-of-sample test of the rules Brock, Lakonishok and LeBaron (1992) found worked best, Fang, Jacobsen and Qin found no evidence that they predicted stock markets over 1987 to 2011[2]. For currency markets, Neely and Weller report that out-of-sample tests and adjusted statistical tests indicate the early returns of simple rules were genuine[1].
Park and Irwin counted 56 of 95 modern studies as positive, 20 negative and 19 mixed. They add that most studies suffer from problems in testing, among them data snooping[3].
Sources
- Central bankFederal Reserve Bank of St. LouisTechnical Analysis in the Foreign Exchange Market (Working Paper 2011-001B, Neely & Weller)Checked November 9, 2026
- Professional bodyCFA Institute Research FoundationTechnical Analysis: Modern Perspectives (Literature Review; Scott, Carr & Cremonie, 2016)Checked November 9, 2026
- ResearchRePEc/IDEAS recordWhat Do We Know About the Profitability of Technical Analysis? (Park & Irwin, Journal of Economic Surveys 21(4), 2007, DOI 10.1111/j.1467-6419.2007.00519.x)Checked November 9, 2026
