Out-of-sample testing
Also called out-of-sample test, hold-out test
Checking a rule on data that was not used to choose or tune it, for example picking rules on one period and evaluating them on a later one. It reduces, but does not remove, data-snooping bias.

On this page· 5 sections
A rule fitted to a chart looks good on that chart. Out-of-sample testing asks whether it still works on data it has not seen.
The reason to ask is a trap in the data. It is always possible to find patterns after the fact, including profitable trading strategies, that exist purely by chance in one data set and not in another[1].
How it works
The sample is split into two parts. The in-sample period is used to select rules that perform well. The chosen rules are then tested on the out-of-sample data[1]. This cuts a very large group of rules down to a small group, which can be tested with less concern about how many rules were tried[1].
It targets two related problems[1]:
- Data snooping: researchers choose, consciously or not, to test rules already proven profitable on part or all of the data.
- Data mining: researchers test many rules and base their conclusion on the most successful ones.
A CFA Institute Research Foundation review, summarizing David Aronson, says findings from data mining are upwardly biased and need specialized statistical tests[2].
A stricter version is a true out-of-sample test: wait until enough new data has arrived after a published study, then test the same rules. The cost is the wait[1]. Fang, Jacobsen and Qin did this for simple technical rules that Brock, Lakonishok and LeBaron had found to work best in their sample. For the period 1987 to 2011 they found no evidence that the rules predicted stock markets[2].
Example
Say you have ten years of daily closes for a currency pair.
- You test 400 moving-average pairs on the first seven years, the in-sample period. The best pair earns 12% a year there.
- You then run that one pair, unchanged, on the last three years, the out-of-sample period. It earns 1% a year.
The first number shows how well the rule was fitted. The second is the first evidence of whether it holds elsewhere. Had the pair earned 11% on the last three years, it would have passed one test. It would still be one sample, and one test.
Watch out
Out-of-sample data gets used up. If you look at the result and go back to adjust the rule, that data is part of the selection. Neely and Weller note that it is almost impossible to avoid some data snooping, because data sets and plausible rules are limited[1].
Testing reduces the danger of data mining and data snooping. It does not remove it[1].
A failed test does not say why it failed. Sullivan, Timmermann and White noted that a rule may have worked in the past and markets may since have become more efficient[2]. Neely and Weller add that a rule’s profitability can vary over time[1].
Sources
- Central bankFederal Reserve Bank of St. LouisTechnical Analysis in the Foreign Exchange Market (Working Paper 2011-001B, Neely & Weller)Checked November 9, 2026
- Professional bodyCFA Institute Research FoundationTechnical Analysis: Modern Perspectives (Literature Review; Scott, Carr & Cremonie, 2016)Checked November 9, 2026
