New$10 Sign Up Bonus & $30 Free Trial Reward
Start challenge

Backtesting

Also called back testing, historical testing, strategy testing

Testing a trading rule on historical data to see how it would have performed. Results can overstate future performance, especially when many rules or settings are tried and the best one is kept.

Reading
2 min
Sources
5Central bankProfessional bodyBookReferenceResearch
Updated
On this page· 5 sections
  1. How it works
  2. Example
  3. Watch out
  4. Sources
  5. Related terms

It is relatively easy to make a trading rule look good on the data it was built from[5]. A backtest asks how a rule would have done in the past, and most of the craft lies in keeping that answer honest.

Neely and Weller note that technicians routinely backtest rules and keep only the ones that were profitable on past data[1].

How it works

A backtest is a historical simulation. It runs a fixed rule over past prices and computes the profits and losses the rule would have produced[5]. For a currency rule, researchers compute the signals day by day, then the returns, net of a transaction cost on every trade[1].

The data is split in two. The in-sample period is the data used to design the rule. The out-of-sample period is data the design never touched[5]. A backtest is realistic when the two give consistent results[5]. Researchers select rules on the first period and test them on the second, which reduces the danger of data mining[1].

One standard textbook gives system design and testing a part of its own[3]. ChartSchool’s glossary is blunter: backtesting is rarely done properly and usually ends up as curve fitting[4].

Example

Say you test a two-moving-average rule on five years of daily data. You try short averages from 5 to 50 days and long ones from 60 to 250 days, in steps of 5: 10 × 39 = 390 pairs.

  1. The best pair made 25% over the five years, before costs, with 80 round trips.
  2. Each round trip pays the cost twice. At 0.03% a side, costs take about 80 × 2 × 0.03% = 4.8%[1].
  3. Bailey and co-authors estimate that with five years of data, no more than 45 independent configurations should be tried. At that number, the best of a set of skill-less strategies is expected to show an in-sample Sharpe ratio of 1 and an out-of-sample Sharpe ratio of 0[5].

Neighboring pairs behave alike, so the 390 are not all independent, but they are still far more than 45. An out-of-sample check would pick the pair on the first four years and run it once on the fifth[1][5].

Watch out

The best result is biased upward. Testing every moving-average length from 1 to 10,000 days finds the one that was most profitable over the test. Because so many were tried, the result is not a reliable guide to what comes next[2]. Data snooping means choosing rules already known to have worked on the data; data mining means testing many and reporting the winners[1].

The number of trials matters. A backtest that does not report the number of trials makes the risk of overfitting impossible to judge[5].

Dropping the failures has the same effect. Bailey and co-authors note that fund groups that close poorly performing funds introduce survivorship and selection bias, which works like hiding failed experiments[5].

Good backtests can still fail later. Retested on stock markets from 1987 to 2011, the rules a well-known 1992 study found worked best showed no evidence of predicting the market[2].

All terms in Tronexopedia

Your challenge starts here

Start your challengeFree TrialNew
Registration bonus

Get $10 free in your Wallet when you sign up

Register and use your $10 Wallet credit to reduce the price of your first challenge.

The $10 bonus is credited to your personal Wallet and can only be used to reduce the price of a challenge purchase. Credits cannot be withdrawn or used for payouts.

Start now & get $10