A backtest runs your rules on past prices to show what they would have done. It can’t tell you what will happen next, and it’s easy to make one look better than it should. Use it to reject bad ideas. A good result only earns an idea the next test.
Seven checks
1. Include fees and slippage. Slippage is the gap between the price you wanted and the price you got. A strategy that trades often can go from profit to loss once both are counted.
2. Compare it with holding. A bot that made 30% while the coin made 60% lost to doing nothing. Always put both numbers side by side.
3. Split your data. Build the rules on one period and test them on a later period the rules have never seen. If the result collapses on the second period, the rules were fitted to the first.
4. Count the trades. A handful of trades proves very little. One lucky trade can make the whole result.
5. Cover different markets. Include a rising market, a falling market and a sideways one. Many strategies are one market in disguise.
6. Read the worst moment. Maximum drawdown is the largest fall from a peak. Ask whether you would have left the bot running through it.
7. Check how the test fills orders. TradingView’s broker emulator can fill a limit order at exactly the requested price, or only once the price has gone one tick beyond it. The second setting is more cautious. Assuming every limit order fills at its price flatters the result.
The trap: testing many versions
Test 100 versions of a rule with no real edge, and at a 5% significance level you would expect about 5 of them to look good by chance. The one you keep is the luckiest, and it’s the one you fund.
Bailey, Borwein, López de Prado and Zhu wrote a paper on this problem. They argue that standard hold-out testing is unreliable for investment backtests, and they propose a way to estimate the probability that a backtest is overfit. The practical lesson: keep count of how many versions you tried, and be more suspicious the higher that number gets.
Signs of overfitting:
- The rule has many conditions or oddly exact numbers, such as an RSI of 37.
- Changing a setting slightly destroys the result.
- It works on one coin and fails on similar coins.
If you test in Pine Script
TradingView’s documentation notes that historical bars are evaluated once, at the close, so realtime results can differ from a backtest. It recommends leaving calc_on_every_tick off for backtesting. Built-in trailing stops and process_orders_on_close can also make results repaint. Backtest results can shift as the starting point of the data moves too.
After the backtest
Run the strategy on virtual money, and then with a small live amount. Live results rarely match paper results, and the size of the gap tells you how far to trust the test. In Tradenos, every test is shown next to holding the coin, including when the idea loses.