BullsEdges algo trading guide: Splitting your data into in-sample and out-of-sample periods is the simplest defence against overfitting. Here is how to do it properly.
If you test enough parameter combinations on the same data, one of them will look excellent purely by chance. The out-of-sample test exists to catch that.
BullsEdges algo trading application: Keep the final test period untouched during development; repeatedly tuning to it turns validation data into training data.
The idea
Divide your historical data into two parts:
- In-sample: the period you use to design the rules and choose parameters.
- Out-of-sample: a period you do not look at until the strategy is finished.
When the design is complete, you run it once on the out-of-sample data. If it performs roughly as well as in-sample, there is a good chance the edge is real. If it falls apart, the strategy was fitted to noise.
How to split
- Time-based split: for example, develop on the first four years and test on the last two. This also checks whether the edge survives a changing market.
- Half and half: run the final strategy on each half of the data separately. A real edge should show up in both halves, not just one.
- Multiple symbols: if a gold strategy is based on a general idea such as breakouts, check whether it also works, at least weakly, on another volatile instrument.
Rules that keep the test honest
- Look once. If you go back, change the rules and re-test on the out-of-sample data, it has become in-sample.
- Fix everything first: parameters, filters, costs, position sizing.
- Use the same costs in both periods - and realistic ones.
- Expect some decay. Out-of-sample results are usually somewhat worse. A drop from profit factor 1.5 to 1.3 is normal; a drop from 1.5 to 0.9 is a fail.
Red flags of curve fitting
- Many parameters relative to the number of trades.
- Results that depend on one exact setting: period 47 works, 45 and 50 do not.
- Every variant you try "passes". Real edges are rare; if everything works, suspect the testing code - for example lookahead bias in a higher-timeframe signal.
- A large share of the profit comes from a handful of trades.
Parameter plateaus
A robust strategy sits on a plateau: a whole region of nearby settings gives similar results. When choosing parameters, prefer the middle of a broad, good region over a single peak.
After the out-of-sample test
Passing is not the end. Run the strategy forward on a demo or small account and compare it trade by trade with what the backtest would have done over the same weeks. That is the last and most honest out-of-sample test there is.
Trading forex and gold carries a high level of risk and may not be suitable for every investor. Past performance and backtests do not guarantee future results. This article is education, not financial advice.