Backtesting Is an Argument About the Past

A backtest is evidence about a historical simulation, not proof that a strategy will make money. This distinction keeps financial ML honest. Historical prices contain survivorship bias, changing constituents, revised data, and information that may not have been available when a decision would actually have been made.

The first defence is a walk-forward design: train only on data available before the evaluation period, then advance the window. The second is friction: include spreads, fees, market impact, execution delay, borrow constraints, and turnover. A strategy that survives only before those assumptions are added is not yet a strategy.

Reinforcement learning makes the issue more visible because it can exploit weaknesses in a simulated environment. FinRL frames trading as an environment with states, actions, rewards, and constraints; those components should be designed to reflect the decision process rather than to reward a convenient proxy (Liu et al.).

I would report a distribution of outcomes across regimes rather than a single headline return. Drawdowns, capacity, concentration, and sensitivity to small parameter changes tell a more credible story. The output should be a decision memo: what was tested, what could invalidate it, and what limits will apply if it moves into a paper-trading phase.

The strongest backtest documentation exposes the assumptions that would most change the result. I would state the price source, asset universe, corporate-action handling, rebalancing calendar, execution convention, and all costs. That record allows another researcher to reproduce the simulation and tells an investment committee what deserves sensitivity analysis.

Regime analysis matters because markets are not identically distributed samples. A strategy should be inspected across quiet and volatile periods, different liquidity conditions, and changing rate environments. If its thesis depends on one market structure persisting unchanged, the operational response should be smaller exposure and earlier review—not a more confident chart.