Testing 700 strategy versions can turn a clean holdout into another tuning set

Seeing a backtest built from hundreds of strategy versions raises one question before I care about the return: how many times did the final period influence what survived? A holdout stays clean until its result changes the next decision. If version one fails on the final two years and you adjust a filter, position sizing, session, or market selection before running version two, those two years have joined the development process. It is like letting 700 students take the same final exam and then displaying the best score. The questions may have been hidden at the start, but repeated exposure taught the class what the exam rewards. **A cleaner process:** 1. Freeze the strategy family and the selection criteria before opening the final window. 2. Use development and validation data for iteration, and log every tested version, including the rejects. 3. Promote one candidate from each family into the final period once. 4. Report the number of attempts and the spread of results, not only the winner. If the final test fails, leave it failed and use future data as the next holdout. Real costs, delisted companies and next-candle fills fix implementation problems. They cannot fix selection bias. You can build a realistic simulation and still select noise honestly. The honest version of a 700-variant backtest includes the rejected versions and preserves one final window the research process has never seen.