Survivorship bias can really wreck a backtest

Ok

Get out of here with your informative post. But seriously the list of things which can cause backtests to be misleading is very large

for real, every time you think youve accounted for everything theres another gotcha waiting

I feel people back test completely wrong. Results of backtest don't even really matter much. You can have a high PF or win% but doesn't mean shit. For example we developed one platform where the win% hovers around 36% but it loses little wins huge. I want to stop saying back testing because people build one bot and back test it then go paper then live. Dumb process. You need to develop one strong strategy then run a simulation of it against thousands of its clones vs two different 10 year stretches. Vs. different market condition and universes. Then you can see if it has edge or not. Maybe I'm tired and cranky so I'll see myself out

Good point - but I'd expand it a little bit. I've found one good strategy is not enough, a good collection of complimentary strategies is preferable in my experience.

Truth!

Of course it does. If you do it correctly. Why would you run thousands of clones? Do you want to make money or not? If you find something that works in a market there are reasons for it working and it will probably continue working if you did proper parameter stability tests and not cheating with OOS data. Its like monte-carlo simulations on the trade sequence, when in reality there is path dependency in real markets.

SurvivorSHIP bias? Shouldn’t they call it survivorPLANE bias? Stupid joke aside, I see traders make this mistake all the time.

“strategy i built for 2024+ markets doesn’t work in 1980 markets” in other news water is wet?

Surprisingly I’ve seen many traders make this mistake. I thought the video captured the mistake in a nice vivid way 🤷‍♂️

This sub is especially full of pricks. They're pissy when people get something wrong and pissy when they offer useful advice.

i’m just here to shitpost so yeah youre not far off bud

How far back would you say it needs to be profitable for?

depends on the timeframe you pick, the regimes you label & the regime you’re targeting. concrete example: war / crisis adjusted BRENT / AUX algorithm: pick a longer horizon timeframe, I like swings on 4h or 6h with 12h context, model training & research timeframe 2017-2026 but the actual strategy development 2024-2026 e.g the actual macro tape you’re trying to target.

That’s not exactly a good analogy. It’s not about the strategy but about the universe of stocks and looking into the future since if you just use the current SPX,NDX, DOW then you are automatically picking Apple, NVidia, etc before they were known to blow up. It is true that many Strats don’t work on the prior market - as it behaves differently due to all of the FED interventions. But survivorship bias is different than saying Strats today doesn’t work in 1980s markets. In fact many Strats from back then still work today (trend, momentum, mean reversion). There are always a top 5-10 momentum stocks that lift the market, it was oil in the 80s, internet in the 90s, and tech past the 2000 crash. Momentum strats and mean reversion Strats work but you have to use the stocks available at the time or you are looking into the future of only the current winners. Maybe you understand that, but it’s not what you wrote exactly.

If you're just testing on a ETF like SPY is that safer?

Yes I’d say that’s safe.

point in time constituents, always. today's list doesn't just add the winners, it also drops the names that went to zero or got bought out, so you need the delisted price history as much as the membership dates. Norgate is what most people here use for both.

Absolutely critical and has happened to me a few times where I forgot to filter my tickets by the year the year. My set up is fairly low tech on this front of just using the wayback machine to download the list of the tickers at the end of that year and use them for the forward year but if I anything I would argue it’s fairly conservative as it keeps losers that were dropped during the following year and hasn’t added the winners

I had a live strategy which performed great in back test but kept choosing losers over and over in live trading. Usually the live engine chose the same symbols day after day until they were delisted. Then I realized I didn't train the model with point in time constituents, meaning the model couldn't recognize what a death spiral looks like

Make sure your price data still has the names that got dropped too. A lot of free sources lose a ticker once it's delisted after a buyout or bankruptcy, so even with the right member list those trades just disappear from the test.

Great experiment! Did you hold the picks with equal weights or by their weight in the index? The Dow is price weighted, so if you compared against it, the benchmark side could be off too.