Get your free personalized podcast brief

We scan new podcasts and send you the top 5 insights daily.

Tobias Carlisle used AI to run over 900,000 portfolio backtests across 1,200 asset parameters. When splitting the data in half, top-performing mixes from the first period consistently dropped to the bottom tiers in the second period, showing that over-optimization fails out of sample.

Related Insights

Tobias Carlisle and Jake Taylor note that stock and bond non-correlation was a brief 20-year anomaly rather than a permanent rule. The 2022 bear market crushed 60-40 strategies because both asset classes declined simultaneously, exposing how investors overfit rules of thumb to small sample sizes.

Mathematical models like the Kelly Criterion are only as good as their inputs. Historical data, such as a stock market's return, isn't a fixed 'true' value but rather one random outcome from a distribution of possibilities. Using this single data point as a precise input leads to overconfidence and overallocation of capital.

Agentic AI can explore a vast hypothesis space, massively increasing the risk of overfitting. This requires scientists to be more rigorous than ever, using techniques like hidden holdout datasets to avoid being fooled by the tool's power, which can "slice your fingers off."

When evaluating a backtest, investors should distrust any model that shows impressive returns without also revealing why the strategy is incredibly difficult to implement. A believable backtest must demonstrate the associated pain, such as long periods of underperformance or high career risk, which explains why the potential for future returns exists.

Startups are learning that spending significant time and money fine-tuning a specific open-weight model is a bad strategy. The rapid pace of AI progress means a new, superior model will be released within weeks, rendering the fine-tuning effort obsolete and a waste of precious engineering resources.

Rob Arnott warns that most impressive backtests fail because they are "data-mined"—designed to fit historical data. His firm uses the scientific method: form a logical hypothesis first, then use data only for testing. This approach creates more robust strategies that are less likely to falter when market conditions change.

Rob Ferguson says the state of the art is never one model; models have affinities such as JavaScript versus Python. In 'oracle modeling,' teams run every permutation of tasks and find a mixture wins, so the harness and UI context matter too. That makes measurement across combinations essential.

As AIs optimize investment strategies, they may inadvertently make the overall market system more fragile and susceptible to rare, high-impact "fat tail" events. This could create a niche for patient human investors who can identify and exploit these moments of system-wide fragility, similar to the collapse of Long-Term Capital Management.

To combat unreliable backtests, CFM is building "meta-models" that quantitatively predict whether a new model's results are overfitted. This systematic approach aims to replace human judgment with a data-driven process for deciding if a trading model is robust enough for production.

Don't repeat the industry's mistakes. An AI model can act as a collective memory, learning from every historical success and failure. By causally linking the early signals in a current program to the known outcomes of past trials, leaders can make go/no-go portfolio decisions based on data, not just intuition.

Optimized backtests fail to persist across out-of-sample data | RiffOn