Examine which information and constituents a historical test could really have used.
Read Macro SignalsIntermediate4 min
Keep information dates
Plain English
A historical calculation is not a historical decision
A backtest applies a rule to past data and calculates what would have happened under specified assumptions. It can help inspect a model’s mechanics, but the calculation is hypothetical. The researcher already knows how history unfolded and must reconstruct what information, securities and trading conditions were available at each simulated decision time.
That reconstruction can fail even when the return formula is correct. A database of companies that exist today may omit companies that disappeared. A revised economic figure may replace the version originally released. A strategy can use a later publication date incorrectly, or be chosen because it happens to fit the same historical sample used to report its result.
These are different problems. Survivorship bias concerns which entities remain in the sample. Look-ahead bias concerns information arriving from the simulated future. Overfitting concerns choices tailored to particular historical observations. They can occur together, but identifying them separately makes the research assumptions easier to examine.
Worked Example
Removing a later failure changes the question
Consider four fictional constituents with returns of +20%, +10%, −10% and −80% over the same single period. Start with £100 in each, for £400 total. Assume no intermediate rebalancing, cash flows, fees or taxes. Equal starting monetary weights mean the portfolio return equals the simple average of these four same-period returns: −15%.
The closing values are £120, £110, £90 and £20, totalling £340. Now imagine that the last constituent was removed from a database after the period. A researcher using only the three survivors would start a retrospectively different portfolio with £300 and end with £320. Its reported return would be approximately +6.67%, rounded from 6.66667%.
The omitted loss has not been recovered. The researcher has changed the investable universe using later knowledge, as well as changing the opening denominator. The two returns differ by approximately 21.67 percentage points, but that gap is an illustration of sample selection, not a measured estimate of bias in real markets. These four returns are not successive annual portfolio returns.
Formula
The average same-period return is twenty percent plus ten percent minus ten percent minus eighty percent, divided by four, equal to minus fifteen percent
Input returns are percentage values; the constituents have equal initial monetary weights.
Initial value, all four constituents
£400
Closing value, all four constituents
£340
Full-sample return
−15.00%
Initial value, three survivors
£300
Closing value, three survivors
£320
Survivor-only return
6.67%
Research Method
Keep an information timeline
A point-in-time universe records the constituents eligible at the simulated date, including those later removed. Financial reports also need availability dates, not just the period they describe. A December accounting period may be reported months later. Using its final numbers for a December decision can introduce information that the simulated researcher did not possess.
Revisions and corporate actions need consistent handling too. An adjusted price series can be useful for calculating returns, but the adjustment method and available information should match the test. The research below examines retrospective benchmark membership and repeated model selection; the synthetic example is independently constructed rather than copied from an empirical result.
Publication time zones can matter for decisions near market opening.
Limits and Assumptions
A clean sample does not settle model choice or trading
Trying many rules and presenting only the strongest historical result creates a selection problem even without missing companies. The chosen rule may capture chance patterns in that sample. An untouched evaluation period can provide additional evidence, but repeatedly consulting it turns it into another selection tool. No single split or statistical measure guarantees that a model will remain useful.
Trading assumptions also matter. Commissions, spreads, market impact, borrowing constraints and execution timing can separate a model price from a feasible transaction. The example omits these costs to isolate the universe problem. A research report should state exclusions and failed or alternative specifications, rather than making a smooth chart stand in for a reproducible method.
Common Mistake
Treating a persuasive chart as validation
A strong-looking historical line cannot reveal when information became available or how many discarded rules preceded it. Ask for the dated universe, data versions, rules and cost assumptions behind the line. The purpose is to understand what was tested and what remains uncertain, not to turn historical fit into a promise about future returns.
Self-check
Check your understanding
Why is the four-constituent return −15%?
£400 becomes £340, a £60 loss. Equal initial weights also make the average of the four same-period returns −15%.
Why does excluding one constituent not erase its loss?
The survivor-only calculation describes a retrospectively different £300 starting portfolio, selected using information about later removal.
Does an untouched test period guarantee future reliability?
No. It can add evidence, but sampling uncertainty, changing conditions, costs and repeated consultation still limit interpretation.
Continue learning
Connect the ideas
Follow the related articles below to examine these assumptions in another setting.
Disclaimer
Educational Use Only
This article is for informational and educational purposes only. It does not provide personalised investment advice.
