Alternative data: the backtest is the easy part

Harvey, Liu and Zhu counted hundreds of published return factors and argued that, given how many have been tried, a new factor should clear a t-statistic of about 3.0 rather than the conventional 2.0 ^[1]. A team testing dozens of transformations of one dataset faces the same problem in miniature.

Bailey and López de Prado's deflated Sharpe ratio adjusts a backtest's Sharpe for the number of trials and the non-normality of returns ^[2]. López de Prado's list of reasons machine-learning funds fail starts with the same point: research run as a search over backtests finds noise ^[3].

• Point-in-time data: the history as it was known on each date, without later revisions or backfilled panel members.
• Coverage stability: the panel's size and make-up over time, so a trend is not just panel growth.
• An economic mechanism: why this data should lead reported results, and by how long.
• Out-of-sample proof: performance on a period and universe the research never touched.

In 2021 the SEC brought its first securities fraud case against an alternative data provider. App Annie had told customers its estimates came from aggregated, anonymised data, but from late 2014 to mid-2018 it used non-aggregated, non-anonymised data to adjust them; the company and its co-founder paid more than $10 million ^[4]. For buyers, the lesson is that vendor representations are part of your own compliance file.

Personal data in a dataset also brings privacy law into scope, such as the GDPR in Europe ^[5], even when the vendor says it is anonymised.

• How is the data collected, and what did the people or companies it describes agree to?
• Could any field be material non-public information, and what controls stop it reaching you?
• Is the history point-in-time, and how were past revisions handled?
• Which other funds buy it, and how crowded is the signal likely to be?

Found this useful?

Share the key finding on X.