Frey-Osborne and the Replaceability Index: Compare With Care
By Shawn Geeman · 2 min read · Published · Updated
Frey and Osborne asked a consequential question about the susceptibility of occupations to computerisation. Comparing that work with a newer exposure index can be useful, but disagreement between two estimates is not evidence that either has predicted the future correctly.
The frameworks ask different questions
The original Frey-Osborne study uses a model of occupational susceptibility to computerisation. The Replaceability Index uses its own relative exposure scale, informed by occupational context and later AI-capability research.
A probability from one study and an index value from another are not interchangeable units. Placing both on a 0 to 100 graphic does not remove the difference in what they mean.
Matching occupations comes before calculating agreement
A careful comparison specifies classification versions, code crosswalks, duplicate treatment, and the rules for including or excluding a match. Results can change when broad occupations are compared with detailed ones or when codes change across vintages.
The Index previously published a Spearman correlation of 0.667 across 555 matched occupations. That is a historical claim about agreement between two model outputs. This article does not provide the matched records and analysis needed to independently reproduce it, so it should not be treated as verified predictive performance.
A disagreement can become a research question
If a newer model assigns lower exposure to physical work and higher exposure to language-heavy work, ask which capability assumptions explain the difference. Then seek evidence about actual task performance, adoption, and outcomes.
Do not describe that difference as proof that an earlier study overestimated or underestimated real job losses. Establishing an error against outcomes would require outcome data and a defined evaluation period.
Thresholds can manufacture a dramatic story
Calling a role a “flip” requires choosing a threshold in each framework. Small numerical changes around a boundary can create a category change even when the underlying estimates remain close.
A reported share of category flips is not automatically the share of a policy program that is misallocated. Decisions about training and regional employment need more evidence than model disagreement.
What a responsible comparison would publish
Publish both source versions, matched rows, transformations, thresholds, calculations, and uncertainty or sensitivity analysis. State whether the goal is rank agreement, task exposure, or prediction of observed employment changes.
The original paper is linked from our methodology. The public Index dataset lets readers inspect our published occupation values. Neither should be used alone to decide an individual career.
Frequently asked questions
Does a correlation of 0.667 prove the Index is accurate?
No. It describes a claimed degree of rank agreement with another model, not accuracy against real-world outcomes. Independent reproduction also requires the matching and analysis artifacts.
Can I compare the two score values directly?
Only with clear explanations of their different scales and assumptions. A relative index value is not a calibrated probability.
Sources and methodology
The score is a relative index of task exposure from 0 to 100. It is not a percentage chance of losing a job or a forecast of layoffs. Individual tasks, employer choices, adoption, and regulation all matter.
The Index uses stored BLS-based occupation data. O*NET, OECD capability indicators, and task-exposure research are contextual references. The derivation of the stored occupation scores is not reproducibly documented in the published dataset.
The source snapshot does not record the wage-survey year or employment-projection window. Check the original BLS release before treating these figures as current. Editorial changes and export dates do not establish data freshness.
This essay reflects the evidence available when written; future outcomes remain uncertain.
Discover your free score · Explore the $5 Pro Report · All essays