BlogJanuary 15, 2025

What a viral study about 'ghost engineers' gets right and wrong

Marcos Conci
I came across a viral study on "ghost engineers" and could not stop reading. It started as light reading and turned into something closer to a detective story. The hook: 9.5% of software engineers do virtually nothing. Fifty thousand engineers, hundreds of companies, Stanford research. Numbers like that make executives spill their coffee. Then I looked at the methodology. The researchers used 70 code commits to build a model that evaluated 1.73 million commits. Imagine walking into the Library of Congress with its 39 million books and judging the entire collection from 70 pages. The sample ratio is 0.004%. Java only. Leave-one-out cross-validation, random forest with static analysis. My first instinct was that this could not possibly be reliable. But here is where it got interesting, and where I changed my mind. The reported numbers were: correlation of 0.82 for time estimation, ICC(2,k) of 0.81 for coding-time consistency across reviewers. Those are strong. Despite the sample size, the model's results were surprisingly stable. And the model was not counting commits. It analysed the actual source changes in each commit for complexity and other parameters, and was built to mimic evaluations made by ten independent human experts. That is closer to having a panel of seasoned reviewers assess every change than to any commit-counting metric. So what do I take from it? Two things that sit uncomfortably together. Small samples can produce reliable models when the thing being measured is consistent and the features are well-chosen — sample size alone is not the whole argument, and I was too quick to assume it was. But a model that reliably reproduces expert judgement on Java commits is not the same as a claim about what 9.5% of software engineers do with their working lives. The technical result and the headline are different claims, and only one of them was actually tested. In a world increasingly obsessed with metrics, what we measure — and how we measure it — matters. Even the most advanced models are only as insightful as the data we feed them, and only as honest as the conclusions we draw past them.
Share this post: