BlogJanuary 15, 2025What a viral study about 'ghost engineers' gets right and wrong
I came across a viral study on "ghost engineers" and could not stop reading. It started as
light reading and turned into something closer to a detective story.
The hook: 9.5% of software engineers do virtually nothing. Fifty thousand engineers,
hundreds of companies, Stanford research. Numbers like that make executives spill their
coffee.
Then I looked at the methodology.
The researchers used 70 code commits to build a model that evaluated 1.73 million
commits. Imagine walking into the Library of Congress with its 39 million books and
judging the entire collection from 70 pages. The sample ratio is 0.004%. Java only.
Leave-one-out cross-validation, random forest with static analysis.
My first instinct was that this could not possibly be reliable.
But here is where it got interesting, and where I changed my mind. The reported numbers
were: correlation of 0.82 for time estimation, ICC(2,k) of 0.81 for coding-time
consistency across reviewers. Those are strong. Despite the sample size, the model's
results were surprisingly stable.
And the model was not counting commits. It analysed the actual source changes in each
commit for complexity and other parameters, and was built to mimic evaluations made by
ten independent human experts. That is closer to having a panel of seasoned reviewers
assess every change than to any commit-counting metric.
So what do I take from it?
Two things that sit uncomfortably together. Small samples can produce reliable models
when the thing being measured is consistent and the features are well-chosen — sample
size alone is not the whole argument, and I was too quick to assume it was. But a model
that reliably reproduces expert judgement on Java commits is not the same as a claim
about what 9.5% of software engineers do with their working lives. The technical result
and the headline are different claims, and only one of them was actually tested.
In a world increasingly obsessed with metrics, what we measure — and how we measure it —
matters. Even the most advanced models are only as insightful as the data we feed them,
and only as honest as the conclusions we draw past them.