How (not) to measure progress in science?

I’m a bit late to the party but I’ve been enjoying some Collison podcast backlog and realized I had more to say about the “diminishing returns of science” trope that does the rounds from time to time.

Simply stated, the thesis suggests that a variety of metrics employed to measure progress of science all seemingly concur that despite increasing numbers of PhDs and the net accumulation of knowledge, major new discoveries are few and far between, at least compared to science in prior ages.

For a process that’s devoted to discovering knowledge, science is poorly understood by nearly everyone, including scientists. It may not be surprising to find that GAAP metrics don’t neatly translate into an industry that has resolutely resisted market forces since the beginning of time, but there are less glib reasons why measuring progress in science is actually highly non-trivial.

I’m going to go into more detail later in this post, but the fundamental issue is that science is like Tetris, in that it’s both cumulative and self-compressing. Unlike, say, philosophy, the measure of a field’s maturity is the brevity of its textbooks and the unity of its underlying theoretical framework. With the benefit of hindsight, the discoveries of a century ago appear neatly ordered with the most (contingently) significant results retaining salience while the rest have slipped away.

Where do we start?

I first became aware of this idea being in the startup world around 2014, when Peter Thiel was giving talks such as this one.

I found that Thiel had somewhat limited insight here, though I recognize that a scientific background isn’t necessarily a requirement to contribute to this question. Still, comparing rate of financial return between venture capital and fundamental research is a bit gauche, particularly if one happens to be a VC. Of course, such comparisons will be made but it’s germane to begin with the requisite 45 minutes of throat clearing about Keynesian capital overabundance and selection bias.

That is, we don’t typically get lectures from failed (unlucky) VCs about the genius of the market. My rule of thumb is that if 5 sigma is good enough for particle physics, it’s good enough for VC as well. Anyone can get lucky once. Get lucky 5 times in a row and that’s more interesting.

I was more interested when Patrick Collison and Michael Neilson entered the arena in 2018, with an article in the Atlantic asking if science was stagnant.

I’ve met both men socially a couple of times and must begin by stating I have nothing but the highest respect for them personally, and their intentions in this endeavor. I also think that their suggested program of attempting more detailed study of this area is a damn good idea.

However, getting traction is difficult if the foundations are awry and there are aspects of the article that need further attention. Collison has collated other responses here so it’s gratifying that there is an ongoing conversation in this area. Certainly, science coming to a grinding halt is the stuff of civilizational nightmares.

Much of what I’ll write here may be obvious to some readers, perhaps less so to others. It seemed less obvious to me despite being familiar with these ideas for years and working in science for a decade or so, so I’m writing them down.

In summary, the article leans heavily on statistics and surveys about Nobel Prize-winning discoveries to make the case that despite exponentially increasing PhDs, publications, and science funding, few major discoveries have been made very recently.

At the risk of appearing a bit snarky, I would ask a hypothetical question: Over time Stripe has hired exponentially greater numbers of talented software engineers and yet has continued to ship about one product a year. Why?

[Edit: Since I wrote this, the pace of products at Stripe seems to have picked up a lot. Be careful what you wish for, I guess.]

To be fair, I could ask this of any software startup. The answer, of course, is that it’s complicated. Larger organizations move more slowly. Incremental growth in a market suffers diminishing returns. More ambitious products have more onerous compliance requirements. Competition. Technical debt.

To get a little deeper, like many ambitious companies Stripe has teams working on fairly fundamental research questions in cryptography and computer science. These team members are among the smartest, best resourced humans to have ever lived. How long until we get a Stripe publication that’s as significant as the Church-Turing Thesis?

This is a silly question, intended to provoke more than illuminate. But I think it underscores that progress in science is not purely an organizational problem. Academia is undoubtedly riven with dozens of major inefficiencies, some old and some new. I even wrote a whole book about ways in which assimilating this way of life can challenge humans. See the chapter on leaving academia for more information along these lines. The point is that even private research outside academia, despite enormous leaps in capability, doesn’t necessarily see itself as being in the Nobel Prize game.

And so, with that primer, we turn to the Nobel Prize. Awarded once a year in various fields for outstanding research, the prize itself has numerous well documented limitations. Using it as a mechanism to measure progress in science, no matter how well intentioned, is unlikely to result in deep insight. Doing so rests on a variety of flawed assumptions, chief among them being that science progress is conventionally measurable and the Nobel Prize performs that sort of measurement. Unfortunately, this is fairly far from the truth.

Before we tackle the salience of historical scientific research in general, it’s worth enumerating known biases in the Nobel Prize alone:
– Cadence of once a year, to at most three living participants, doesn’t scale with increased number of scientists, increased lifespans, or scale of scientific research.
– Nobel Committee is notoriously, and increasingly, conservative, reliably shunning certain demographics.
– Nobel Committee strongly prefers established results, meaning that in general old scientists get prizes for work they did when much younger, often work that may not have seemed that significant at the time.
– Scientists who die, or leave the field, or leave science, never get prizes.
– Women hardly ever win, particularly in physics.
– Standards and practices on the Committee have changed over time.
– Standards and practices of science outside the Committee have changed over time.

Earlier in the article, however, Collison and Nielson talk about a survey which asked scientists to evaluate which of a pair of given Nobel Prize discoveries were more significant. This approach ameliorates some of the peculiarities of the Prize system, however as designed cannot give unequivocal evidence. Science is progressive, and later results build on earlier ones. The significance, and salience, of earlier discoveries is enhanced, or overshadowed, by later ones in the same area.

Judging the relative merit of the discovery of the neutron or the Higgs Boson is a pointless exercise. The Higgs is the final page of a story that began with the neutron, a story that involves the contributions of at least tens of thousands of scientists, most unknown even to their own close families. If the Higgs had been discovered before the neutron, it would be more significant by far! So in some ways the question as posed asks “which of these discoveries was made first?” Are we so surprised to find that answers to this question, averaged and graphed, are biased to the left?

Collison and Nielson’s article follows this by talking about the lack of Earth-shaking discoveries, such as Einstein’s final formulation of General Relativity in 1915. Today that seems like as good a date as any to bake a cake with equations on it, but the reality for working scientists is that the ways in which GR “radically changed our understanding of space, time, mass, energy, and gravity” began, in many ways, with Maxwell in the 1870s and continue to the present day. There were several other prominent mathematicians (Hilbert among them) also working on similar geometric formulations of gravity, while GR was not widely accepted in physics for decades. Many of the more interesting cosmological consequences were not appreciated until the universe was found to be expanding and the cosmic microwave background (CMB) discovered, and details are still being actively researched today. I worked in this field for five years and I am 100% certain that by 2050, 90% of what I learned will be utterly irrelevant, I just have no way of knowing what.

The article discusses (and largely rejects) the idea that science is reaching a point of diminishing returns because all the easy stuff has been found and we’re approaching a more-or-less complete knowledge of nature. The death of physics has been predicted in the past, just prior to the discovery of quantum mechanics. It’s true that subfields of physics wax and wane depending on funding priorities and the somewhat stochastic fine-grained nature of discovery. Nuclear physics has run out of superpowers and high energy physics has certainly run out of big accelerators for the time being. But grad students and postdocs are highly fungible and may be counted on to reliably find the next big thing. Just because we haven’t yet digested the significance of the body of knowledge produced by our own generation doesn’t mean that it’s intrinsically worthless.

The article closes with a brief discussion of the idea of productivity slowdown. Economists measuring nation-state level productivity find that gains in per-person productivity in the US and other developed nations have largely tailed off since their peaks between post WW2 1950s-1970s. Obviously at this scale economic behavior is multifactorial (to say the least) but the specific mention of the Concord as a false harbinger does highlight the omission of the single most glaring factor in economic changes in the last quarter of the 20th century: the loss of predictably cheap oil. If I’m right, exploding capacity and plunging electricity costs currently occurring due to developments in photovoltaics and batteries will reverse this trend and enable supersonic air travel. We’ll see before we’re old!

Finally, let’s talk about the single most troubling aspect of the science progress measurement problem. Science is axiomatically different from nearly every other human pursuit, in that it’s cumulative and self-compressing. Scientists often joke that if they’re lucky they’ll get a Nature paper in a career – a single really solid bit of research that, years later, will be half a paragraph or a footnote in a textbook. Scientific progress often isn’t measured in pages of text produced, but pages of text removed. A great insight will allow two previously disparate phenomena to be understood under the same concept, and thus, the knowledge is compressed.

So when we attempt to measure the salience of the net contribution of an individual historical scientist today, it’s very difficult to propagate that distribution forwards or backwards in time, or to analyse it in isolation to other contemporary work. This is not simply a matter of bunging a bunch of weights into the Perron-Frobenius model and redoing PageRank. Everything is contingent.

I will, however, suggest a useful mental model. Imagine that the contribution of individual scientists can be modeled with a power law distribution. Landau actually tried to do something like this, for real.

Over time the rankings will shift and the absolute value will rise and fall but, generally speaking, salience falls with time, and often for reasons beyond any individual’s control, and often for reasons unrelated to the intrinsic quality or utility of that person’s work. Within someone’s career, if their salience is consistently high and the gods are favorable, the Swedish Committee may bestow The Prize but in many ways this is an inherently (and inconsistently) biased sample of an already biased probability distribution.

This all seems a bit handwavey, so I’ll give a concrete example.

I studied physics as an undergrad in building A28 at the University of Sydney. Built in 1920, its striking facade is decorated with the debossed names of famous physicists of the time. Feynman is not listed. Indeed, 1920 predates almost all of quantum mechanics and subatomic physics. It is a fun exercise to read “Anathem” by Neal Stephenson and map all the renamed physics in there to relatively obscure physics arcana in our own universe.

As someone who has read a couple of undergraduate physics textbooks, I could associate each of the names with an effect or equation, but I would be surprised if non-physicists knew most of the names, which are: Archimedes, Roger Bacon, Copernicus, Kepler, Galileo, Newton, Huyghens, Dalton, Fresnel, Fourier, Carnot, Faraday, Maxwell, Helmholtz, Kelvin, Boltzmann, Roentgen, and Bessel. [Edit: the foyer, completed a few years later, contains the names of Lorentz, Einstein, Planck, Aston, Hertz, Laue, Bohr, Rutherford, Thomson, Townsend.] I challenge the interested reader to write a paragraph, from memory, on the key contributions of each of these people.

For the purposes of this blog, I decided to research who did what and when, and it turns out that most of these physicists made their discoveries between 1800 and 1850, which is to say, 70-120 years before the building was built. A more comprehensive list of physicists active during this period can be found here. Attentive readers will have noticed that Collison and Nielson’s primary thesis is that exciting discoveries in physics dried up by 1950, 70 years ago. Coincidence, or perhaps low confidence in one’s ability to predict the long term value of recent discoveries isn’t a new phenomenon?

To take just one example, consider Thomas Young, who is best known today for the classic double slit experiment. A renowned polymath “who made notable contributions to the fields of vision, light, solid mechanics, energy, physiology, language, musical harmony, and Egyptology.” Widely considered to be the smartest working scientist of his generation, though apparently not good enough for the facade of A28. Best known today for an experiment that can be repeated by a four year old with a cheap laser. And yet in his own lifetime, despite considerable talents, he struggled as well as anyone else to wrest knowledge from the unordered chaos of the universe.

Science is a largely artisanal endeavor whose discoveries are always made by a huge number of people working in parallel. Tiny pieces of the puzzle are worked out by people who, in many cases, remain unaware of the other’s existence. The lucky few get an obscure equation named after them. Measuring the rate of equation naming is not a good way to understand the progress of science!

So how might we go about measuring the progress of science?

To take a utilitarian perspective, I think it’s fairly widely agreed that the human condition, both individually and collectively, has improved markedly over the last century. Amongst many others, Human Progress has collated impressive datasets showing rapid, and accelerating, improvement in key indicators such as hunger, poverty, literacy, freedom, life expectancy, exposure to violence, and access to markets. For sure, much of the improvement can be attributed to wider implementation of existing technology, and in some cases rather antique technology at that.

Food scarcity was largely (and unexpectedly) solved in the 20th century with the invention of the Haber process (1913) and Berlaug’s dwarf wheat (1950s), which were widely deployed within decades.

There is no doubt in my mind, however, that on a per-minute or per-smile basis, the material resources my contemporaries enjoy are overwhelmingly the result of new inventions, which is to say, new applications of relatively recently discovered science. The most transformative of these are personal computers and the internet, but I am convinced that we’re not even half way through chapter one of that story.

Manufacturing and automation are also salient examples. One could argue that there’s no reason that, for example, Tesla cars couldn’t have been built on a Ford production line in 1920 (perhaps without the autopilot and computer screen) but that would require overlooking the vast foundation of incremental knowledge gains necessary to make something as banal (and alien!) as a lithium battery cell for only $1.50 – cheaper than a loaf of bread.

A more thorough accounting premised on applied utility will show, I believe, accelerating scientific knowledge generation, diffusion, and application for the improvement of the human condition in every corner of the globe.