Three recent developments to note:
First, one of the most exciting events in 2022 is that the Office of Management and Budget (OMB) worked with ICPSR (a private research consortium at the University of Michigan) and 16 other federal statistical agencies to release ResearchDataGov.
This is a website where you can browse and request confidential data in a streamlined and unified way (as opposed to working with a bunch of laborious and bespoke processes at each agency). OMB also released a lengthy memo on how the standard application process works, and how it all arose from the requirements of the Evidence Act in 2018.
One might wonder: What does this mean for NIH, NSF, DOE, and any other agency that funds science?
Not much — yet. The new process applies only to confidential data kept by 16 specific “statistical agencies” as defined under federal law. That includes some well-known ones (such as the Census Bureau or the Bureau of Labor Statistics) and some less-famous ones (such as the USDA’s National Animal Health Monitoring System).
As yet, that doesn’t include data specifically about the internal workings of science funding agencies, whether it be NSF, NIH, DOD, DOE, or whatever.
Thus, there is a key opportunity here. Those agencies have typically claimed that data on unfunded applications, peer review scores, etc., are private and confidential. Only on rare occasions have they allowed external scholars access to that data so that they can study questions like, “How likely is this agency to fund high-impact research?”
Even if you think that those agencies are too overprotective and cautious (which is true!—nothing in the law justifies their current approach), the federal government has now rolled out a platform for making highly confidential data available to outside researchers through a streamlined process.
NIH and NSF should try to participate in the streamlined process as to their own internal data. This would show a robust commitment to the principles of the Evidence Act. In fact, the OMB memo above even suggests (on page 5) that “Other Executive Branch agencies or units” consider using the system to allow researchers to work with confidential data.
If NIH and NSF acted in the spirit of the Evidence Act by participating in the official government-wide process for confidential data, that would open up a wide range of possibilities for meta-scientists to study many open questions about how science is funded, how peer review works, and the like. It would be a huge win for society.
Second, the NIH released a plan to streamline its peer review process, which has been in the works for nearly three years now. We can all submit comments on this plan by March 10, 2023. See here.
For the new plan, reviewers will consider only three main categories:
- Importance of the Research
- Rigor and Feasibility
- Expertise and Resources
Only the first two criteria receive a numerical score (this seems fine and well). The new approach (if finalized) seems like an improvement on the status quo. It will be more efficient and more focused on the few criteria that really matter. As well, NIH intends to reduce the role of possible bias in favor of large, well-resourced universities:
Many institutions — such as minority-serving universities — produce strong research, says Noni Byrnes, the director of the CSR. But these data show that because of reputational bias, these institutions have not been on a level playing field in the review process, she adds. “And ultimately, what we want is the science to work.”
This all seems great, but there’s a missed opportunity here. As we said in our comments on the Center for Scientific Review’s draft strategic plan, NIH should take bold steps to experiment with new ideas as to peer review, such as limited lotteries, golden tickets, funding projects with bimodal scores, and more.
Point being, if the NIH is going to spend several years revamping peer review, it could do much more to address the complaint raised by innumerable scholars for decades now, including the eminent Sydney Brenner—peer review puts far too emphasis on preliminary data and guaranteed results, rather than being open to new ideas.
Indeed, one university VP of Research told me, “The NIH doesn’t fund research, it REfunds research.” In saying this, he was paying homage to the utterly routine observation that you need to have completed half or two-thirds of your research before you ask the NIH for to (pretend to) fund that work, whereupon you take the money and try to create preliminary data for the next grant. Seems like NIH could do something to cut back on this elaborate charade.
Third, the NIH’s Advisory Committee to the Director (ACD) has a Working Group on Re-Envisioning NIH Supported Postdoctoral Training. That is a noble effort, and one that is direly needed (a few recent headlines: “As professors struggle to recruit postdocs, calls for structural change in academia intensify,” and “48,000 University of California academic workers go on strike”).
A Working Group presentation is here. It mostly looks fine, but one page has the potential to be quite misleading in claiming that postdocs have had an “increasing stipend over time”:
This chart seems to suggest that post-docs have seen dramatic increases in pay since 2010. But the chart doesn’t adjust for inflation.
If you go to a basic inflation calculator, $50k in 2010 (the purple line) would be equal to $68,265 today. Thus, what looks like a dramatic increase on the chart is actually around a $3k cut in pay compared to 2010.
In any event, my recommendation to the Working Group would be this: Go back to the National Academies report from 1998 titled, “Trends in the Early Career of Life Scientists.” That report identified the same issues that we’re struggling with today — overproduction of trainees, inadequate pay, etc.
That report recommended:
- that we restrain the “rate of growth of the number of graduate students in the life sciences,”
- that “every life science department” should be “required to provide to its prospective graduate students specific information” about career prospects,
- that “all federal agencies that support life-science education and research” put their money in “training grants and individual graduate fellowships as preferable to research grants,”
- that we give “career-transition” grants to “senior postdoctoral fellows” to “give them financial independence,” and
- that universities try to dissuade people from pursuing PhD’s if a master’s degree would be “more appropriate, more efficient and less costly” for a career outside of academia.
Could have been written today, eh? But did NIH adopt and implement the National Academies’ recommendations? Basically no, according to a 2012 report in Science. Indeed, another Advisory Committee to the Director working group at that time (headed by Shirley Tilghman) made many of the same recommendations, but they seem to have ended up as minor tweaks or “encouragement” (rather than requirements).
The current working group could make progress just by going back to decades of previous reports on the postdoc/trainee issue, and actually implementing the recommendations with mandates — not just rules amounting to “our thoughts and hopes are with you.”
No need to reinvent the wheel, especially if no one bothered to use the previous wheel.