Subscribe to Pittwire Today
Get the most interesting and important stories from the University of Pittsburgh.How quickly could you find a decade-and-a-half-old file on your computer?
When Sergey Frolov’s collaborator asked to see 15-year-old data, Frolov didn’t hesitate. In less than 20 minutes he’d found the file and sent the raw data to his colleague.
Frolov, professor in the Kenneth P. Dietrich School of Arts and Sciences’ Department of Physics, tells his students to be ready to do the same. “They save all their data, and it’s backed up and shared,” he said.
It’s an approach that, in a new Science paper, he and his collaborators from the University of Minnesota and Institut Neel in France, advocate for everyone in their field of condensed matter physics to follow.
The replication crisis
In science, replication is key. If an experiment claims to have uncovered a new finding, then that finding should hold for anyone who repeats the experiment in similar conditions.
But that isn’t always the case.
An experiment that can’t be replicated is a sign that its results don’t reflect something universal. The consequences of scientific journals legitimizing such results range widely: They could impede a young scientist’s career, affect a company’s stock price or slow scientific advancement. In the case of medicine, the consequences can be deadly.
Recent issues in biology — and even mathematics, of all things — point to the scope of this replication crisis. Incentives to rush ahead with what seems to be an amazing finding are the same across disciplines: front page headlines, additional funding, a Nobel Prize.
“Some theoretical predictions can bias you as a scientist,” Frolov explained. “You see a pattern that you’re looking for, you can even convince yourself, ‘This must be it.’”
He has narrowed in on being cautious of “smoking gun” data — the kind that all but shouts, “Here I am! Here’s what you were looking for!” — a graph with a near-perfect slope, or a peak in a measurement at just the right time.
“When it comes to making a scientific discovery, a smoking gun is a piece of data that contains the full proof of the phenomenon,” Frolov said. “It’s a single figure that tells the entire story.” He’s found, however, that such figures can be deceiving.
Replicating failure
Frolov’s research focuses on new states of matter that could jumpstart quantum computing. Technology companies are interested, too, and in 2018 researchers working for Microsoft announced that they had found one of these new states of matter: a Majorana fermion.
It was a classic smoking gun. “You look at the paper, and you say, ‘Well, what else could it be?’ It was such a striking, dramatic pattern,” Frolov said. But he was doing similar research and had not seen such results, so he asked to see data that wasn’t included in the published paper.
Using that data, Frolov and others in the field spotted problems such as non-representative data selection and data manipulation. The Microsoft authors retracted their paper.
It was an indication that Frolov’s field faced the same issues as others, and he wanted his physics colleagues to take notice. He decided to tackle the problem where it was most visible, journals where he’d seen claims of smoking-gun results. He and his graduate students Yifan Jiang, Bomin Zhang and Seth Byard along with postdoc Po Zhang tried to replicate four such experiments.
Each time, they found alternative explanations for seemingly extraordinary results. When they submitted some of their findings to the same journals that had published the original papers, however, they were rejected. Editors said replication studies weren’t novel, or that they were longer relevant since it had been years since the original papers were published.
They tried again. “I thought, let’s make a larger point,” Frolov said. This time the team bundled four experiments that seemed to produce smoking-gun results if they were not considered in the broader context of a full dataset.
The paper was published on Jan. 8 in Science, two years after the team’s initial submission.
Frolov and co-authors suggested changes to the standard publication process — that researchers include more information about alternative explanations for their findings and share as much experimental data as possible when a paper is published. With more perspectives, a signal that seems unique may fade into mundanity.
Several fields have already made changes toward this end, Frolov said. “My colleagues in astrophysics, not everyone, but to them it’s like, ‘of course you share data.’ It’s a lot more accepted.”
Another opportunity lies with journals. Their editors decide what to publish and what to reject, but readers don’t have a window into the editors’ concerns about a given paper. Providing readers with more information on the process could let them decide for themselves if additional data would provide better insight.
Frolov is keenly aware he’s suggesting systemic change to an institution with hundreds of years of momentum. In 2024, he and researchers from several universities held a conference at Pitt, along with producing a conference report and a 35-minute video, to build support for improving reproducibility in the field.
“I’d be a fool if I expected a sudden change,” Frolov said. “There will be talking, then some changes, then there will be some backlash because changes cause irritation. But then there will be some adaptation, and there will be more talking.”
And, he hopes, it will all lead to a more open and honest system.
Photography by Aimee Obidzinski