AI-Generated Junk Science Research a Growing Problem, Experts Say

A surge of artificial intelligence-generated fake research papers is permeating academic search engines like Google Scholar, potentially eroding public trust in scientific findings and derailing product development across industries that rely on cutting-edge research.

A study from Harvard Kennedy School Misinformation Review uncovered an academic research trend, first reported by Newsweek. The researchers identified 139 papers suspected of being generated by AI tools, with more than half focused on topics including health, environmental issues and computing technology.

“Large language models (LLMs) generate results based on a probability skewed to the data on which the foundation model has been trained,” Sid Rao, CEO and co-founder of AI company Positron Networks, told PYMNTS. “This can result in biases in the text that have no relation to the scientific method used to conceive the paper, as the foundation model is not required to follow a rigorous, fact-based process.”

“[T]he public release of ChatGPT in 2022, together with the way Google Scholar works, has increased the likelihood of lay people (e.g., media, politicians, patients, students) coming across questionable (or even entirely GPT-fabricated) papers and other problematic research findings,” wrote the paper’s authors.

This flood of fabricated studies poses risks to companies investing in research and development. It could lead to misguided product launches and wasted resources. It also threatens to undermine public trust in science and the reliability of evidence-based decision-making.

Eroding Trust and R&D Risks

The consequences of this trend could be far-reaching, affecting not just academic circles but also consumer trust in scientific claims.

“Fake research is a cancer to consumer trust,” Andy Stapleton, an AI education YouTuber with over 250,000 subscribers, told PYMNTS. “Once people realize that the ‘science-backed’ label can be bought or fabricated, they’ll start treating real research like snake oil. It’s a one-way ticket to a world where facts are optional and trust in legitimate innovation takes a nosedive. Consumers will stop believing any company that claims to have science on their side.”

Rao said AI hallucinations produce inaccurate results and subtly generate erroneous content. For example, a paper could present the correct conclusion but still have unreferenced or subjective supporting statements.

“Even at a 1% error or hallucination rate, these two problems would fundamentally erode trust in scientific research,” Rao said. “We have already seen this behavior in psychiatric telemedicine chatbots that have accidentally told patients to harm themselves.”

The implications for research and development investments are significant.

“AI-generated papers are a huge liability,” Stapleton explained. “If investors can’t tell what’s real and what’s algorithmic fluff, they’ll start pulling back. R&D is already risky enough — adding a layer of uncertainty from questionable AI-driven publications makes it even worse. You’re not just losing credibility; you’re bleeding money because bad data leads to bad decisions.”

Real-World Consequences

The impact of fake papers on business regulations could also be severe.

“Unreliable studies muddy the waters for regulators,” Stapleton said. “If the science behind a product is shaky, lawmakers will either clamp down with over-regulation to protect consumers or worse, they’ll make bad policies based on false data. Either way, businesses get stuck in a mess of red tape and uncertainty. The bottom line? Bad studies lead to bad laws, which is a death sentence for innovation.”

Rao warned that regulators might respond with overly broad restrictions, potentially banning AI use in medical research altogether, despite the technology’s applications in areas like forecasting and data analysis.

“Worse yet, in critical environments such as medicine, healthcare, civil engineering or material sciences, faulty papers’ negative real-world and material consequences will potentially shut down legitimate avenues of scientific research,” he added.

The study’s lead author, Jutta Haider, told Newsweek that not only did the study find that many of the fake papers were easy to spot because they contained obvious phrases common to ChatGPT but that “a few of those came from quite well-established journals that [serve as a guide for researchers to] check if a journal has proper safeguards and peer-review.”

Despite these challenges, Rao and Stapleton told PYMNTS that they see a role for AI in scientific research.

“AI can be a valuable assistant, but ultimately the scientist must personally own and carefully review the work, the content of the paper, and the fundamental results,” Rao said. “Just like a scientist may use Grammarly today to review content for grammatical accuracy, generative AI can find content, generate potential hypotheses and experiments, or create scientific computing code to execute and simulate the experiments.”

However, he cautioned: “For more foundational advancements — where the model independently creates scientific content — the model must be highly specialized and trained in the scientist’s research domain, carefully earning trust as a valued independent researcher in the scientist’s team.”

For all PYMNTS AI coverage, subscribe to the daily AI Newsletter.

The post AI-Generated Junk Science Research a Growing Problem, Experts Say appeared first on PYMNTS.com.