LLMs Struggle to Reliably Judge Novelty in Scientific Research Ideas
This summary and analysis were generated by AI from the original article at Hugging Face Blog and may contain errors (how Viqus works). Read the source for full details.
7
What is the Viqus Verdict?
We evaluate each news story based on its real impact versus its media hype to offer a clear and objective perspective.
AI Analysis:
The hype around AI's ability to automate scientific discovery is outpacing the reality, as this paper proves the core mechanism—novelty judgment—is deeply flawed.
Article Summary
The study introduces 'Old Ideas, Novel Problems,' a new benchmark and pipeline to test how reliably LLMs can evaluate the originality of scientific concepts. Researchers found that minor prompt changes cause dramatic shifts in judgments across multiple leading LLMs (GPT-5.x, Claude Sonnet/Opus). Furthermore, the evaluation proved particularly brittle when applied to AI-generated ideas compared to human-authored ones. The research concludes that current LLM-based novelty evaluation is an unstable, unsolved problem, suggesting that simple prompting can outperform complex, expensive retrieval-augmented systems.
Key Points
- Novelty evaluation using LLMs is highly unstable, with small prompt changes causing massive shifts in judgment accuracy.
- The reliability of novelty judgment degrades significantly when applied to AI-generated ideas compared to human-authored ones.
- Simple, well-prompted baselines can outperform complex, expensive, literature-grounded retrieval systems for novelty assessment.
Why It Matters
This research highlights a critical failure point in the automation of scientific discovery: the inability to reliably measure true originality. If AI systems are to become primary engines for generating research hypotheses, the industry must solve the 'novelty judge' problem, as current methods are shown to be brittle, expensive, and inconsistent across different models and prompts.