METR's Updated Time Horizon Plot Shows AI Still Lags on Judgment-Heavy Tasks
This summary and analysis were generated by AI from the original article at MIT Technology Review AI and may contain errors (how Viqus works). Read the source for full details.
5
What is the Viqus Verdict?
We evaluate each news story based on its real impact versus its media hype to offer a clear and objective perspective.
AI Analysis:
The piece is framed around the fear of automated AI R&D, but the report itself shows models failing open-ended tasks and its authors admit the trends are hard to establish, so the real-world signal is modest.
Article Summary
The AI evaluation nonprofit METR has updated its widely shared time horizon plot, which measures how long a task a model can complete, as judged by human completion times. The original plot showed the length of tasks frontier models could finish doubling roughly every four months. The new version adds longer tasks and more open-ended ones, including designing AI evaluations and scoring philosophical arguments. On these, models fall short of humans: none of the AI-designed evaluations were good enough for METR staff to use in their research, though newer models such as Opus 5.5 reached a "borderline above the bar" level, up from "clearly below the bar" a year earlier. Much of the new completion-time data is estimated rather than measured, and the philosophy task relied on one rater, which limits how far the results can be trusted. Outside experts, including Sayash Kapoor and Najoung Kim, welcome the broader evaluation design but say it is too early to extrapolate trends or to judge how close AI is to automating AI research and development.
Key Points
- METR's updated time horizon plot confirms exponential gains on verifiable tasks but shows models still trail humans on open-ended, judgment-heavy work.
- No AI-designed evaluation was good enough for METR staff to use, though newer models are approaching the usable threshold.
- Many of the new human completion times are estimates, and some judgments rest on a single rater, so trend predictions remain uncertain.
Why It Matters
This is a useful methodological update rather than a new capability breakthrough. It matters because the gap between verifiable tasks, where reinforcement learning works well, and judgment-heavy tasks like research taste is central to whether AI can automate R&D. The report gives evaluators a more honest picture, but its own authors say the hard-to-verify results are difficult to quantify and extrapolate, so it does not settle the question of recursive self-improvement.

