AI
METR Report Says AI Is Significantly Accelerating Anthropic R&D
After an 11-week investigation of Anthropic, AI safety research organization METR says AI is significantly accelerating the company’s model development, with the past year roughly equivalent to two years of historical progress, and that the main contribution is not simply increased computing power. The report has also raised concerns among researchers because large portions were redacted.
METR released a report on Thursday assessing Anthropic’s AI R&D. The investigation lasted 11 weeks and was distinct from the program in which the company gave security researchers system access similar to employee-level access after the summer hacking incident. The report says Anthropic’s model benchmark results show that over the past year it made roughly two years of historical progress, with the rate of improvement about twice what it was a year ago.
METR researchers said they are confident that this acceleration did not come solely from expanded training computing power. As of October, most of the acceleration was attributable to AI, and AI’s contribution was still growing. The evidence included interviews with Anthropic employees and an informal Slack survey; employees said that after using the company’s most powerful internal model, their average work efficiency was about 12 times that without AI assistance, roughly double their self-assessed result from using Mythos 5 several months earlier.
The report notes that the acceleration currently comes mainly from AI assisting human researchers, rather than AI working entirely independently, but there are already signs that autonomous contributions could soon play a larger role, and no clear diminishing returns or bottlenecks capable of stopping further acceleration were found. The report also mentions that Anthropic internally uses three new models that have not yet been made public. Because the agreement allowed Anthropic to redact the final report and the company appears to have made extensive use of that authority, the extent to which outsiders can understand the pace and risks of frontier AI progress has also been called into question.