A coding benchmark score belongs to the model and its tools together

In January 2025, Anthropic reported 49% on SWE-bench Verified for an upgraded Claude 3.5 Sonnet agent. That is a historical result, not a statement about today’s frontier.

The useful part is the surrounding system. The model received shell and editing tools, instructions, a repository, and a loop in which it could inspect the consequences of its actions. Anthropic emphasized that changing this scaffolding could materially change the score without changing the underlying model.

One small repair was requiring absolute file paths, preventing mistakes after the agent changed directories. Tool descriptions were treated as interface design rather than incidental documentation. The benchmark involved fixing real repository issues against tests, but it still measured this particular model–environment combination. Comparing headline percentages without comparing those conditions can conceal where the improvement actually came from.