Coding: Designing and writing computer software

METR built a collection of 228 tasks, mostly in software engineering, machine learning, and cybersecurity. To learn how hard each task is, METR paid skilled professionals to do the tasks and timed them. These experts had about five years of experience on average, and most attended one of the world’s top 100 universities. The tasks range from a few seconds to more than eight hours of human work.

METR then gave the same tasks to AI agents. For each AI model, it worked out the task length at which the AI succeeds half the time. METR calls this the model’s “50% time horizon.” A time horizon of two hours means the AI succeeds about half the time on tasks that take a human expert about two hours.

Here is how that number has grown for leading models, using METR’s current measurements:^{[1]METR: Task-Completion Time Horizons of Frontier AI Models}

• GPT-4 (released March 2023): about 3.5 minutes
• Claude 3.7 Sonnet (February 2025): about 1 hour
• GPT-5 (August 2025): about 3.5 hours
• Claude Opus 4.5 (November 2025): about 5.3 hours
• Claude Mythos Preview, early version (tested March 2026): about 17 hours

The Mythos Preview result is very uncertain. METR said the time horizon was “at least 16 hours,” with a likely range from 8.5 to 55 hours. METR also warned that any measurement above 16 hours is unreliable, because it does not have enough very long tasks. In other words, the best model outgrew the test.

From 2019 to 2025, the time horizon doubled about every seven months. Since 2024, it has sped up, doubling about every 89 days, or roughly three months. METR also tracks an “80% time horizon,” the task length at which the AI succeeds four times out of five. Those numbers are several times shorter, but they are growing at a similar rate.

METR points out several limits. Its tasks are clean and clearly defined, with an automatic way to check success. Real jobs are messier, and AI agents do worse on messier tasks. METR says its time horizon is closer to what a new hire with no background on a project could do than what a long-time team member could do. The time horizon also measures how hard a task is, not how long the AI spends on it. On tasks they complete, AI agents are usually several times faster than the human experts. METR stopped actively updating this page in September 2026.

In June 2026, the Anthropic Institute, part of the company that makes the Claude AI models, published data on how Anthropic’s own staff use AI to write code. This is a company reporting on its own product, and the numbers have not been independently checked.

Who writes the code. As of May 2026, Claude wrote more than 80% of the new code added to Anthropic’s code library. Before Anthropic launched its coding agent, Claude Code, in February 2025, that share was in the low single digits.

How much code each engineer produces. From 2021 through 2024, the amount of code each Anthropic engineer added per day stayed flat. It began to climb in 2025, when Claude started running code itself instead of only suggesting code for an engineer to copy and paste. It climbed faster in 2026, as models began working on their own for longer stretches. By the second quarter of 2026, the typical engineer was adding 8 times as much code per day as in 2024.^{[5]Anthropic Institute: When AI builds itself} The engineers mostly direct and review the work, and Claude does most of the writing.

Anthropic notes that counting lines of code is a rough measure. More code is not the same as better code, so “8 times as much code” almost certainly overstates the real gain in productivity. In a March 2026 poll of 130 Anthropic research staff, the typical person estimated they got about 4 times as much done with AI as without it. Anthropic thinks the true number was somewhat lower.

How good the code is. Many Anthropic staff felt that Claude’s code was worse than the code their human colleagues wrote in late 2025, and roughly as good by mid-2026. They expected it to be better within a year. Anthropic now has Claude review every proposed change to its code. Looking back at past failures of its claude.ai service, Anthropic estimated that this AI reviewer would have caught about a third of the bugs behind them. Those were bugs that Anthropic’s own expert engineers had missed.

Harder, open-ended work. Anthropic tracks how often a Claude coding session succeeds without a person having to step in and correct it. A Claude model judges whether each session succeeded, which is a limitation of this measure. On the hardest category, open-ended problems where even the engineer does not know what the answer looks like, the success rate rose from about 26% in August 2025 to 76% in May 2026, and to about 91% by September 2026. In one example, a routine upgrade began crashing tens of thousands of AI training jobs. Given little more than access to the computers involved, Claude tested settings one at a time, found the single obscure setting causing the crashes, and confirmed a fix in about two hours. Anthropic says that would normally take two to three days.

Speed on experiments. With each new model, Anthropic runs the same test: take a piece of code that trains a small AI model, and make it run as fast as possible while still passing the same correctness checks. In May 2025, Claude Opus 4 made the code about 3 times faster on average. By April 2026, Claude Mythos Preview made it about 52 times faster. For comparison, a skilled human researcher needs four to eight hours to make it about 4 times faster.

Work that would not have happened otherwise. In April 2026, Claude made more than 800 fixes that cut one type of error in Anthropic’s software by a factor of 1,000. The engineer overseeing the work estimated it would have taken a human four years.

Where humans still lead. Anthropic says Claude can now be handed a vague problem and work out how to solve it: people supply the goal, but no longer need to supply the method. The biggest remaining gap is judgment about which problems are worth working on in the first place.

The surge in AI-written code shows up outside Anthropic too. GitHub, the website where most of the world’s software is stored, saw about 1 billion commits (saved changes to code) in all of 2025. By mid-2026, it was seeing about 275 million per week.

SWE-bench (short for Software Engineering benchmark) is the most widely used test of whether AI can do real programming work. Researchers at Princeton University released it in 2023. Each task comes from a real problem that someone reported in a popular open-source software project on GitHub. The AI gets the project’s actual code and the problem report, and must write a change that fixes the problem. The project’s own hidden tests then check whether the fix works and whether anything else broke.

SWE-bench Verified. In August 2024, OpenAI and the SWE-bench authors released a cleaned-up set of 500 of these tasks, all in the Python programming language. Human software engineers checked each one to make sure it was fair and could be solved. Scores rose quickly:

• When the original SWE-bench came out in late 2023, the best AI model fixed about 2% of its problems.
• When SWE-bench Verified came out in August 2024, the best result was about 33%.
• In July 2026, Anthropic reported that Claude Opus 5 fixed 96%.

In February 2026, OpenAI said it had stopped reporting SWE-bench Verified scores. It reviewed 138 of the hardest problems and found that about 59% of them had flawed tests or problem descriptions, in some cases rejecting correct answers. OpenAI also found signs of “contamination”: because the problems and their fixes are public, AI models may have seen the answers during training. OpenAI recommended using SWE-bench Pro instead.

SWE-bench Pro. The company Scale AI released SWE-bench Pro in September 2025 to be harder and harder to game. It has 1,865 tasks drawn from many kinds of software, including business applications, written in Python, Go, JavaScript, and TypeScript. Many tasks require changes across several files and would take a professional engineer hours or more. Part of the test uses private code from real companies, which AI models could not have seen during training.

At launch, the best models, OpenAI’s GPT-5 and Anthropic’s Claude Opus 4.1, solved only about 23% of the public tasks. On SWE-bench Verified, the same kinds of models were scoring above 70%.

About a year later, AI companies were reporting far higher scores. Anthropic reported about 79% for Claude Opus 5 in July 2026 and 89.9% for Claude Opus 5.5 in September 2026. These company-reported scores use each company’s own tools and settings. They are not directly comparable to Scale’s standardized leaderboard, where every model is run the same way.

AtCoder is a Japanese company that runs one of the world’s largest websites for programming contests. Each year, it invites the top performers from a full year of its contests to the World Tour Finals in Tokyo. Unlike student contests, the finals are open to programmers of any age, so the field includes many of the strongest competitive programmers in the world. The finals have two divisions:

• Algorithm: a set of very hard puzzles, each with a correct answer. A solution counts only if it is completely correct.
• Heuristic: a single, very hard optimization problem with no known perfect answer. Competitors try to get as close to the best possible answer as they can, and the highest score wins.

Since 2025, OpenAI has sponsored the event and entered an AI agent in a “Human vs. AI” exhibition match. The AI follows the same rules and time limits as the human finalists, but its results do not count in the official standings. AtCoder offered a special “Humanity Prevails Award” to any human who both beat the AI and won the contest.

2025: a human narrowly wins. At the July 2025 Heuristic finals, an OpenAI model worked fully on its own through the 10-hour contest. It led for much of the day but was overtaken by Polish programmer Przemysław Dębiak, known online as Psyho. The AI finished second out of 12. OpenAI called it the first known top-three finish by an AI at a leading programming or math competition.

2026 Heuristic division: AI wins by a wide margin. In July 2026, 12 human finalists, who had qualified through AtCoder’s contests during 2025, worked on a single optimization problem over two days. OpenAI’s AI scored more than seven times higher than the best human. No human claimed the Humanity Prevails Award. Psyho, who did not compete this time but followed the contest closely, summed it up on X: “Humanity has not prevailed.”

2026 Algorithm division: AI solves every problem. The next day, the Algorithm division gave competitors five problems and seven hours. Observers said two of them, problems D and E, were unusually hard even by AtCoder’s standards. The AI needed about three hours to crack problem D, but it solved all five problems in about six hours. It scored 8,300 points. The top human, who won the official contest, scored 4,300. No human solved problem C or problem E.

According to Borys Minaiev, an OpenAI researcher and former world champion of the ICPC (International Collegiate Programming Contest), the system was a model similar to OpenAI’s GPT-5.6, plus a small setup that let it spend extra computing power working on each problem. It had no internet access. He said that six months earlier, OpenAI’s system would not have been able to solve most of these problems.

The same pattern shows up in student contests. At the 2025 ICPC World Finals, the top contest for university teams, OpenAI’s system solved all 12 problems and outscored every team. At the International Olympiad in Informatics, a contest for high school students, OpenAI’s system went from the 49th percentile in 2024 to the 98th percentile in 2025.

AlphaEvolve is an AI system from Google DeepMind that writes and improves computer programs in a repeating loop. Google’s Gemini AI models propose many versions of a program. Automated checkers run each version and score how well it works. The best versions are kept and used as starting points for the next round, similar to how evolution keeps the traits that work best. This approach only works on problems where a computer can automatically measure how good an answer is.

Google reported these results in May 2025:

• Data centers. AlphaEvolve found a simple rule for scheduling work across Google’s data centers. It had been running for more than a year and frees up, on average, 0.7% of Google’s computing power worldwide. Google noted that the code was easy for humans to read and check.
• Computer chips. It proposed a change to a key circuit that removed unneeded parts. The change passed strict checks and was included in an upcoming TPU (Tensor Processing Unit), Google’s custom AI chip.
• AI training. It sped up an important piece of code inside Gemini by 23%, which cut the total time needed to train Gemini by 1%. This included training the same models that power AlphaEvolve. Google said it shrank this kind of tuning work from weeks of expert effort to days. It also found speedups of up to 32.5% in a very low-level piece of AI code that human engineers rarely change by hand.
• Math and algorithms. It found a way to multiply two 4-by-4 grids of complex numbers using 48 multiplication steps. That beat a method from 1969 that had been the best known for this case. DeepMind also pointed AlphaEvolve at more than 50 open math problems. It matched the best known answer in about 75% of them and found a better answer in about 20%. For example, it found an arrangement of 593 spheres touching a central sphere in 11 dimensions, setting a new record for a problem mathematicians have studied for more than 300 years.