OpenAI Says Its Models Are Getting Better at Moving Old COBOL Code, but Experts Say Replacing Legacy Systems Is Far Harder

Much of the world’s critical infrastructure runs on software that almost nobody wants to touch. Banks, airlines, the IRS and state governments depend on programs written decades ago, and replacing them is risky because a single broken step could bring services down. OpenAI now says its newest models could help. The question is whether that claim holds up outside the company’s own tests.

On October 19, OpenAI said its latest models showed “substantial improvements” in migrating aging code into modern programming languages. The company said the work could help organizations with “maintaining and modernizing older systems.” COBOL, short for “common business-oriented language,” was one of the languages OpenAI highlighted. It dates to 1959 and is still widely used.

The scale of COBOL’s footprint is hard to overstate. According to IBM, the language is the foundation for more than 40 percent of online banking systems. It supports 80 percent of in-person credit card transactions and handles 95 percent of ATM transactions. It also underpins many government systems, including Social Security and the IRS, as well as services run by most U.S. states. One rough estimate puts the annual cost of COBOL’s inefficiencies to the U.S. economy at about $105 billion.

Replacing it is another matter. Other legacy languages pose similar problems. Fortran still runs scientific computing, weather forecasting and nuclear security codes. Many of these systems are too important to risk breaking, and many of the programmers who understood them have retired or died.

In its blog post, OpenAI reported migrating COBOL code into C++, Java, Kotlin, Python and Rust without changing what the original software does. That condition is central. OpenAI itself said a migration “is considered successful only if all tests pass.”

How accurate is OpenAI's COBOL code migration?

The company’s numbers focused on how often the new code matched the old code’s behavior. GPT-6.1 Sol achieved 87 percent behavior preservation accuracy at low reasoning effort, a measure of how many computational steps a model spends working through a problem before answering. At higher reasoning effort, that figure climbed to 98 percent. At low reasoning effort, GPT-6 Astra scored about 86 percent, GPT-6 Sol scored 21 percent and GPT-5.6 Sol scored 5 percent. OpenAI also said its models had improved at building database software in several languages, COBOL included.

Anthropic has also gone after the legacy market. In February, it published a step-by-step “code modernization playbook” and promoted its coding agent, Claude Code, as a tool for converting COBOL into languages such as Java or Python. The company offered no performance benchmarks. It did say that “the economics of COBOL modernization have shifted.” IBM, which earns part of its revenue helping companies maintain legacy systems, saw its shares drop sharply after the announcement. Anthropic did not provide comment in time for publication.

Researchers who spoke to The Verge agreed that AI tools have improved at handling legacy code. They said OpenAI’s results are far from enough to replace the systems that matter most.

OpenAI spokesperson Oscar Haines told The Verge in an email that legacy code “is still far from a solved problem.” He said the company is gathering feedback from developers and enterprises, and is exploring ways to build better tests and deepen its models’ understanding of programming languages.

Why is replacing COBOL systems so hard?

Vadim Zaytsev, an associate professor of software evolution at the University of Twente in the Netherlands, put it bluntly. “Rewriting code is not the same as replacing a system.” Size is part of the problem. Zaytsev said the largest system he had seen contained 250 million lines of COBOL, which he described as “definitely not the world’s largest.” But code is only one piece. A system may span several languages, store data in multiple formats and connect to other systems. People and organizations also have expectations the new system has to meet. Zaytsev said all of this means “AI is not playing a major role at the moment in replacing or migrating legacy COBOL systems.”

Arie van Deursen, a software engineering professor at Delft University of Technology in the Netherlands, called OpenAI’s blog “superficial.” He said it offered few details about the migration task. He also said the company’s language benchmark gave models a relatively simple, well-defined job, on systems similar to ones they likely saw in training. “Maintaining is of course a lot easier than modernizing,” van Deursen said, calling the tasks OpenAI used “lightyears” away from real modernization. “The evaluations shown by OpenAI don’t address the key difficulty: Breaking up a complex legacy system into smaller units that can be replaced individually.”

Alastair Donaldson, a professor of programming languages at Imperial College London, said the biggest obstacle is often comprehension rather than rewriting. “The main challenge is indeed not understanding how the system works,” he said. He called that gap “very common” for systems built in COBOL, and noted that very few people still know how to program in it. Donaldson also said “nobody in their right mind would use COBOL for a new development project.”

Can AI test legacy code migrations?

Testing is where the experts see the most promise. Donaldson suggested running the same workflows through old and new systems to confirm they behave identically. AI could also document its translations so people familiar with the legacy language can review the work. Jacopo Mauro, a mathematics and computer science professor at the University of Southern Denmark, pointed to formal verification, which uses mathematical techniques to check that code behaves correctly. Those methods already help verify AI-generated mathematical results. Applying them to legacy code would take a “huge amount of work,” Mauro said. Once in place, he added, automated checks would be “better than having a human check that the translation is correct.” For now, he said, “human verification remains essential.”

The stakes explain the caution. “In many cases even minor differences in system behaviour before and after migration are unacceptable,” Zaytsev said. AI can likely cut costs and speed up parts of the work. But partial automation may not be enough for systems this critical. “Suppose you have a magic AI wand that handles 99 percent of your code,” Zaytsev said. “What do you do with the remaining 1 percent? That 1 percent can be a million lines of code, spread all over the place and subtly failing in more ways you can imagine within your lifetime.”

Questo articolo è stato scritto con l'assistenza dell'IA.
News Factory APP - notizie agentiche per potenziare il tuo SEO e AEO.