Developers often treat a large language model as a quick fix for old code. Paste in a legacy file, type "modernize this," and wait for the rewrite. A new paper posted to arXiv on Oct. 29 asks whether that shortcut actually leaves the code in better shape. For most of the files tested, the answer was no.
The researchers gave five LLMs the same task: take 100 PHP files from WordPress 4.0 and modernize them to PHP 8.3 while keeping their behavior intact. The models worked on each file alone. They had no information about the wider project and no feedback from running the code.
To grade the results, the team used 630 hidden behavioral checks, which tested whether each file still did what it was supposed to do. They also counted 511 modernization findings flagged by Rector, a rule-based tool that rewrites code according to fixed rules. Every output was measured against two baselines: the original, unchanged files and Rector's output.
The unchanged originals came out ahead. Without any edits, 83 files passed every check. The best-performing LLM left only 59 files in that state.
Does AI Break Legacy Code That Already Works?
The models also did real damage. Each one broke between 32 and 51 files that had been working before. At most, the same model repaired 10 files that had been failing. Rector broke seven files and repaired six, a far smaller swing in both directions.
Much of the breakage was avoidable. The researchers found that only 37 of the 511 findings, about 7 percent, required a change to run under PHP 8.3. Most of the code did not need to be touched at all. Yet in files that were already compatible and needed no required change, the LLMs broke between 36 and 60 percent of them. Rector broke 8 percent of those same kinds of files.
The model with the highest modernization score was not any more compatible than its closest competitors. In other words, a model that made more changes was not more likely to produce working code. Changing more lines simply meant more chances to break something.
Most of the regressed checks traced back to files that no longer loaded at all. The paper points to type declarations that PHP itself rejects as a frequent cause. The authors note that these errors do not require knowledge of the wider project to avoid. A model that understood PHP's rules could, in principle, have caught them.
That point carries weight for anyone evaluating AI coding tools. The authors argue that studies of LLM-based modernization should compare results against a no-change baseline, since doing nothing is often the safest option. They also recommend reporting modernization and behavioral preservation as separate measures, so that a tool's ability to update code is not mistaken for its ability to keep that code working.
Should Teams Use an LLM to Modernize Legacy Code?
The study covers only single files, and it does not reflect how a developer might use an LLM inside a full project with test suites and iterative feedback. The authors frame their findings within that narrower setting. Still, the numbers suggest that a one-line prompt is not a substitute for careful review, and that teams weighing an AI rewrite of legacy code should start by asking whether the code needs changing at all.
Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.