Study: LLM Legacy Code Modernization Breaks Code That Already Worked

Key Takeaways

• A preprint posted to arXiv on October 29, 2026 asked five LLMs to modernize the same 100 legacy WordPress 4.0 PHP files to PHP 8.3, one file at a time, with no project context and no feedback.
• Left untouched, 83 of the 100 files passed every hidden behavioral check. The best model, Claude Sonnet 4, left 59. Each model broke 32 to 51 files that had worked and repaired at most 10.
• Rector, a rule-based PHP refactoring tool, broke 7 working files and fixed 6, ending at 82 compatible files.
• Only 37 of the 511 modernization findings (7%) were changes PHP 8.3 actually required. In compatible files that needed no required change, the models broke 36% to 60%.
• Most damage was easy to catch: files that no longer loaded, often because of type declarations PHP itself rejects. The study tested older models (Claude Sonnet 4, GPT-5 Codex, Gemini 2.5) and their first answer only.

LLM legacy code modernization can leave you with less working code than you started with. That's the finding of "When Modernization Breaks Working Code," a study from researchers at the Bangladesh University of Engineering and Technology and BRAC University, posted to arXiv on October 29, 2026. They pasted legacy PHP files into five models, asked for PHP 8.3 with behavior preserved, and compared the result against the simplest baseline there is: not changing the file at all. Every model lost to it.

What happened

Afridi Mahmud, Rifat Shahriyar and three co-authors posted "When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP" on October 29. It tests the shortcut many engineers have tried: paste an old file into a chat assistant, ask it to "modernize this," and take the answer.

The timing is good. A day later, Fortune reported that Cisco president Jeetu Patel had given a 27-year-old engineer unlimited AI tokens to refactor code. Patel's stated goal is refactoring 98% of Cisco's legacy code within 24 months. "I've given him unlimited tokens because he's going to refactor code," Patel told Fortune. Cisco hasn't published how that work is checked. This study shows what the check has to catch.

How the LLM legacy code modernization study worked

The authors took WordPress 4.0, released in 2014 before PHP 7, and picked 100 of its 482 PHP files. The sample deliberately favors files with many outdated patterns. It runs from 31 files of 200 lines or fewer to 12 files over 1,000 lines.

Each model got the full file and the same fixed instruction, which asked for preservation explicitly:

"Your task is to migrate this legacy PHP code up to PHP 8.3 standards while preserving functional equivalence. Preserve observable behavior, including inputs, outputs, side effects, and call compatibility."

No other files, tests, error messages or runtime feedback. The first complete answer per file was kept. The five models were Claude Sonnet 4, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-5 Codex and Llama 3.3 70B Instruct.

Every output then faced 630 hidden PHPUnit checks under PHP 8.3, written before anyone looked at model output and reviewed by a second author. The same checks ran on two baselines: the unchanged originals and Rector, the rule-based modernization tool a PHP developer could run instead.

The details that matter to engineers

Every model lost to doing nothing

Unchanged, the files passed 531 of 630 checks (84.3%), and 83 of 100 files passed all of theirs. So a modernizer had far more working behavior to protect than broken behavior to fix.

Approach	Files passing every check	Working files broken	Failing files fixed	Check regressions
Unchanged originals	83	0	0	0
Rector	82	7	6	12
Claude Sonnet 4	59	32	8	126
GPT-5 Codex	58	33	8	122
Gemini 2.5 Pro	55	38	10	147
Gemini 2.5 Flash	50	40	7	198
Llama 3.3 70B	33	51	1	215

Each model broke at least three times as many files as it fixed. Every model ended with significantly fewer compatible files than the originals. Rector's 82 was statistically indistinguishable from the originals' 83, and it wasn't standing still: it repaired 62 originally failing checks, about as many as four of the five models (58 to 68).

Most of the changes weren't needed

The authors split Rector's 511 modernization findings by necessity. Only 37 (7%) were required, meaning the old construct fails or changes behavior under PHP 8.3. Another 85 were deprecated, and 389 were optional, such as long array syntax. Optional changes made up 82% to 87% of what each model resolved.

That's where the damage came from. Of the files that passed every check and needed no required change, 73 in all, the models broke 26 to 44 (36% to 60%) and fixed at most one. Rector broke 6 (8%). Where a change really was required, the models roughly broke even: four of them fixed 6 to 9 files (Llama 3.3 70B fixed 1), and all five broke 5 to 7.

A higher modernization score didn't help. Gemini 2.5 Pro resolved the most findings (72.8%) but ended with 55 compatible files, behind Claude Sonnet 4 (53.2% resolved, 59 files) and GPT-5 Codex (58.1%, 58 files).

The failures were not subtle

Between 66% and 81% of each model's regressed checks sat in files that no longer loaded or crashed on a tested path. Load-time rejection hit 13 to 23 files per model. Many came from new type declarations PHP won't accept:

• Claude Sonnet 4 and GPT-5 Codex each added a non-void return type to __set() in three files. PHP requires void there, so the file fails to load and every check in it fails.
• Both also introduced a ?callable property type, which PHP disallows.
• Other regressions changed what a function accepts or returns, such as returning integer 123 where callers expected the string '123', or made legacy public properties private.

The authors stress that these errors need "no project context to avoid." You only need to know PHP.

The numbers behind it, and their limits

The cheapest fix the authors tested was a lint fallback: if php -l rejects the output, keep the original file. That recovered 8 to 15 files per model. The best model then reached 69, still 14 short of leaving the code alone. In the paper's words, "lint is necessary, not sufficient."

Be careful how far you carry this:

• The models are older. The study used Claude Sonnet 4, GPT-5 Codex and Gemini 2.5, not current frontier models. The authors say "newer models may avoid specific errors."
• It's a single-shot chat setting. No agent loop, no tests, no error feedback. A coding agent that runs the test suite and retries works differently, and the study doesn't measure it.
• One project, one language. 100 purposely selected WordPress files, one prompt, one generation per file. The authors limit their conclusions to those files.
• The checks cover selected paths. Untested code could hide more regressions, or more repairs.

What holds regardless of model version is the method. If you don't compare against the unchanged file, a 53% modernization score looks like progress when the code got worse.

What this means for you

If you're asked to modernize legacy code with AI

• Measure against the no-change baseline. Run the same tests on the original and the rewrite, and count files broken as well as findings resolved.
• Ask for required changes only. Most breakage here came from optional edits. Name the target runtime's breaking changes (the PHP 8.0 migration guide is the list for PHP) and tell the model to leave the rest alone.
• Gate each file in stages: lint, load it with its real dependencies, then run behavior checks. The paper calls this a "staged workflow" and notes it would catch the load-time failures that dominated.
• Run the deterministic tool first. Where a rule-based migrator exists, such as Rector for PHP, use it for the mechanical part, and save the model for what rules can't do.

If you review AI-generated refactors

• Look hardest at new type declarations, signatures and visibility changes. In this study they were the most visible sign of modernization and a recurring source of regressions.
• Check overrides in the same file. One model added a return type to a parent method but not to its subclass override.
• Treat the first answer as a draft. That's the authors' own advice.

If you lead a migration

Report preserved behavior, regressions and repairs separately from "findings fixed." A dashboard that shows only the last number would have rated every model in this study as progress.

How this connects, and how it comes up in interviews

Our analysis of whether AI will replace software engineers argues that verification and review are now the most valuable everyday skills. This study puts numbers on why: the output looked modern, and even the best model broke 32 of the 83 files that had worked.

It also fits the two coding-agent studies we covered this month. One found coding agents follow stale instruction files over the code. The other found coding agents follow repository precedent over your intent. All three point to the same habit: decide what "correct" means before the model writes anything, then check it.

In AI-assisted interview rounds, that habit is what's being graded. Interviewers watch whether you test what the assistant produced before you accept it. We cover those formats in how coding interviews changed in the age of AI.

FAQ

Can LLMs modernize legacy code safely?

Not in a single pass without checks, according to this study. Five models modernizing 100 legacy PHP files each broke 32 to 51 files that had worked, against 7 for the rule-based tool Rector.

Is Rector better than ChatGPT or Claude for PHP upgrades?

For preserving behavior in this study, yes. Rector kept 82 of 100 files fully compatible, against 59 for the best model. The models made more optional modernizations, which is where most of their breakage came from.

Why did the LLM-modernized files break?

Mostly because they no longer loaded. New type declarations that PHP rejects, such as a non-void return type on __set(), were a common cause. Other failures changed accepted values, interfaces or outputs.

Do newer models still have this problem?

Unknown. The study tested Claude Sonnet 4, GPT-5 Codex, Gemini 2.5 Pro, Gemini 2.5 Flash and Llama 3.3 70B, each giving one answer with no feedback. The authors say newer models may avoid specific errors.

Further Reading

• Mahmud et al., "When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP" (arXiv:2610.40269): the abstract and PDF, with per-model tables and limitations.
• Fortune: Cisco's president on unlimited AI tokens to refactor legacy code (October 30, 2026).
• Rector documentation: the rule-based baseline used in the study.