5.4% to 86.6%: OpenAI's Models Just Learned COBOL Migration, and the Rounding Rules Are the Hard Part
GPT-6.1 Sol now migrates COBOL programs with 86.6% accuracy at low reasoning effort — up from 5.4% for GPT-5.6 Sol and 39% cheaper — while GPT-6 Astra lifts average MultiLangBench pass rates from 26.8% to 87.4% across 20 languages, with COBOL still the weakest at 72.1%.
The language that clears 95% of the world's ATM transactions was, until this week, the one frontier models were worst at.
On 19 October OpenAI published two evaluations aimed squarely at the least fashionable corner of software: COBOL, Fortran and Visual Basic. In its own framing, COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions (IBM). Those are the numbers that make a benchmark table worth reading.
What actually happened
OpenAI built two tests, and they measure opposite halves of the same job.
MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in one of 20 programming languages. The model has to organise a codebase, manage state, handle errors and keep earlier features working. Because SQLite database files act as a shared interface, the same test suite can judge a COBOL implementation and a Python one — and the pass rate measures how much working software came out of a long coding session, not whether the code looks plausible.
At maximum reasoning effort, GPT-6 Astra averages an 87.4% test pass rate across the 20 languages, against 26.8% for GPT-5.6 Sol, with gains in every language tested. The per-language jumps are the interesting part: COBOL from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, Visual Basic from 27.5% to 89.4%, OCaml from 25.6% to 85.8%. Astra clears 83% in 19 of the 20 languages. The twentieth is COBOL, still the hardest at 72.1%.
The cheaper tier moved too. GPT-6 Luna averages 40.3% — better than the previous generation's 26.8% — at an estimated average API cost of $0.77 per sample against $3.48, roughly 78% lower.
The COBOL migration evaluation is the one that matters commercially. A model is given an existing COBOL program and asked to migrate it to C++, Java, Kotlin, Python or Rust while preserving behaviour. Success is not "the new code compiles": the model is scored against hidden tests that require byte-for-byte parity on observable outputs, and a migration counts only if every test passes.
GPT-6.1 Sol reaches 86.6% accuracy at low reasoning effort on that test, against 5.4% for GPT-5.6 Sol at the same setting — at 61% of the previous API cost, or 39% less. At higher reasoning effort, accuracy reaches 97.9%, which OpenAI describes as close to saturating the evaluation.
OpenAI has published the MultiLangBench setup, test suite and per-language results on GitHub, so the harness can be inspected rather than taken on trust.
The trap OpenAI put in its own announcement
The most useful paragraph in the post is not a score. It is about rounding.
COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested. A rewrite into Python or Java that rounds those values instead will run without a single error and produce different financial results. That is why the evaluation demands byte-for-byte parity on outputs rather than functional similarity, and why the company adds that applying this to production software "still requires expert review and testing against the system's business rules and dependencies."
That caveat is the whole story for anyone planning real work. A 97.9% benchmark score and a 100% correct ledger are not the same claim, and the gap between them is where migration projects have historically died — not on translation, on reconciliation.
Two honest limits on the numbers themselves: OpenAI wrote the evaluations it is reporting on, and its published figures are its own measurements of its own models. The suite being public makes the claim inspectable, which is more than most vendor benchmarks offer, but inspectable is not independent. Nothing here has been run by a regulator, an auditor, or on your system.
Why this matters for Indian SMBs and mid-market teams
The cost of reading old code was the moat, and it is thinning. The reason COBOL cores have survived for decades is not that banks love COBOL. It is that understanding what a forty-year-old program actually does — the interest rule nobody documented, the fee exception added in 1998 — costs more than replacing the hardware it runs on. A model that accurately recovers intent from code changes the shape of that cost. It does not make migration cheap; it moves the expense from "figure out what this does" to "prove the new thing does exactly the same".
In India the same arithmetic shows up in unglamorous places. Core banking and insurance policy administration in public-sector and cooperative institutions, provident-fund and pension processing, telecom billing, utility billing, and a long tail of government systems written long before anyone here was writing APIs. Add the domestic IT services business that has earned a living maintaining exactly this software. The buying question is no longer whether a model can translate a COBOL program. It is who signs off that the translated program computes the same numbers.
Treat the benchmark as a screening tool, not a project plan. $0.77 versus $3.48 per sample is a unit cost for one evaluation sample, not a migration budget. Before any pilot, a mid-market team should build a parity corpus from its own code: a hundred real programs, the inputs that exercise each business rule, and the exact expected outputs. Run the model against that, not against MultiLangBench. Where the pass rate lands on your code — not someone's 20-language average — is the only number that predicts your project.
Wrap before you rewrite. For most organisations the first win is not conversion. It is documentation: have a model read the COBOL and write down the business rules in plain language, then have a human confirm those rules against the actual outputs. That produces value even if you never migrate a line, and it de-risks the migration decision instead of pre-committing to it.
Where Guruji Corporation fits in
We do not migrate core banking systems, and we will not pretend a benchmark is a warranty. What we do is build the measurement layer that makes a migration decision defensible: a task set drawn from your own code and documents, a parity harness that diffs outputs rather than eyeballing them, and run records that preserve which model version answered which request so a result can be replayed six months later.
For an IT team inside a bank, insurer or government department, that is usually the cheapest part of the programme and the part that decides whether the expensive part is safe to start. We also do the narrower job of turning legacy logic into documentation a current employee can read — often the step that stops a "modernisation project" from becoming a rewrite nobody can scope.
If your team is being shown a migration demo and asked to approve a budget on the strength of it, talk to us. We will build the parity test from your own systems, run it against the shortlist, and tell you plainly which numbers held and which ones only held on a vendor's benchmark.
Sources
Want to see how we can build your project?
Send us your project outline or chat directly on WhatsApp.