OpenAI's COBOL result is striking. Its test asks a narrower question than a production migration

Two tests, two different meanings of success

The 19 October research publication describes MultiLangBench as an extended coding task: build a SQLite-compatible database engine in a specified language, run SQL queries against an initial database, and compare the resulting database with reference SQLite. The same interface allows tests across 20 languages, including COBOL, Fortran, OCaml and Visual Basic. OpenAI says it adapted tests from SQLite's test suite and has released the evaluation setup and per-language results. This tests sustained implementation under a common functional harness. It does not measure whether a model can safely modify an organisation's existing multi-decade application, understand undocumented policy, or run its production controls.

The second evaluation asks models to migrate existing COBOL programs into C++, Java, Kotlin, Python or Rust. Hidden tests require byte-for-byte parity of observable output. That is a stronger constraint than producing code that compiles, but it is still bounded by the programs and cases in the evaluation. OpenAI itself gives a revealing example: COBOL fixed-point arithmetic can discard extra decimal digits unless rounding is requested. A migration that silently rounds may look reasonable and pass simple examples while changing financial results. That is precisely why high test scores do not replace domain review and independent regression tests.

an evaluation-boundary comparison. Both evaluations describe OpenAI's 19 October tests, not independent deployment results.

MultiLangBench

• Input and task: Build a SQLite-compatible engine in one of 20 languages.
• Reported measure: Mean fraction of functional tests passed.
• What it does not prove: Reliability on an existing enterprise estate.

COBOL migration

• Input and task: Port specified programs to another language.
• Reported measure: Share of migrations passing all hidden output-parity tests.
• What it does not prove: Completeness of unknown business rules, integrations or operational controls.

How the released MultiLangBench harness scores an attempt

The public README makes the test more concrete than a percentage alone. It specifies SQLite 3.40.1 and turns an original SQLite TCL test into three pieces: SQL that creates the starting database, a user query passed to the candidate executable, and a validation query run with real SQLite against the resulting database. The candidate passes that case only when the validation output matches the expected string exactly. If the candidate crashes, exceeds the 60-second timeout or produces the wrong output, the case fails. A solution written partly or wholly in a different language is supposed to receive zero for that language.

Of roughly 51,000 original TCL tests, OpenAI says approximately 24,000 could be converted to this input/output form. The README recommends using 272 selected cases for ordinary evaluation because executing all 24,000 is expensive; a full-suite run is possible but should be identified as such. This matters when comparing a reproduction with the publication. The denominator can change if a third party chooses the full suite, a different subset or additional tests. A 72.1% COBOL result is a share of benchmark cases passed under the stated harness, not a probability that arbitrary COBOL code generated for a bank will work.

The environment rules matter as much as the test files. The README says the model may use SQLite documentation, but should have no internet access and no installed SQLite implementation to copy or call. A rerun that leaves a system SQLite library accessible is testing a different capability. The released repository provides tests and setup instructions. The published model figures still reflect OpenAI's runs; sharing a harness enables independent checks but does not itself make those runs independent.

What the numbers actually compare

On MultiLangBench at maximum reasoning effort, OpenAI reports 87.4% average test pass rate for GPT-6 Astra against 26.8% for GPT-5.6 Sol, a difference of 60.6 percentage points. Its COBOL subset rises from 10.9% to 72.1%, still leaving COBOL the lowest of the 20 languages for Astra in this evaluation. Fortran moves from 21.4% to 83.5%, OCaml from 25.6% to 85.8%, and Visual Basic from 27.5% to 89.4%. A percentage-point difference is the right arithmetic here; calling it a 60.6% relative improvement would be a different and misleading quantity.

The lower-cost comparison is GPT-6 Luna versus GPT-5.6 Sol on the same MultiLangBench setup. OpenAI reports 40.3% versus 26.8% average pass rate at estimated API costs of $0.77 versus $3.48 per sample. That is a 13.5-point pass-rate difference and about 78% lower stated per-sample cost. It does not show that Luna is more accurate than Astra, and the sample cost is an evaluation estimate, not a quotation for a production migration.

A compact comparison of four named languages is more useful than a single averaged bar. These are OpenAI's MultiLangBench pass rates at the settings reported in its 19 October publication; all entries have the same test family and unit. The table is a selected subset, not a ranking of all 20 languages.

The four comparisons below all use the MultiLangBench test-pass percentage reported by OpenAI on 19 October 2026 at maximum reasoning effort. The gain is Astra minus GPT-5.6 Sol in percentage points. Each row keeps its language and three model results together so the comparison remains readable on a phone.

• COBOL: GPT-5.6 Sol 10.9%; GPT-6 Luna 17.1%; GPT-6 Astra 72.1%. Astra gain: 61.2 points.
• Fortran: GPT-5.6 Sol 21.4%; GPT-6 Luna 31.4%; GPT-6 Astra 83.5%. Astra gain: 62.1 points.
• OCaml: GPT-5.6 Sol 25.6%; GPT-6 Luna 34.1%; GPT-6 Astra 85.8%. Astra gain: 60.2 points.
• Visual Basic: GPT-5.6 Sol 27.5%; GPT-6 Luna 41.6%; GPT-6 Astra 89.4%. Astra gain: 61.9 points.

For COBOL migration, OpenAI reports 86.6% for GPT-6.1 Sol at low reasoning effort versus 5.4% for GPT-5.6 Sol at the same setting, with 39% lower API cost. It says higher reasoning effort reaches 97.9%. These percentages belong to the migration evaluation and must not be plotted on the same axis as MultiLangBench pass rates as though they share a denominator.

What the two scores cannot be combined to claim

MultiLangBench scores the fraction of SQLite-style functional cases passed by a newly built database engine. The migration evaluation scores the fraction of whole program conversions that pass every hidden output-parity test. A 97.9% migration success rate is therefore not higher on one shared scale than an 87.4% MultiLangBench pass rate. The tasks, denominators, model choices and reasoning settings differ. Likewise, OpenAI's $0.77 and $3.48 figures estimate API spending per benchmark sample, not the total cost of a migration including test design, people, review, integration, incident handling and subsequent maintenance.

A migration team also needs to ask which behaviours are visible to a byte-for-byte output test. A specific test can catch fixed-point rounding, field padding or missing-record handling when its fixtures exercise those paths. It cannot guarantee that an undocumented path, batch dependency, operator procedure or regulatory control has been covered. That is a test-coverage question, not a criticism of exact-output grading; exact output is stricter than compilation but bounded by its fixtures.

What a careful buyer should test

An organisation evaluating AI-assisted maintenance can use OpenAI's released suite to understand the tasks and reproduce results, then create tests from its own repository. The relevant sample includes data layouts, decimal arithmetic, missing-record behaviour, batch jobs, file encoding, external interfaces, error handling and audit requirements. Compare the generated program with the original on real and adversarial fixtures, including cases the model has not seen. Record the model, reasoning setting, total cost, time, human corrections, test coverage and defects found after review. A low-cost high-score demonstration is useful evidence of capability; it is not authority to migrate a payment path without acceptance testing.

OpenAI is both the model provider and evaluator. Its public release of the test setup improves inspectability, but the figures are not independent replication. A useful pilot would take a small, permissioned slice of an actual estate and freeze its acceptance criteria before running any model. Preserve original programs and data fixtures; compare input/output files across normal, boundary and malformed cases; review arithmetic, date handling, encodings and exception paths; and require accountable human approval before replacing a production component. Track each model's first-pass result and the full repair loop separately. Otherwise a low token bill can hide a large engineering bill.

The next material evidence would be reproducible third-party runs on the released 272-case subset and, where practical, the full converted suite, with environment and model settings disclosed. The separate COBOL migration claim needs inspectable task coverage and independent runs on held-out programs. Real maintenance work should report failed cases, human-review effort, elapsed time and regressions after deployment as well as headline pass rates.

Sources checked 20 October 2026: OpenAI publication; OpenAI MultiLangBench repository.