CodeT
Samples both candidate programs and synthetic test suites, then runs a RANSAC dual-execution consensus algorithm to select the best program.
The loop
- Proposes
A code language model independently generates N candidate implementation programs and M candidate unit-test assertions from the problem prompt.
- Judged by
A sandboxed execution engine runs all N x M program-test pairs to build a bipartite pass/fail consistency matrix.
- Steers
A RANSAC-style consensus algorithm identifies the bipartite clique maximizing (#passing_programs x #passed_tests) and selects its representative program.
Headline result
Raised pass@1 on HumanEval to 65.8% (+18.8 points over Codex greedy decoding) and improved MBPP and APPS accuracy by over 15 points.
What this does not prove
If the LLM shares a systematic misconception across both its generated implementations and its generated tests, the consensus clique can still favour a flawed behavior.
How it generalizes the harness
A deterministic RANSAC bipartite consensus algorithm searches the N x M execution matrix to select the most consistent program cluster.
Classification
| What it proposes | Program |
|---|---|
| Generative model | Codex / InCoder code generator |
| Algorithmic authority | Sandboxed execution matrix and RANSAC bipartite consensus solver |
| Feedback | Test results |
| What gets updated | Population of candidates |
| When it runs | At inference time |
| Validity | Empirical (measured) |
| Evidence | Code released |
| Lineage | Original system |