LLM-Modulo Framework
An LLM proposes candidate plans inside a generate-test-critique loop governed by external symbolic verifiers and programmatic constraint critics.
The loop
- Proposes
Candidate multi-day travel itineraries or symbolic plans generated by an LLM
- Judged by
A bank of external model-based verifiers and programmatic critics checking format, hard budget and scheduling rules, and commonsense constraints
- Steers
A meta-controller passes the critics' critiques to the model as the next prompt until all hard critics sign off.
Headline result
Proposed at ICML 2024 (arXiv:2402.01817) and evaluated on the TravelPlanner benchmark (arXiv:2405.20625), pairing GPT-4-Turbo with external programmatic constraint critics in a bi-directional loop raised the final all-constraints pass rate from 4.4% (direct prompting) to 20.6% (a 4.6x gain), and raised GPT-3.5-Turbo from 0% to 5.0%.
What this does not prove
Even with external critics, 20.6% leaves nearly four out of five complex multi-constraint itineraries unsolved within the retry budget, because critique-guided local edits can still fail when constraints interact globally without an exact solver.
How it generalizes the harness
The LLM-Modulo loop rejects LLM self-verification in favor of sound external model-based verifiers that gate acceptance and drive iterative back-prompting.
Classification
| What it proposes | Program |
|---|---|
| Generative model | GPT-4-Turbo and GPT-3.5-Turbo (frozen) |
| Algorithmic authority | External symbolic verifiers (VAL for PDDL) and programmatic hard-constraint and commonsense critics |
| Feedback | Verifier errors |
| What gets updated | Prompt context |
| When it runs | At inference time |
| Validity | Formal (machine-checked) |
| Evidence | Experiments reported |
| Lineage | Original system |