LLAMBO
Integrates LLM conditional sampling into a Bayesian Optimization loop as a zero-shot warm-starter, surrogate model, and acquisition sampler.
The loop
- Proposes
An LLM conditioned on the optimization history and problem metadata proposes candidate hyperparameter configurations.
- Judged by
True black-box objective evaluations (training and scoring ML models on Bayesmark / HPOBench) measure validation loss.
- Steers
Expected Improvement utility scores rank candidate configurations and append the evaluated (x, f(x)) pair to the active-learning history.
Headline result
Outperformed Gaussian Process and Random Forest Bayesian Optimization baselines in the low-trial regime across 25 Bayesmark tasks.
What this does not prove
In high-dimensional continuous spaces (>20 variables) with ratusan of trials, classical numerical GP kernels overtake in-context LLM surrogates.
How it generalizes the harness
- The algorithm is the solver
- Inverted control
- Search over a population
- Multi-fidelity screening ladder
- Intra-algorithmic surgery (inside the while-loop)
A sequential Bayesian Optimization active-learning loop queries the LLM for candidate proposals and scores them on the black-box objective.
Classification
| What it proposes | Formal specification |
|---|---|
| Generative model | GPT-3.5 in-context surrogate and candidate sampler |
| Algorithmic authority | Expected Improvement acquisition ranker and black-box objective evaluator |
| Feedback | Score |
| What gets updated | Prompt context and population of candidates |
| When it runs | At inference time |
| Validity | Empirical (measured) |
| Evidence | Code released |
| Lineage | Original system |