1. [Publications](/publications)
2. Using Large Language Models for Hyperparameter Optimization

 # Using Large Language Models for Hyperparameter Optimization

  ![Publication image](/sites/default/files/styles/wide/public/default_images/default.jpeg?itok=TfIobf92 "Publication image")

 This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.

 ## Authors

Michael R. Zhang (University of Toronto, Vector Institute)

Nishkrit Desai (University of Toronto, Vector Institute)

Juhan Bae (University of Toronto, Vector Institute)

Jonathan Lorraine (University of Toronto, Vector Institute)

Jimmy Ba (University of Toronto, Vector Institute)

 ## Publication Date

Thursday, December 7, 2023

 ## Published in

[NeurIPS](https://neurips.cc/Conferences/2023)

 ## Research Area

[Algorithms and Numerical Methods](/research-area/algorithms)

[Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence)

 ## External Links

[Paper](https://arxiv.org/abs/2312.04528)

[OpenReview](https://openreview.net/forum?id=FUdZ6HEOre)

 ## Award

Oral