A new open-source language model named Lemur, introduced in a paper from researchers at the University of Hong Kong and other institutes, aims to bridge the gap between natural language and programming capabilities in large language models. This harmonization of skills is intended to pave the way for more capable and versatile language agents.
Current open-source models tend to specialize in either natural language or code, lacking the balance required for language agents that must understand instructions, reason, and execute actions in code. For example, models like CodeLlama and StarCoder have excellent coding abilities but trail in language benchmarks, while Llama and BLOOM shine on language but do not match proprietary models like GPT-3.5 on coding tests.
Lemur seeks to combine the best of both worlds through a two-stage training process. First, the researchers pre-trained the Llama-2-70B model on a 90 billion token dataset with a 10:1 code-to-text ratio with focus on scripting or interpreted languages (Python, SQL, Bash, Perl, etc.). Next, they instruction fine-tuned Lemur on 300k examples to enable following natural language instructions and code.
Evaluations demonstrate Lemur’s balanced proficiencies, with a 4.3% and 14.8% overall improvement over Llama-2-70B and Llama-2-70B-Chat as well as 1.9% and 9.4% over respective CodeLlama models on combined language and code tests. The chat version, Lemur-Chat, outperforms other open-source models on 12 of 13 new “agent benchmarks” that assess skills like tool usage, incorporating feedback, and exploring environments.
This research provides key insights into optimizing language model capabilities for real-world agent applications through the synergistic blend of natural and programming language skills. Lemur models have been open-sourced to empower the community to build more advanced agents adept at reasoning, planning, and interacting across diverse environments. Their results indicate open-source models are reaching parity with closed alternatives on agent abilities.
The authors highlight Lemur’s potential to power next-generation assistants, control systems, creative tools, and other multifaceted applications requiring tight integration between understanding and executing instructions. By combining intellect and action, Lemur represents an important step towards fully autonomous AI agents able to operate seamlessly across the digital and physical worlds.