Sam Manning, Pamela Mishkin, CHAI’s Gillian Hadfield, Tyna Eloundou, and Emily Eisner published a paper titled A Research Agenda for Assessing the Economic Impacts of Code Generation Models for OpenAI.

They explain that OpenAI is developing a research program to assess the economic impacts of code generation models and is inviting collaboration with external researchers. Rapid advances in the capabilities of large language models(LLMs) trained on code have made it increasingly important to study their economic impacts on individuals, firms, and society. Codex – an LLM developed by OpenAI by fine-tuning GPT-3 on billions of lines of publicly available code from GitHub – has been shown to generate functionally correct code 28.8% of the time on a sample of evaluation problems (Chenet al.2021). This may have important implications for the future of coding and the economics of the industries that depend on it. In this document, the researchers lay out a research agenda to assess the effects of Codexon economic factors of interest to policymakers, firms, and the public. We make a case for this research agenda by highlighting the potentially broad applicability of code generation models to software development, the potential for other LLMs to create significant social and economic impact as model capabilities advance, and the value of using Codex to generate evidence and establish methodologies that may be applicable to research on the economic impacts of future models. They propose that academic and policy research focus on studying code generation models and other LLMs so that evidence on their economic impacts can be used to inform decision-making in three key areas: Deployment policy, AI system design, and public policy.