Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

За пределами игры в имитацию: количественная оценка и экстраполяция возможностей языковых моделей
Christopher D. Manning, Adam Santoro, Zijian Wang, Daphne Ippolito, Chris Callison-Burch, Hannaneh Hajishirzi, Danqi Chen, Dan Hendrycks, Andy Zou, Mantas Mazeika, Amanda Askell, Yejin Choi, Yuntao Bai, Deep Ganguli, Danny Hernandez, Jared Kaplan A, Stella Biderman, Leo Gao, Kyle McDonell, Jason Phang, Laria Reynolds, Gaurav Mishra, Pascale Fung, Tal Schuster, Abhinav Rastogi, Gerard de Melo, Andrew M. Dai, Manaal Faruqui, Anders Andreassen, Dan Garrette, Clara E. Rivera, Luheng He, Behnam Neyshabur, Siamak Shakeri, Vinay Ramasesh, Ethan Dyer, Piyush Patil, Vedant Misra, Shyam Upadhyay, Swaroop Ranjan Mishra, Alicia Parrish, Mihir Kale, Tushar Khot, Niklas Muennighoff, Kyle Richardson, Michihiro Yasunaga, Omer Levy, Yufang Hou, Sam Shleifer, Joshua B. Tenenbaum, Abubakar Abid, James Zou, Dar Gilboa, Sajant Anand, Colin Raffel, Barret Zoph, Maarten Bosma, Percy Liang, Michael Strube, Ruslan Salakhutdinov, Shixiang Gu, Sebastian Gehrmann, Rui Zhang, Xiang Ren, Mohit Bansal, Samuel R. Bowman, Christopher Potts, Samuel S. Schoenholz, Emanuele Rodolà, Stefano Ermon, Louis‐Philippe Morency, Jiacheng Xu, Martin Potthast, Matthias Hagen, Benno Stein, Mohit Tiwari, Jonathan Berant, Owain Evans, Stephanie Lin, Daniel Khashabi, Nayeon Lee, Ashish Sabharwal, Chitta R. Baral, Katja Markert, Nitish Shirish Keskar, Kevin Gimpel, Ludwig Schmidt, Karl Krauth, Ekaterina Shutova, Vivek Srikumar, Chandan Singh, Damien Sileo, Xinyi Wu, Gonzalo Jaimovitch-López, Cèsar Ferri, José Hernández‐Orallo, Fernando Martínez‐Plumed, Joshua Rule, Steven T. Piantadosi, Xinran Zhao, Jason Yosinski, Jascha Sohl‐Dickstein, Elizabeth A. Barnes, Chiyu Wu, Li Zhang, Hannah Rashkin, Giambattista Parascandolo, Catherine Stinson, Alice Xiang, Maarten Sap, Sneha Priscilla Makini, Trenton Chang, Sahib Singh, Saif M. Mohammad, John Burden, Francesca G. E. Happe, Aitor Lewkowycz, Stanislas Dehaene, Karen Livescu, Diyi Yang, Ekin D. Cubuk, Rif A. Saurous, Soo-Hwan Lee, Germán Kruszewski, Vera Demberg, Svetlana Kiritchenko, Ellie Pavlick, Yonatan Belinkov, Denis Emelin, Harsh Mehta, Paul Pu Liang, Liam D. Dugan, Luke Metz, Tal Linzen, Stuart M. Shieber, Sam M. Wiseman, Yangqiu Song, Evgenii Zheltonozhskii, Kory W. Mathewson, Alethea Power, Jiaming Song, Riku Arakawa, Guy Gur-Ari, Noah Fiedel, Chu, Eric, Jaehoon Lee, Rosanne Liu, Qiaozhu Mei, Yasaman Bahri, Paul A. Vicol, Nanyun Peng, Matthew L. Leavitt, Dan Roth, Noah Constant, Hinrich Schütze, Genta Indra Winata, Andrea Madotto, Timo Schick, Jesse Engel, Anu Venkatesh, Jackson Kernion, Marco Marelli, Nikita Nangia, Lianhui Qin, Phu Mon Htut, Jan Kocoń, Pegah Alipoormolabashi, Xudong Shen, Siva Reddy, Diganta Misra, Angela Jiang, Jesujoba Oluwadara Alabi, Clemencia Siro, Wei, Jason, Mirac Süzgün, Zijie J. Wang, Derek Tam, Simone Melzi, Rachel Etta Rudolph, Alexander W. Kocurek, Uri Shaham, Andreas Stuhlmüller, Jaap Jumelet, Martha Lewis, Ziyi Wu, Tianle Wang, Raphaël Millière, Chenlin Meng, David Jurgens, Sharon Zhou, Sean T. Casey, Tobias Gerstenberg, Avia Efrat, Animesh Gupta, Christian Claus Voigt, Allen Nie, Frieda Rong, Robert Frank, Gregor Betz, Hongming Zhang, Ethan Hyungkeun Kim, Jörg Frohberg, Andrea Santilli, Jos Rozen, Trishala Neeraj, Vikas Raunak, Jekaterina Novikova, Arul Menezes, Shubham Toshniwal, Aditya Gupta, Yichi Yang, Maheen Farooqi, Derek Chen, Dong‐Ho Lee, Debajyoti Datta, Dohan, David, Levy, Daniel, Jaime F. Fisac, Kumar Shridhar, Mor Geva, Elad Segal, Niklas Deckers, Aykut Erdem, Dieuwke Hupkes, Yifu Chen, Kamil Kanclerz, Piotr Miłkowski, Nathan Chi, Erkut Erdem, Cristina Gârbacea, Spencer Torene, Ray, Alex, Rao, Abhishek, Aarohi Srivastava, Abu Awal Md Shoeb, Fisch, Adam, Adam R. Brown, Adrià Garriga-Alonso, Agnieszka Kluska, Akshat Agarwal, Alex Warstadt, Ali Safaya, Ali Tazarv, Hussain, Aman, Amanda Dsouza, Slone, Ambrose, Ameet Rahane, Anantharaman S. Iyer, Andrew La, Andrew K. Lampinen, Angelica Chen, Anh Vuong, Anna Gottardi, Antonio Norelli, Arash Gholamidavoodi, Arfa Tabassum, Arun Kirubarajan, Asher Mullokandov, Austin Herrick, Ayla Karakaş, Roberts R, Bao Sheng Loe, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Benjamin Inden, Berk Ekmekci, Lin, Bill Yuchen, Blake Stephen Howald, Orinion, Bryan, Cameron Diao, Cameron Dour, Cedrick Argueta, Charles A. Rathkopf, Chris Waites, Ramirez, Cindy, Courtney Ashcraft, Dan Kilman, Freeman, Daniel, González, Daniel Moseguí, Perszyk, Danielle, Eckersley, Peter, Deniz Yüret, David Drakard, Denis V. Kleyko, Dilyar Buzan, Dimitri Coelho Mollo, Schrader, Dylan, Eleanor Hagerman, Elizabeth P. Donoway, Lam, Emma, Tang, Eric, Ernie Chang, Ethan A. Chi, Ethan Jerzak, Eunice Engefu Manyasi, Fanyue Xia, Fatemeh Siar, Chollet, Francois, Mariani, Giorgio, Wang, Gloria, Hana Galijasevic, Hannah Kim, Hayden Bogar, Henry Shevlin, Hiromu Yakura, Hugh Mee Wong, Ian Ng, Isaac Noble, Jack Geissinger, Hilton, Jacob, James B. Simon, James Koppel, Zheng, James, Thompson, Jana, Wingfield, Janelle, Jarema Radom, Jelle Bosscher, Marsh, Jennifer, Jeremy Kim, Jeroen Taal, Jillian Tang, Joan Waweru, Miller, John, Balis, John U., Batchelder, Jonathan, Joseph Boudeman, Guerr, Joseph, Joseph Jones, Joyce Chua, Gopalakrishnan, Karthik, Katerina Ignatyeva, Kaustubh Dhole, Kevin Omondi, Kristen Chiafullo, Ksenia Shkaruta, Lidia Contreras-Ochando, Luca Moschella, Lucas Lam, Lucy Noble, Luis Oliveros Colón, Lütfi Kerem Şenel, Maartje ter Hoeve, Marco Baturan, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Cohen, Michael, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michał Swędrowski, Bevilacqua, Michele, Mike Cain, Mimee Xu, Walker, Mitch, Moin Aminnaseri, Mozhdeh Gheini, Mukund Varma T, Neta Gur-Ari Krakover, Cameron, Nicholas, Roberts, Nicholas, Nick Doiron, Martinez, Nicole, Niveditha S. Iyer, Nuan Wen, Oliver R. Zhang, Agha, Omar, Omar Elbaghdadi, Pablo A. M. Casares, Parth Doshi, Peiyuan Liao, Chang, Peter, Pinyu Hwang, Pouya Pezeshkpour, Priti Oli, Lyu, Qing, Qinlang Chen, Rabin Banjade, Raefer Gabriel, Rahel Habacker, Risco, Ramon, Rhythm Garg, Barnes, Richard, Robbe Raymaekers, Rohan Sikand, Novak, Roman, Roman Sitelew, Ronan LeBras, Rowan Jacobs, Ryan Chi, Lee, Ryan, Ryan Stovall, Ryan Teehan, Rylan Yang, Sam Dillavou, Samuel Gruetter, Han, Sanghyun, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sebastian Bischoff, Sebastian Schuster, Sadeghi, Sepideh, Shadi Sameh Hamdan, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shubh Pachchigar, Shyamolima, Debnath, Simon Thormeyer, Sriharsha Hatwar, Stefan Divic, Stephen Prasad, Summer Misherghi, Tao Li, Yu, Tao, Ali, Tariq, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Phan, Thomas, Tiberius Nkinyili, Timofei Kornev, Titus Tunduny, Shultz, Tyler, Victoria Nyamai, Vinay Uday Prabhu, Vishakh Padmakumar, William Fedus, WILLIAM H. JUN. SAUNDERS, William Zhang, Wout Vossen, Tong, Xiaoyu, Yadollah Yaghoobzadeh, Yair Lakretz, Yiding Hao, Yu Hou, Zachary Seid, Zhuoye Zhao, Wang, Zirui
2022-06-09

BIG-bench benchmarklanguage model capabilitiesscaling lawssocial biassparse transformers
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
1
Across models ranging from millions to hundreds of billions of parameters, performance and calibration generally improve with scale but remain poor relative to human expert raters.
2
BIG-bench introduces 204 diverse tasks contributed by 450 authors across 132 institutions to evaluate emerging and difficult language-model capabilities.
3
Gradual, predictable scaling is common for knowledge- or memorization-heavy tasks, while critical-scale breakthroughs often involve multistep reasoning, multiple components, or brittle metrics.
4
Performance is remarkably similar across dense and sparse model classes, although sparsity provides measurable benefits.
5
Social bias typically increases with scale in ambiguous-context settings, but prompting can reduce this effect.

language models, including dense and sparse transformer architectures, evaluated across scales

their capabilities, limitations, scaling behavior, calibration, breakthrough performance, and social bias across diverse tasks

Publication Details
Publication Date
2022-06-09
Journal
Publisher
ISSN
Cited by
1021
Access Type
Author Information
Authors
Christopher D. Manning
Adam Santoro
Zijian Wang
Daphne Ippolito
Chris Callison-Burch
Hannaneh Hajishirzi
Danqi Chen
Dan Hendrycks
Andy Zou
Mantas Mazeika
Amanda Askell
Yejin Choi
Yuntao Bai
Deep Ganguli
Danny Hernandez
Jared Kaplan A
Stella Biderman
Leo Gao
Kyle McDonell
Jason Phang
Laria Reynolds
Gaurav Mishra
Pascale Fung
Tal Schuster
Abhinav Rastogi
Gerard de Melo
Andrew M. Dai
Manaal Faruqui
Anders Andreassen
Dan Garrette
Clara E. Rivera
Luheng He
Behnam Neyshabur
Siamak Shakeri
Vinay Ramasesh
Ethan Dyer
Piyush Patil
Vedant Misra
Shyam Upadhyay
Swaroop Ranjan Mishra
Alicia Parrish
Mihir Kale
Tushar Khot
Niklas Muennighoff
Kyle Richardson
Michihiro Yasunaga
Omer Levy
Yufang Hou
Sam Shleifer
Joshua B. Tenenbaum
Abubakar Abid
James Zou
Dar Gilboa
Sajant Anand
Colin Raffel
Barret Zoph
Maarten Bosma
Percy Liang
Michael Strube
Ruslan Salakhutdinov
Shixiang Gu
Sebastian Gehrmann
Rui Zhang
Xiang Ren
Mohit Bansal
Samuel R. Bowman
Christopher Potts
Samuel S. Schoenholz
Emanuele Rodolà
Stefano Ermon
Louis‐Philippe Morency
Jiacheng Xu
Martin Potthast
Matthias Hagen
Benno Stein
Mohit Tiwari
Jonathan Berant
Owain Evans
Stephanie Lin
Daniel Khashabi
Nayeon Lee
Ashish Sabharwal
Chitta R. Baral
Katja Markert
Nitish Shirish Keskar
Kevin Gimpel
Ludwig Schmidt
Karl Krauth
Ekaterina Shutova
Vivek Srikumar
Chandan Singh
Damien Sileo
Xinyi Wu
Gonzalo Jaimovitch-López
Cèsar Ferri
José Hernández‐Orallo
Fernando Martínez‐Plumed
Joshua Rule
Steven T. Piantadosi
Xinran Zhao
Jason Yosinski
Jascha Sohl‐Dickstein
Elizabeth A. Barnes
Chiyu Wu
Li Zhang
Hannah Rashkin
Giambattista Parascandolo
Catherine Stinson
Alice Xiang
Maarten Sap
Sneha Priscilla Makini
Trenton Chang
Sahib Singh
Saif M. Mohammad
John Burden
Francesca G. E. Happe
Aitor Lewkowycz
Stanislas Dehaene
Karen Livescu
Diyi Yang
Ekin D. Cubuk
Rif A. Saurous
Soo-Hwan Lee
Germán Kruszewski
Vera Demberg
Svetlana Kiritchenko
Ellie Pavlick
Yonatan Belinkov
Denis Emelin
Harsh Mehta
Paul Pu Liang
Liam D. Dugan
Luke Metz
Tal Linzen
Stuart M. Shieber
Sam M. Wiseman
Yangqiu Song
Evgenii Zheltonozhskii
Kory W. Mathewson
Alethea Power
Jiaming Song
Riku Arakawa
Guy Gur-Ari
Noah Fiedel
Chu, Eric
Jaehoon Lee
Rosanne Liu
Qiaozhu Mei
Yasaman Bahri
Paul A. Vicol
Nanyun Peng
Matthew L. Leavitt
Dan Roth
Noah Constant
Hinrich Schütze
Genta Indra Winata
Andrea Madotto
Timo Schick
Jesse Engel
Anu Venkatesh
Jackson Kernion
Marco Marelli
Nikita Nangia
Lianhui Qin
Phu Mon Htut
Jan Kocoń
Pegah Alipoormolabashi
Xudong Shen
Siva Reddy
Diganta Misra
Angela Jiang
Jesujoba Oluwadara Alabi
Clemencia Siro
Wei, Jason
Mirac Süzgün
Zijie J. Wang
Derek Tam
Simone Melzi
Rachel Etta Rudolph
Alexander W. Kocurek
Uri Shaham
Andreas Stuhlmüller
Jaap Jumelet
Martha Lewis
Ziyi Wu
Tianle Wang
Raphaël Millière
Chenlin Meng
David Jurgens
Sharon Zhou
Sean T. Casey
Tobias Gerstenberg
Avia Efrat
Animesh Gupta
Christian Claus Voigt
Allen Nie
Frieda Rong
Robert Frank
Gregor Betz
Hongming Zhang
Ethan Hyungkeun Kim
Jörg Frohberg
Andrea Santilli
Jos Rozen
Trishala Neeraj
Vikas Raunak
Jekaterina Novikova
Arul Menezes
Shubham Toshniwal
Aditya Gupta
Yichi Yang
Maheen Farooqi
Derek Chen
Dong‐Ho Lee
Debajyoti Datta
Dohan, David
Levy, Daniel
Jaime F. Fisac
Kumar Shridhar
Mor Geva
Elad Segal
Niklas Deckers
Aykut Erdem
Dieuwke Hupkes
Yifu Chen
Kamil Kanclerz
Piotr Miłkowski
Nathan Chi
Erkut Erdem
Cristina Gârbacea
Spencer Torene
Ray, Alex
Rao, Abhishek
Aarohi Srivastava
Abu Awal Md Shoeb
Fisch, Adam
Adam R. Brown
Adrià Garriga-Alonso
Agnieszka Kluska
Akshat Agarwal
Alex Warstadt
Ali Safaya
Ali Tazarv
Hussain, Aman
Amanda Dsouza
Slone, Ambrose
Ameet Rahane
Anantharaman S. Iyer
Andrew La
Andrew K. Lampinen
Angelica Chen
Anh Vuong
Anna Gottardi
Antonio Norelli
Arash Gholamidavoodi
Arfa Tabassum
Arun Kirubarajan
Asher Mullokandov
Austin Herrick
Ayla Karakaş
Roberts R
Bao Sheng Loe
Bartłomiej Bojanowski
Batuhan Özyurt
Behnam Hedayatnia
Benjamin Inden
Berk Ekmekci
Lin, Bill Yuchen
Blake Stephen Howald
Orinion, Bryan
Cameron Diao
Cameron Dour
Cedrick Argueta
Charles A. Rathkopf
Chris Waites
Ramirez, Cindy
Courtney Ashcraft
Dan Kilman
Freeman, Daniel
González, Daniel Moseguí
Perszyk, Danielle
Eckersley, Peter
Deniz Yüret
David Drakard
Denis V. Kleyko
Dilyar Buzan
Dimitri Coelho Mollo
Schrader, Dylan
Eleanor Hagerman
Elizabeth P. Donoway
Lam, Emma
Tang, Eric
Ernie Chang
Ethan A. Chi
Ethan Jerzak
Eunice Engefu Manyasi
Fanyue Xia
Fatemeh Siar
Chollet, Francois
Mariani, Giorgio
Wang, Gloria
Hana Galijasevic
Hannah Kim
Hayden Bogar
Henry Shevlin
Hiromu Yakura
Hugh Mee Wong
Ian Ng
Isaac Noble
Jack Geissinger
Hilton, Jacob
James B. Simon
James Koppel
Zheng, James
Thompson, Jana
Wingfield, Janelle
Jarema Radom
Jelle Bosscher
Marsh, Jennifer
Jeremy Kim
Jeroen Taal
Jillian Tang
Joan Waweru
Miller, John
Balis, John U.
Batchelder, Jonathan
Joseph Boudeman
Guerr, Joseph
Joseph Jones
Joyce Chua
Gopalakrishnan, Karthik
Katerina Ignatyeva
Kaustubh Dhole
Kevin Omondi
Kristen Chiafullo
Ksenia Shkaruta
Lidia Contreras-Ochando
Luca Moschella
Lucas Lam
Lucy Noble
Luis Oliveros Colón
Lütfi Kerem Şenel
Maartje ter Hoeve
Marco Baturan
Marco Maru
Maria Jose Ramírez Quintana
Marie Tolkiehn
Mario Giulianelli
Mátyás Schubert
Medina Orduna Baitemirova
Melody Arnaud
Melvin McElrath
Michael A. Yee
Cohen, Michael
Michael Gu
Michael Ivanitskiy
Michael Starritt
Michał Swędrowski
Bevilacqua, Michele
Mike Cain
Mimee Xu
Walker, Mitch
Moin Aminnaseri
Mozhdeh Gheini
Mukund Varma T
Neta Gur-Ari Krakover
Cameron, Nicholas
Roberts, Nicholas
Nick Doiron
Martinez, Nicole
Niveditha S. Iyer
Nuan Wen
Oliver R. Zhang
Agha, Omar
Omar Elbaghdadi
Pablo A. M. Casares
Parth Doshi
Peiyuan Liao
Chang, Peter
Pinyu Hwang
Pouya Pezeshkpour
Priti Oli
Lyu, Qing
Qinlang Chen
Rabin Banjade
Raefer Gabriel
Rahel Habacker
Risco, Ramon
Rhythm Garg
Barnes, Richard
Robbe Raymaekers
Rohan Sikand
Novak, Roman
Roman Sitelew
Ronan LeBras
Rowan Jacobs
Ryan Chi
Lee, Ryan
Ryan Stovall
Ryan Teehan
Rylan Yang
Sam Dillavou
Samuel Gruetter
Han, Sanghyun
Sanjeev Kwatra
Sarah A. Rous
Sarik Ghazarian
Sayan Ghosh
Sebastian Bischoff
Sebastian Schuster
Sadeghi, Sepideh
Shadi Sameh Hamdan
Shashank Srivastava
Sherry Shi
Shikhar Singh
Shima Asaadi
Shubh Pachchigar
Shyamolima
Debnath
Simon Thormeyer
Sriharsha Hatwar
Stefan Divic
Stephen Prasad
Summer Misherghi
Tao Li
Yu, Tao
Ali, Tariq
Tatsu Hashimoto
Te-Lin Wu
Théo Desbordes
Theodore Rothschild
Phan, Thomas
Tiberius Nkinyili
Timofei Kornev
Titus Tunduny
Shultz, Tyler
Victoria Nyamai
Vinay Uday Prabhu
Vishakh Padmakumar
William Fedus
WILLIAM H. JUN. SAUNDERS
William Zhang
Wout Vossen
Tong, Xiaoyu
Yadollah Yaghoobzadeh
Yair Lakretz
Yiding Hao
Yu Hou
Zachary Seid
Zhuoye Zhao
Wang, Zirui
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%