BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BLOOM: многоязычная открытая языковая модель с 176 миллиардами параметров
Ofir Press, Victor Sanh, Peter Henderson, Deepak Narayanan, Margaret Mitchell, Douwe Kiela, Stella Biderman, Leo Gao, Jason Phang, Pascale Fung, Thomas Scialom, Dan Garrette, Albert Webson, Nora Kassner, Amit Alfassy, Iz Beltagy, Niklas Muennighoff, Jesse Dodge, Kyle Lo, Mohammad Shoeybi, Thomas Wolf, Patrick von Platen, Yacine Jernite, Teven Le Scao, Quentin Lhoest, Alexander M. Rush, Chirag Jain, Thomas J. Wang, Sampo Pyysalo, Rishi Bommasani, Colin Raffel, Long Phan, Pedro Ortiz Suárez, Benoît Sagot, Matteo Manica, Xiangru Tang, Sebastian Gehrmann, Dragomir Radev, John Giorgi, Adam Roberts, Edward Tan, Irene Solaiman, Scao, Teven Le, Thibault Févry, A. Villanova del Moral, Suraj Patil, Mario Šaško, Gunjan Chhablani, Angelina McMillan-Major, Stas Bekman, Zhiqing Sun, Jason Fries, Mustafa Ghaleb, Hyung Won Chung, Shayne Longpre, Simon Ott, Matthias Samwald, Nouamane Tazi, Aurélie Névéol, Ayush Singh, Wojciech Kusa, Valentin Danchev, Fatima N. Mirza, Marissa Gerchick, Alison Callahan, Nazneen Fatema Rajani, Danish Contractor, Lintang Sutawika, Khalid Almubarak, Zaid Alyafeai, Tim Dettmers, Hady Elsahar, Hendrik Strobelt, Verena Rieser, Anna Rogers, Germán Kruszewski, Matthias Gallé, Mario Sänger, Benjamin Heinzerling, Ludovic Tanguy, Yonatan Belinkov, Huu Du Nguyen, Wilson Y. Lee, Michael J. McKenna, Urmish Thakker, Anne‐Laure Ligozat, Samyam Rajbhandari, Minjia Zhang, Jeff Rasley, Jared Casper, Shaden Smith, Olatunji Ruwase, Max T. Huang, Nafis Abrar, Genta Indra Winata, Samuel Cahyawijaya, Timo Schick, Ehud Reiter, Hieu Tran, Jonathan Chang, Dedong Yu, Chenxi Zhou, Patrick Haller, Е. В. Волошина, Yada Pruksachatkun, Tristan Thrush, Rachel Bawden, Marine Carpuat, Elizabeth Salesky, Chris Chinenye Emezue, Salomey Osei, David Ifeoluwa Adelani, Shamsuddeen Hassan Muhammad, Sabrina J. Mielke, Stefan Schweter, Davis David, Stephen H. Bach, Charles Lovering, Alexandra Sasha Luccioni, Pawan Sasanka Ammanamanchi, Max Ryabinin, Paulo Villegas, Zeerak Talat, Miruna Clinciu, Manan Dey, Maraim Masoud, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak R. Tunuguntla, Oskar van der Wal, Lucile Saulnier, Eduardo González Ponferrada, Gérard Dupont, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Daniel van Strien, Itziar González-Dios, Aitor Soroa, Aaron Gokaslan, Ian Yu, Suhas Pai, Krahmer, Emiel, Clouth, Felix, Hommes, Saar, Vromans, Ruben, Pauws, Steffen, Vermunt, Jeroen, van de Poll-Franse, Lonneke, Verbeek, Xander, Daniel Hesslow, Tatiana Shavrina, Maximin Coavoux, François Yvon, Gully Burns, Sylvain Viguier, Marc Pàmies, Workshop, BigScience, :, Fan, Angela, Akiki, Christopher, Pavlick, Ellie, Ilić, Suzana, Roman Castagné, Jonathan Tow, Hugo Laurençon, Launay, Julien, Adi Simhi, Aji, Alham Fikri, Ariel Kreisberg Nitzav, Chi Xu, Chenghao Mou, Christopher Klamm, Colin Leong, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Hamza Benyamina, Idris Abdulmumin, Isaac Johnson, Javier de la Rosa, Jenny Chim, Zhu, Jian, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Kimbo Chen, Leandro Von Werra, allal, Loubna Ben, María Grandury, Singh, Mayank, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad Ali Jauhar, Subramani, Nishant, Nurulaqilla Khamis, Omar Espejel, Ona de Gibert, Colombo, Pierre, Priscilla Amondi Amuok, Rheza Harliman, Roberto Luis López, Rui Ribeiro, Shamik Bose, Somaieh Nikpoor, Stanislav Silberberg, Sydney Zink, Tiago Timponi Torrent, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Arun Raja, Si, Chenglei, Taşar, Davut Emre, Sharma, Abheesht, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Wang, Han, Harshit Pandey, Jos Rozen, M Saiful Bari, Maged S. Al-shaibani, Nihal V. Nayak, Teehan, Ryan, Albanie, Samuel, Shen, Sheng, Srulik Ben-David, Taewoon Kim, Tali Bers, Trishala Neeraj, Vikas Raunak, Zheng-Xin Yong, Shaked Brody, Yallow Uri, Hadar Tojarieh, Conglong Li, Hatim Bourfoune, Mayank Mishra, Myriam Peyrounette, Nicolas Patry, Omar Sanseviero, Pierre Cornette, Pierre François Lavallée, Lacroix, Rémi, Sanchit Gandhi, Stéphane Requena, Ahmed Baruwa, Singh, Amanpreet, Anastasia Cheveleva, Ekaterina Taktasheva, Eli Bogdanov, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Kawamura, Ken, Liam Hazan, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Rui 蕊 Zhang 张, Ruochen Zhang, Mirkin, Shachar, Shani Pais, Yun, Tian, Tomasz Limisiewicz, Vitaly Protasov, Mikhailov, Vladislav, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Ahmed Khan, Amy Faranak, Santos, Ana, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Olusola Ajibade, Bharat Kumar Saxena, Carlos Muñoz Ferrandis, McDuff, Daniel, David Lansky, Duong A. Nguyen, Emi Baylor, Ezinwanne Ozoani, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Lívia Almeida Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Martha Akinlolu, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, An, Ran, R. Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Anima Shukla, Antonio Miranda-Escalada, Benjamin Beilharz, Wang, Bo, Caio Brito, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Gıyaseddin Bayrak, Helena U. Vrabec, Imane Bello, Ishani Dash, Kang, Jihyun, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Liu, Lu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Maria A Castillo, Marianna Nezhurina, Michael Cullan, Michael C. Weinberg, M. W. P. de Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Kang, Myungsun, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, N. Muellner, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Sang-aroonsiri, Sinee, Srishti Kumar, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Xu, Yifan, Yan Xu, Xu, Yu, Zhe Tan, Xie, Zhongli, Zifan Ye, Mathilde Bras, Younes Belkada, Hesslow, Daniel
176B-parameterBLOOMROOTS corpusdecoder-only Transformermultilingual language model
Figures from the paper
Abstract (AI)
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.
Key Findings
1
BLOOM achieves competitive performance on a wide variety of benchmarks compared to existing models.
2
BLOOM is a 176-billion-parameter open-access decoder-only Transformer language model developed by a large collaborative effort.
3
BLOOM was trained on the ROOTS corpus containing hundreds of sources across 59 languages (46 natural languages and 13 programming languages).
4
Multitask prompted finetuning further improves BLOOM's performance on evaluated tasks.
5
The authors publicly release BLOOM models and code under the Responsible AI License to support democratized research and applications.
Research Object
BLOOM, a 176-billion-parameter open-access decoder-only multilingual language model trained on the ROOTS corpus
Research Subject
Model performance and accessibility: evaluation of BLOOM's competitive performance across diverse benchmarks and the effects of multitask prompted finetuning, alongside its open-access release to democratize LLM technology