MM-LLMs: Recent Advances in MultiModal Large Language Models
MM-LLMs: последние достижения в области мультимодальных больших языковых моделей
2024-01-01
SCID: 54.1/4yq98hph
Discuss with AI
LLM augmentationmodel architecturemultimodal large language modelsmultimodal taskstraining pipeline
Figures from the paper
Abstract (AI)
In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies.The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks.In this paper, we provide a comprehensive survey aimed at facilitating further research on MM-LLMs.Initially, we outline general design formulations for model architecture and training pipeline.Subsequently, we introduce a taxonomy encompassing 126 MM-LLMs, each characterized by its specific formulations.Furthermore, we review the performance of selected MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs.Finally, we explore promising directions for MM-LLMs while concurrently maintaining a real-time tracking website 1 for the latest developments in the field.We hope that this survey contributes to the ongoing advancement of the MM-LLMs domain.
Key Findings
1
It introduces a taxonomy of 126 MM-LLMs, categorizing each according to its architectural and training formulations.
2
MultiModal Large Language Models increasingly extend off-the-shelf LLMs to process multimodal inputs or generate multimodal outputs through cost-effective training strategies.
3
The survey formalizes general MM-LLM architecture designs and training pipelines, providing a framework for analyzing the field.
4
The survey reviews benchmark performance, summarizes effective training recipes, identifies promising research directions, and maintains a real-time tracking website for field developments.
5
These models retain the reasoning and decision-making capabilities of LLMs while supporting a diverse range of multimodal tasks.
Research Object
MultiModal Large Language Models (MM-LLMs)
Research Subject
Recent advances, architectural and training formulations, capabilities, benchmark performance, and development directions of MM-LLMs
Publication Details
Publication Date
2024-01-01
Journal
Findings of the Association for Computational Linguistics: ACL (Findings of ACL)
Publisher
Association for Computational Linguistics
ISSN
Cited by
798
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai14
Deep Residual Learning for Image Recognition2016
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
LLaMA: Open and Efficient Foundation Language Models2023
A Survey of Large Language Models2026
Retrieval-Augmented Generation for Large Language Models: A Survey2023
A Comprehensive Overview of Large Language Models2025
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units2021
GPT-4 Technical Report2023
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering2019
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models2023
GLM: General Language Model Pretraining with Autoregressive Blank Infilling2022
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation2023
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent2024