A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
Обзор основ и передовых направлений мультимодальных агентных фреймворков: методы и приложения
2026-06-28
SCID: 54.1/zn8p6dw3
Discuss with AI
large multimodal modelsmodality-centric taxonomymultimodal agentic frameworksmultimodal fusion architecturesmultimodal reasoning
Figures from the paper
Abstract (AI)
Advances in large language models (LLMs) have fueled a wave of research into agency: the ability to reason, plan, and act. This effort has produced agentic frameworks that orchestrate perception, memory, and decision-making around powerful LLM backbones. With the advent of large multimodal models (LMMs), these systems can process and integrate diverse modalities, including images, audio, and video, thereby improving their real-world applicability. Yet, while surveys of LLM-based agents exist, the role of multimodality in shaping agency has not been systematically examined in recent years. This survey fills the gap by analyzing the impact of multimodality across the core functional modules of the agentic framework: perception, reasoning, planning, memory, and action. Using this lens, we trace the evolution from text-centric agents to multimodal frameworks, examine how modalities are integrated through delegated, late-fusion, and early-fusion architectures, and assess the emergence of agentic behaviors enabled by grounded perception and multimodal reasoning. We organize existing work through a modality-centric taxonomy that links architectural design choices to agent capabilities. Moreover, we review multimodal agentic systems across various application domains, including Robotics, GUI & Web Navigation, Multimedia Content Generation & Editing, and Long-form Video Understanding & Retrieval. Beyond capabilities, we analyze performance across these settings and discuss efficiency-scalability trade-offs, including training and inference costs, latency, and deployment constraints. By focusing on the impact of multimodality in agentic design, we aim to identify key gaps and chart a roadmap toward robust and general-purpose intelligent systems.
Key Findings
1
A modality-centric taxonomy connects architectural integration choices with the capabilities and agentic behaviors enabled by grounded perception and multimodal reasoning.
2
It traces the transition from text-centric agents to multimodal systems integrating images, audio, and video through delegated, late-fusion, and early-fusion architectures.
3
Multimodal agentic systems are reviewed across robotics, GUI and web navigation, multimedia generation and editing, and long-form video understanding and retrieval.
4
The survey identifies efficiency–scalability trade-offs involving training and inference costs, latency, and deployment constraints, while highlighting gaps toward robust general-purpose systems.
5
The survey systematically analyzes how multimodality affects perception, reasoning, planning, memory, and action in agentic frameworks.
Research Object
multimodal agentic frameworks
Research Subject
the impact of multimodality on agentic capabilities, functional modules, architectures, applications, performance, and efficiency–scalability trade-offs
Publication Details
Publication Date
2026-06-28
Journal
Publisher
ISSN
Cited by
3
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai12
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
LLaMA: Open and Efficient Foundation Language Models2023
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks2020
Robust Speech Recognition via Large-Scale Weak Supervision2022
A survey on large language model based autonomous agents2024
Generative Agents: Interactive Simulacra of Human Behavior2023
GPT-4 Technical Report2023
Evaluating Large Language Models Trained on Code2021
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models2023
From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review2026
LLM-Based Agents for Tool Learning: A Survey2025