A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications

Обзор основ и передовых направлений мультимодальных агентных фреймворков: методы и приложения
Philip H. S. Torr, Mike Zheng Shou, Dinesh Manocha, Mohamed Elhoseiny, Shraman Pramanick, Kevin Qinghong Lin, Sayan Nag, Arda Senocak, Neel Mokaria, Rishie Raj, Dheeraj Baiju, Xiaoqian Shen, Yapeng Tian, Ruohan Gao, Salman Khan, Sanjoy Chowdhury
2026-06-28

large multimodal modelsmodality-centric taxonomymultimodal agentic frameworksmultimodal fusion architecturesmultimodal reasoning
Advances in large language models (LLMs) have fueled a wave of research into agency: the ability to reason, plan, and act. This effort has produced agentic frameworks that orchestrate perception, memory, and decision-making around powerful LLM backbones. With the advent of large multimodal models (LMMs), these systems can process and integrate diverse modalities, including images, audio, and video, thereby improving their real-world applicability. Yet, while surveys of LLM-based agents exist, the role of multimodality in shaping agency has not been systematically examined in recent years. This survey fills the gap by analyzing the impact of multimodality across the core functional modules of the agentic framework: perception, reasoning, planning, memory, and action. Using this lens, we trace the evolution from text-centric agents to multimodal frameworks, examine how modalities are integrated through delegated, late-fusion, and early-fusion architectures, and assess the emergence of agentic behaviors enabled by grounded perception and multimodal reasoning. We organize existing work through a modality-centric taxonomy that links architectural design choices to agent capabilities. Moreover, we review multimodal agentic systems across various application domains, including Robotics, GUI & Web Navigation, Multimedia Content Generation & Editing, and Long-form Video Understanding & Retrieval. Beyond capabilities, we analyze performance across these settings and discuss efficiency-scalability trade-offs, including training and inference costs, latency, and deployment constraints. By focusing on the impact of multimodality in agentic design, we aim to identify key gaps and chart a roadmap toward robust and general-purpose intelligent systems.
1
A modality-centric taxonomy connects architectural integration choices with the capabilities and agentic behaviors enabled by grounded perception and multimodal reasoning.
2
It traces the transition from text-centric agents to multimodal systems integrating images, audio, and video through delegated, late-fusion, and early-fusion architectures.
3
Multimodal agentic systems are reviewed across robotics, GUI and web navigation, multimedia generation and editing, and long-form video understanding and retrieval.
4
The survey identifies efficiency–scalability trade-offs involving training and inference costs, latency, and deployment constraints, while highlighting gaps toward robust general-purpose systems.
5
The survey systematically analyzes how multimodality affects perception, reasoning, planning, memory, and action in agentic frameworks.

multimodal agentic frameworks

the impact of multimodality on agentic capabilities, functional modules, architectures, applications, performance, and efficiency–scalability trade-offs

Publication Details
Publication Date
2026-06-28
Journal
Publisher
ISSN
Cited by
3
Access Type
Author Information
Authors
Philip H. S. Torr
Mike Zheng Shou
Dinesh Manocha
Mohamed Elhoseiny
Shraman Pramanick
Kevin Qinghong Lin
Sayan Nag
Arda Senocak
Neel Mokaria
Rishie Raj
Dheeraj Baiju
Xiaoqian Shen
Yapeng Tian
Ruohan Gao
Salman Khan
Sanjoy Chowdhury
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%