Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Visual ChatGPT: общение, рисование и редактирование с использованием визуальных базовых моделей
2023-03-08
SCID: 54.1/eg5hjk3d
Discuss with AI
Stable DiffusionVisual ChatGPTVisual Transformersvisual editingvisual foundation models
Figures from the paper
Abstract (AI)
ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called \textbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{https://github.com/microsoft/visual-chatgpt}.
Key Findings
1
A prompt-based mechanism injects visual model capabilities into ChatGPT, accommodating models with multiple input/output types and visual-feedback requirements.
2
The system enables complex visual question answering and image-editing instructions through collaboration among multiple models across multi-step workflows.
3
The system is publicly released, providing a platform for investigating ChatGPT’s visual capabilities with Visual Foundation Models.
4
Users can provide visual feedback and request corrected results, extending interaction beyond fixed, single-round model inputs and outputs.
5
Visual ChatGPT integrates multiple Visual Foundation Models with ChatGPT to support multimodal language-and-image interaction.
Research Object
Visual ChatGPT system integrating multiple Visual Foundation Models
Research Subject
Multimodal conversational interaction, visual question answering, image generation and editing through multi-model, multi-step collaboration with feedback
Publication Details
Publication Date
2023-03-08
Journal
Publisher
ISSN
Cited by
1098
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai5
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
Transformers: State-of-the-Art Natural Language Processing2020
Training language models to follow instructions with human feedback2022
Scaling Vision Transformers2022
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model2022
Cited by10
A Survey of Large Language Models2026
A Comprehensive Overview of Large Language Models2025
ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope2023
MM-LLMs: Recent Advances in MultiModal Large Language Models2024
A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges2023
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent2024
Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations2025
Materials science in the era of large language models: a perspective2024
Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language model2024
A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications2026