Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
Является ли ChatGPT универсальным решателем задач обработки естественного языка?
2023-01-01
SCID: 54.1/7c6gr5jr
Discuss with AI
ChatGPTarithmetic reasoningnatural language processing taskssequence taggingzero-shot learning
Figures from the paper
Abstract (AI)
Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot-i.e., without adaptation on downstream data.Recently, the debut of ChatGPT 1 has drawn a great deal of attention from the natural language processing (NLP) community due to the fact that it can generate high-quality responses to human input and self-correct previous mistakes based on subsequent conversations.However, it is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.In this work, we empirically analyze the zero-shot learning ability of ChatGPT by evaluating it on 20 popular NLP datasets covering 7 representative task categories.With extensive empirical studies, we demonstrate both the effectiveness and limitations of the current version of ChatGPT.We find that ChatGPT performs well on many tasks favoring reasoning capabilities (e.g., arithmetic reasoning) while it still faces challenges when solving specific tasks such as sequence tagging.We additionally provide in-depth analysis through qualitative case studies.
Key Findings
1
ChatGPT performs effectively on many NLP tasks that favor reasoning capabilities, including arithmetic reasoning.
2
ChatGPT remains limited on task-specific problems such as sequence tagging, indicating it is not uniformly general-purpose.
3
Qualitative case studies provide deeper analysis of ChatGPT’s zero-shot successes and failure modes.
4
The study evaluates ChatGPT’s zero-shot performance on 20 popular NLP datasets spanning seven representative task categories.
Research Object
ChatGPT as a large language model evaluated on 20 NLP datasets spanning 7 task categories
Research Subject
ChatGPT’s zero-shot performance, effectiveness, and limitations across diverse NLP tasks, including reasoning and sequence tagging
Publication Details
Publication Date
2023-01-01
Journal
Publisher
ISSN
Cited by
888
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by8
A Survey on Evaluation of Large Language Models2024
A Comprehensive Overview of Large Language Models2025
ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope2023
Can Large Language Models Transform Computational Social Science?2023
ChatGPT and Open-AI Models: A Preliminary Review2023
Mental-LLM2024
Knowledge Graphs and Their Reciprocal Relationship with Large Language Models2025
Large language model agent as a mechanical designer2026