A survey on data‐efficient algorithms in big data era
Обзор алгоритмов с экономным использованием данных в эпоху больших данных
2021-01-26
SCID: 54.1/m3c9s6rp
Discuss with AI
data augmentationdata-efficient algorithmssmall-sample learningtransfer learningunsupervised learning
Figures from the paper
Abstract (AI)
Abstract The leading approaches in Machine Learning are notoriously data-hungry. Unfortunately, many application domains do not have access to big data because acquiring data involves a process that is expensive or time-consuming. This has triggered a serious debate in both the industrial and academic communities calling for more data-efficient models that harness the power of artificial learners while achieving good results with less training data and in particular less human supervision. In light of this debate, this work investigates the issue of algorithms’ data hungriness. First, it surveys the issue from different perspectives. Then, it presents a comprehensive review of existing data-efficient methods and systematizes them into four categories. Specifically, the survey covers solution strategies that handle data-efficiency by (i) using non-supervised algorithms that are, by nature, more data-efficient, by (ii) creating artificially more data, by (iii) transferring knowledge from rich-data domains into poor-data domains, or by (iv) altering data-hungry algorithms to reduce their dependency upon the amount of samples, in a way they can perform well in small samples regime. Each strategy is extensively reviewed and discussed. In addition, the emphasis is put on how the four strategies interplay with each other in order to motivate exploration of more robust and data-efficient algorithms. Finally, the survey delineates the limitations, discusses research challenges, and suggests future opportunities to advance the research on data-efficiency in machine learning.
Key Findings
1
Altering data-hungry algorithms to reduce sample dependency enables those algorithms to perform well in small-sample regimes.
2
Creating artificial data (data augmentation) is a distinct strategy to increase effective training samples and improve performance with limited real data.
3
Machine learning methods are generally data-hungry and many application domains lack large labeled datasets due to expensive or time-consuming data acquisition.
4
The paper identifies limitations, research challenges, and future opportunities to advance data-efficiency in machine learning.
5
The survey emphasizes interactions among the four strategies, suggesting combined use may yield more robust and data-efficient algorithms.
6
The survey organizes data-efficient approaches into four main strategies: unsupervised algorithms, data augmentation (creating artificial data), transfer learning, and modifying data-hungry algorithms to work in small-sample regimes.
7
Transferring knowledge from rich-data domains to poor-data domains is a key approach to leverage existing labeled resources and improve performance where data is scarce.
8
Unsupervised algorithms are characterized as inherently more data-efficient and are a primary strategy for reducing reliance on labeled data.
Research Object
data-efficient algorithms in machine learning
Research Subject
methods and strategies to reduce algorithms' data hungriness, enabling good performance with limited training data and reduced supervision (including unsupervised methods, data augmentation, transfer learning, and algorithmic adaptations)
Publication Details
Publication Date
2021-01-26
Journal
Publisher
ISSN
Cited by
368
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai16
Deep Residual Learning for Image Recognition2016
ImageNet classification with deep convolutional neural networks2017
Gradient-based learning applied to document recognition1998
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
Image-to-Image Translation with Conditional Adversarial Networks2017
A survey on Image Data Augmentation for Deep Learning2019
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks2015
Proceedings of the 24th international conference on Machine learning2007
A survey of transfer learning2016
Image Style Transfer Using Convolutional Neural Networks2016
Randaugment: Practical automated data augmentation with a reduced search space2020
Deep Reinforcement Learning: A Brief Survey2017
Continual lifelong learning with neural networks: A review2019
A survey on semi-supervised learning2019
EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks2019
Distribution Consistency Based Covariance Metric Networks for Few-Shot Learning2019