mixup: Beyond Empirical Risk Minimization
mixup: За пределами эмпирической минимизации риска
2017-10-25
SCID: 54.1/atxd5sm9
Discuss with AI
convex combinations of examples and labelsmixupreduced memorization of corrupt labelsregularization via linear behavior between examplesrobustness to adversarial examples
Figures from the paper
Abstract (AI)
Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.
Key Findings
1
mixup improves generalization of state-of-the-art architectures on ImageNet-2012, CIFAR-10, CIFAR-100, Google commands, and UCI datasets.
2
mixup increases robustness to adversarial examples, making models less sensitive to adversarial perturbations.
3
mixup reduces memorization of corrupted labels, mitigating overfitting to noisy annotations.
4
mixup stabilizes the training of generative adversarial networks, improving training dynamics.
5
mixup trains neural networks on convex combinations of pairs of examples and their labels, enforcing simple linear behavior between training points.
Research Object
Deep neural network models trained with the mixup data augmentation principle
Research Subject
Effects of training on convex combinations of example pairs (mixup) on generalization, memorization of corrupted labels, adversarial robustness, and training stability of neural networks
Publication Details
Publication Date
2017-10-25
Journal
Publisher
ISSN
Cited by
5659
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai6
Very Deep Convolutional Networks for Large-Scale Image Recognition2014
Rethinking the Inception Architecture for Computer Vision2016
Statistical Learning Theory1999
Aggregated Residual Transformations for Deep Neural Networks2017
Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups2012
Data Driven VRP: A Neural Network Model to Learn Hidden Preferences for VRP2021
Cited by16
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions2021
Randaugment: Practical automated data augmentation with a reduced search space2020
A survey on semi-supervised learning2019
Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks2023
PVT v2: Improved baselines with pyramid vision transformer2022
TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios2021
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet2021
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification2021
Automated Machine Learning2019
Vision Transformer with Deformable Attention2022
Transformer in Transformer2021
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better2023
Potential, challenges and future directions for deep learning in prognostics and health management applications2020
A Small-Sized Object Detection Oriented Multi-Scale Feature Fusion Approach With Application to Defect Detection2022
Automated data processing and feature engineering for deep learning and big data applications: A survey2024