Greedy function approximation: A gradient boosting machine.
2001-10-01
SCID: 54.1/7n6awp6b
Discuss with AI
additive expansionsboostinggradient boostingregression treessteepest-descent minimization
Figures from the paper
Abstract (AI)
Function estimation/approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest-descent minimization. A general gradient descent “boosting” paradigm is developed for additive expansions based on any fitting criterion.Specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification. Special enhancements are derived for the particular case where the individual additive components are regression trees, and tools for interpreting such “TreeBoost” models are presented. Gradient boosting of regression trees produces competitive, highly robust, interpretable procedures for both regression and classification, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire and Friedman, Hastie and Tibshirani are discussed.
Key Findings
1
A general gradient descent boosting paradigm is developed for building additive models using any fitting criterion.
2
Connections are established between this gradient boosting approach and prior boosting methods of Freund & Shapire and Friedman, Hastie & Tibshirani.
3
Gradient boosting of regression trees yields competitive, robust, and interpretable methods for regression and classification, suitable for noisy data.
4
Special enhancements and interpretation tools are derived for additive components that are regression trees (TreeBoost).
5
Specific algorithms are provided for least-squares, least absolute deviation, Huber-M regression, and multiclass logistic classification.
6
Viewing function estimation as optimization in function space links stagewise additive expansions to steepest-descent minimization.
Research Object
Gradient boosting machine for additive function approximation (including TreeBoost using regression trees)
Research Subject
The optimization paradigm and algorithmic behavior: stagewise additive expansions via gradient-descent in function space, including algorithms for various loss functions, enhancements for regression-tree components, robustness, interpretability, and performance for regression and classification
Publication Details
Publication Date
2001-10-01
Journal
Publisher
ISSN
Cited by
30275
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by20
XGBoost2016
Data mining: concepts and techniques2012
Least angle regression2004
Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)2000
Are Transformers Effective for Time Series Forecasting?2023
MoleculeNet: a benchmark for molecular machine learning2017
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence2023
Recent advances and applications of machine learning in solid-state materials science2019
CatBoost for big data: an interdisciplinary review2020
Machine Learning Interpretability: A Survey on Methods and Metrics2019
Automated Machine Learning2019
Materials discovery and design using machine learning2017
A comprehensive survey on machine learning for networking: evolution, applications and research opportunities2018
Questioning the AI: Informing Design Practices for Explainable AI User Experiences2020
Interpretable machine learning: Fundamental principles and 10 grand challenges2022
Self-Driving Laboratories for Chemistry and Materials Science2024
Machine learning advances for time series forecasting2021
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI2023
Deep learning in the construction industry: A review of present status and future innovations2020
High Resolution Mapping of Soil Properties Using Remote Sensing Variables in South-Western Burkina Faso: A Comparison of Machine Learning and Multiple Linear Regression Models2017