diff --git a/doc/index.md b/doc/index.md index 7c41d15e2..cdf1b12a2 100644 --- a/doc/index.md +++ b/doc/index.md @@ -52,6 +52,7 @@ User Guide ---------- * [Frequently Asked Questions](faq.md) * [Introduction to Boosted Trees](http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf) +* [Introduction to the Model of XGBoost](model.md) * [Using XGBoost in Python](python/python_intro.md) * [Using XGBoost in R](../R-package/vignettes/xgboostPresentation.Rmd) * [Learning to use XGBoost by Example](../demo) diff --git a/doc/model.md b/doc/model.md index aa42f7272..d55d262c2 100644 --- a/doc/model.md +++ b/doc/model.md @@ -15,15 +15,15 @@ XGBoost is used for supervised learning problems, where we use the training data Based on different understanding or assumption of ``$ y_i $``, we can have different problems as regression, classification, ordering, etc. To model different problems, we use a so-called `loss function` to describe how good is our model's performance. The function usually takes two parameters: the true value ``$ y_i $`` and the prediction ``$ \hat{y}_i $``. For example, we can use Rooted Mean Squared Error (RMSE) -`` `math +```math l(y_i, \hat{y}_i) = (y_i-\hat{y}_i)^2 -`` ` +``` for a regression problem, and logistic loss function -`` `math +```math l(y_i, \hat{y}_i) = y_i\ln (1+e^{-\hat{y}_i}) + (1-y_i)\ln (1+e^{\hat{y}_i}) -`` ` +``` for a classification problem. @@ -35,9 +35,9 @@ Besides we need to control the complexity of our model. A model achieving a perf Combining the loss function and the regularization, we have our objective for the supervised learning model as -`` `math +```math Obj(\Theta) = L(\Theta) + \Omega(\Theta) -`` ` +``` where ``$ L $`` is the loss function, and ``$ \Omega $`` is the regularization term. The first one is making our model being accurate, while the second one is preventing our model being overfitting. We want to have a balance between these two parts when optimizing the objective. The optimization algorithm depends on the structure of our model. The following content will introduce the details. @@ -59,60 +59,60 @@ However a single CART model is not so strong in practice. How about predict with Now we are predicting with two trees, by predict on each tree individually and then sum the scores up. Mathematically, we can write our model into the form -`` `math +```math \hat{y}_i = \sum_{k=1}^K f_k(x_i), f_k \in F -`` ` +``` where ``$ f $`` is a function in the functional space ``$ F $``, and ``$ F $`` is the set of all possible CARTs. Therefore our objective to optimize can be written as -`` `math +```math obj(\Theta) = \sum_i^n l(y_i, \hat{y}_i) + \sum_{k=1}^K \Omega(f_k) -`` ` +``` ### Additive Training It is not easy to train all the trees at once. Instead, we use the strategy to train them in a sequence so that everytime we train one CART and add it to the model. We note the prediction value at step `t` by ``$ \hat{y}_i^{(t)}$``, so we have -`` `math +```math \hat{y}_i^{(0)} = 0\\ \hat{y}_i^{(1)} = f_1(x_i) = \hat{y}_i^{(0)} + f_1(x_i)\\ \hat{y}_i^{(2)} = f_1(x_i) + f_2(x_i)= \hat{y}_i^{(1)} + f_2(x_i)\\ \dots\\ \hat{y}_i^{(t)} = \sum_{k=1}^t f_k(x_i)= \hat{y}_i^{(t-1)} + f_t(x_i) -`` ` +``` Which CART do we want at each step? Of course we want to add the one that minimize our objective. -`` `math +```math Obj^{(t)} & = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t)}) + \sum_{i=1}^t\Omega(f_i) \\ & = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega(f_t) + constant -`` ` +``` Let's consider using RMSE as our loss function -`` `math +```math Obj^{(t)} & = \sum_{i=1}^n (y_i - (\hat{y}_i^{(t-1)} + f_t(x_i)))^2 + \sum_{i=1}^t\Omega(f_i) \\ & = \sum_{i=1}^n [2(\hat{y}_i^{(t-1)} - y_i)f_t(x_i) + f_t(x_i)^2] + \Omega(f_t) + constant -`` ` +``` The form of RMSE is friendly. But other loss functions could be tricky to expand. For convenience we calculate the Taylor expansion of the loss function up to the second order -`` `math +```math Obj^{(t)} = \sum_{i=1}^n [l(y_i, \hat{y}_i^{(t-1)}) + g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)] + \Omega(f_t) + constant -`` ` +``` where -`` `math +```math g_i &= \partial_{\hat{y}_i^{(t)}} l(y_i, \hat{y}_i^{(t-1)})\\ h_i &= \partial_{\hat{y}_i^{(t)}}^2 l(y_i, \hat{y}_i^{(t-1)}) -`` ` +``` So we can remove all the constant at the t-th step and the specific objective is -`` `math +```math \sum_{i=1}^n [g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)] + \Omega(f_t) -`` ` +``` One of the benifit of this definition is as long as the loss function has the first and second order derivative, we can optimized every loss function within the same framework. @@ -120,15 +120,15 @@ One of the benifit of this definition is as long as the loss function has the fi We have introduced the details in the loss function, next we talk about the regularization term. We want to control the complexity of a tree, thus we need to define it first. We define a tree ``$ f(x) $`` as -`` `math +```math f_t(x) = w_{q(x)}, w\inR^T, q:R^d\rightarrow \{1,2,\cdots,T\} -`` ` +``` where ``$ w $`` is the vector of scores on leaves, ``$ q $`` is a function assigning each data point to the corresponding leaf and ``$ T $`` is the number of leaves. In XGBoost, we define the complexity as -`` `math +```math \Omega(f) = \gamma T + \frac{1}{2}\lambda \sum_{j=1}^T w_j^2 -`` ` +``` It is possible to define other form of regularization terms, but this one works well in practice. @@ -136,23 +136,23 @@ It is possible to define other form of regularization terms, but this one works Now we have the objective value with the ``$ t $``-th tree added: -`` `math +```math Obj^{(t)} &\approx \sum_{i=1}^n [g_i w_q(x_i) + \frac{1}{2} h_i w_{q(x_i)}^2] + \gamma T + \frac{1}{2}\lambda \sum_{j=1}^T w_j^2\\ &= \sum^T_{j=1} [(\sum_{i\in I_j} g_i) w_j + \frac{1}{2} (\sum_{i\in I_j} h_i + \lambda) w_j^2 ] + \gamma T -`` ` +``` where ``$ I_j = \{i|q(x_i)=j\} $`` is the set of indices of data points assigned to the ``$ j $``-th leaf. Notice that in the second line we have change the index of the summation because all the data points on the same leaf get the same score. We could further compress the expression by defining ``$ G_j = \sum_{i\in I_j} g_i $`` and ``$ H_j = \sum_{i\in I_j} h_i $``: -`` `math +```math Obj^{(t)} = \sum^T_{j=1} [G_jw_j + \frac{1}{2} (H_j+\lambda) w_j^2] +\gamma T -`` ` +``` In this equation ``$ w_j $`` are independent to each other, the form ``$ G_jw_j+\frac{1}{2}(H_j+\lambda)w_j^2 $`` is quadratic and the best ``$ w_j $`` to minimize it can be solved deterministically: -`` `math +```math w_j^\ast = -\frac{G_j}{H_j+\lambda}\\ Obj = -\frac{1}{2} \sum_{j=1}^T \frac{G_j^2}{H_j+\lambda} + \gamma T -`` ` +``` **Therefore, given the parameters, the gradients and the structure of the tree, we know how to set the score on each leaf.** @@ -162,11 +162,11 @@ Our algorithm aims at optimizing the objective, so it also guides us to a good t Specifically we try to split a leaf into two leaves, and the score it gains is -`` `math +```math Gain = \frac{1}{2} [\frac{G_L^2}{H_L+\lambda}+\frac{G_R^2}{H_R+\lambda}-\frac{(G_L+G_R)^2}{H_L+H_R+\lambda}] - \gamma -`` ` +``` -This formula can be decomposited as 1) the score on the new left leaf 2) the score on the new right leaf 3) The score on the original leaf 4) regularization on the additional leaf. +This formula can be decomposited as 1) the score on the new left leaf, 2) the score on the new right leaf, 3) The score on the original leaf and 4) regularization on the additional leaf. The regularization in the end can be seen as the minimum increment from this split. In the end, we will prune out the split with a negative gain.