Dietterich 2008 gradient tree boosting for training conditional random fields

Citation

Gradient Tree Boosting for Training Conditional Random Fields. By T. G Dietterich, G. Hao, A. Ashenfelter. In Journal of Machine Learning Research, vol. 9 ({{{issue}}}), 2008.

Online Version

This Paper is available online [1].

Summary

The paper addresses the problem of combinatorial explosion of parameters of CRFs when new features are introduced. It represents the potential functions as sums of regression trees. The authors claim that adding a regression tree is a big step in the feature space and hence it reduces the number of iterations. This leads to a significant performance improvement.

Method

Functional Gradient Descent

In a traditional gradient descent method, a function $\Psi$ is represented by linear function of parameters. Thus, the parameter values after $m$ steps would be $\Theta _{m}=\Theta _{0}+\delta _{1}+\cdots +\delta _{m}$. Instead of parameterizing, functional gradient ascent assumes that $\Psi$ is a weighted sum of functions; $\Psi _{m}=\Psi _{0}+\Delta _{1}+\cdots +\Delta _{m}$.