Generalized Expectation Criteria
Contents
Summary
This can be viewed as a parameter estimation method that can augment/replace traditional parameter estimation methods such as maximum likelihood estimation. M
Support Vector Machines or Conditional Random Fields to efficiently optimize the objective function, especially in the online setting. Stochastic optimizations like this method are known to be faster when trained with large, redundant data sets.
Expectation
Let be some set of variables and their assignments be . Let be the parameters of a model that defines a probability distribution . The expectation of a function according to the model is
![{\displaystyle E_{\theta }[f(X)]=\sum _{\mathbf {x} \in {\mathcal {X}}}{p_{\theta }(\mathbf {x} )f(\mathbf {x} )}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/08581329d8a6af7e4900c2e16d4be87f4631f1ec)
We can partition the variables into "input" variables and "output" variables that is conditioned on the input variables. When the assignment of the input variables are provided, the conditional expectation is
![{\displaystyle E_{\theta }[f(X,Y)\vert {\tilde {\mathcal {X}}}]={1 \over \vert {\tilde {\mathcal {X}}}\vert }\sum _{\mathbf {x} \in {\mathcal {\tilde {\mathcal {X}}}}}{\sum _{\mathbf {y} \in Y}{p_{\theta }(\mathbf {y} \vert \mathbf {x} )f(\mathbf {x} ,\mathbf {y} )}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/11162b92913638414c96e8e6b9dc5cb06a6c6703)
Generalized Expectation
A generalized expectation (GE) criteria is a function G that takes the model's expectation of as an argument and returns a scalar. The criteria is then added as a term in the parameter estimation objective function.
![{\displaystyle G(E_{\theta }[f(X)])\rightarrow \mathbb {R} }](https://wikimedia.org/api/rest_v1/media/math/render/svg/2555f6e0027bf6d2411243efcdf344c1c67c1d1f)
Or can be defined based on a distance to a target value for . Let be the target value and be some distance function, then we can define in the following way:
![{\displaystyle E_{\theta }[f(X)]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/92daab708e2c2b29919e631a8de0177b2abfc599)
![{\displaystyle G_{\tilde {f}}(E_{\theta }[f(X)])=-\Delta (E_{\theta }[f(X)],{\tilde {f}})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/24d37e16a7e46f4285f29a03a9d7a3762b5f9beb)
Stochastic Gradient Descent
Pros
When this method is used for very large data sets that has redundant information among examples, it is much faster than the plain gradient descent because it requires less computation each iteration. Also, it is known to be better with noisy data since it samples example to compute gradient.
Cons
The convergence rate is slower than second-order gradient methods. However the speedup coming from computationally efficient iterations are usually greater and the method can converge faster if learning rate is adjusted as the procedure goes on. Also it tends to keep bouncing around the minimum unless the learning rate is reduced in the later iterations.
