Nearest-neighbor chain algorithm

In machine learning, the hinge loss is a loss function used for training classifiers. The hinge loss is used for "maximum-margin" classification, most notably for support vector machines (SVMs). For an intended output t = ±1 and a classifier score y, the hinge loss of the prediction y is defined as

$\ell (y)=\max(0,1-t\cdot y)$

Note that y should be the "raw" output of the SVM's decision function, not the predicted class label. E.g., in linear SVMs, $y=\mathbf {w} \cdot \mathbf {x} +b$.

It can be seen that when $t$ and $y$ have the same sign (meaning $y$ predicts the right class) and $y\geq 1$, $\ell (y)=0$, but when they have opposite sign, $\ell (y)$ increases linearly with $y$ (one-sided error).

Extensions

While SVMs are commonly extended to multiclass classification in a one-vs.-all or one-vs.-one fashion, there exists a "true" multiclass version of the hinge loss due to Crammer and Singer, defined for a linear classifier as

$\ell (y)=\max(0,1+\max _{y\neq t}\mathbf {w} _{y}\mathbf {x} -\mathbf {w} _{t}\mathbf {x} )$

In structured prediction, the hinge loss can be further extended to structured output spaces. Structured SVMs use the following variant, where w denotes the SVM's parameters, φ the joint feature function, and Δ the Hamming loss:

${\begin{aligned}\ell (\mathbf {y} )&=\Delta (\mathbf {y} ,\mathbf {t} )+\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {y} )\rangle -\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {t} )\rangle \\&=\max _{y\in {\mathcal {Y}}}\left(\Delta (\mathbf {y} ,\mathbf {t} +\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {y} )\rangle )\right)-\langle \mathbf {w} ,\phi (\mathbf {x} ,\mathbf {t} )\rangle \end{aligned}}$

Optimization

The hinge loss is a convex function, so many of the usual convex optimizers used in machine learning can work with it. It is not differentiable, but has a subgradient with respect to model parameters $\mathbf {w}$ of a linear SVM with score function $y=\mathbf {w} \cdot \mathbf {x}$ that is given by

${\frac {\partial \ell }{\partial w_{i}}}={\begin{cases}-t\cdot x_{i}&{\text{if }}t\cdot y<1\\0&{\text{otherwise}}\end{cases}}$

References

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.