Legrange Multipliers

Legrange Multipliers allow us to optimize a function (maximize or minimize) subject to constraints (such as the solution must lie on a specific curve. Dan Klein has a nice tutorial on Lagrange Multipliers, including inequality constraints (less-than or greater-than). The MIT Math Department has a nice applet that shows the solution space for f(x,y) and g(x,y). Notice that as you move the point around, the gradient of f and g (shown as arrows) are parallel at solution points.

Handwritten numbers... data!

Our upcoming projects will use the MNIST handwritten digit database as training and test data. Note that the link has classification results for many techniques taken from the pattern recognition literature. Also notice that the dataset has 60K training patterns and 10K test patterns, where each sample is a 400 dimensional vector. Consider how this might affect a naive least square linear discriminant fit.