10-601 Deep Learning 2

This a lecture used in the Syllabus for Machine Learning 10-601B in Spring 2016

Slides

• Slides in PowerPoint,Slides in PDF.

Wrapup from next lecture:

• Slides in PowerPoint,Slides in PDF.

Readings

This area is moving very fast and the textbooks are not up-to-date. Some recommended readings:

• Stanford CS class CS231n: Convolutional Neural Networks for Visual Recognition has nice on-line notes.

I also used some on-line visualizations in the materials for the lecture, especially the part on ConvNets.

• the Wikipedia page for convolutions has nice animations of 1-D convolutions.
• On-line demo of 2-D convolutions for image processing.
• There's an on-line demo of CNNs which are trained in your browser (!)
• 3D visualization of a trained net.

The LSTM figures and examples I used are mostly from

• Christopher Olah's blog
• The unreasonable effectiveness of RNNs
• For a great counterpoint: see Yoav Goldberg's response

Things to remember

• How backprop can be generalized to a sequence of assignment operations
• Convolutional networks
  • 2-d convolution
  • How to construct a convolution layer
  • Architecture of CNN: convolution/downsampling pairs
• Recurrent neural networks
  • When they are useful
  • Why they are hard to train (if trained naively)
  • The basic ideas used in an LSTM: forget, insert, and output gates